Logs no longer shown after task completed CeleryExecutor

[juraa14]

Apache Airflow version

2.6.0

What happened

Stream logging works as long as the task is running. Once the task finishes, no logs are printed to the UI (only the hostname of the worker is printed)

What you think should happen instead

Expected to see the complete log of a task

How to reproduce

Start an airflow task. You should be able to see the logs coming in as a stream, once it finishes, the logs are gone

Operating System

CentOS 7

Versions of Apache Airflow Providers

airflow-provider-great-expectations==0.1.5
apache-airflow==2.6.0
apache-airflow-providers-airbyte==3.1.0
apache-airflow-providers-apache-hive==4.0.1
apache-airflow-providers-apache-spark==3.0.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-oracle==3.4.0
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-sqlite==3.3.2

Deployment

Virtualenv installation

Deployment details

Celery with 4 workers nodes/VMs. Scheduler and Webserver on a different VM

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[rohan472000]

This issue could be related to the logging configuration of your Airflow instance. By default, Airflow uses a logging level of WARNING, which means that only log messages with severity levels of WARNING, ERROR, and CRITICAL will be shown in the UI.

To check the logging configuration, you can go to your Airflow home directory and open the airflow.cfg file. Look for the [logging] section and check the values of the following parameters:

logging_level = WARNING
logging_config_class = log_config.LOGGING_CONFIG

If you want to see more log messages in the UI, you can set the logging level to a lower value, such as INFO or DEBUG:

logging_level = INFO

Save the changes to the airflow.cfg file and restart the Airflow services.

[juraa14]

Apologies for not giving all parameters. Logging level is already set to INFO but the bug persists. Also seems the issue was already posted before #30997

[zachliu]

i closed #30997 because although the line

airflow/airflow/utils/log/file_task_handler.py

Line 323 in ab54c63

if not (remote_logs and ti.state not in State.unfinished):

is hard to read, it does fulfill the expected logic cases as i mentioned #30997 (comment)

case#	remote_logs	task state	triple not logic final output
1	no remote logs	unfinished	True (read local)
2	got remote logs	finished	False (don't read local)
3	no remote logs	finished	True (read local)
4	got remote logs	unfinished	True (read local)

i didn't know case 2 was causing issues for other people 🤔
in my implementation, i only care about case 1,3,4 (i do have remote logging as permanent records but i have my own log_config.LOGGING_CONFIG to force remote logs to be empty so that airflow always go to case 1,3,4 because reading from local is much faster)

[potiuk]

Can you please explain what logging configuration you have @juraa14 (cc: @dstandish I guess that's something to look at quickly).

[juraa14]

Logging level set to INFO. 4 worker nodes executing tasks, each on a different VM. Webserver located on a separate VM fetches logs from these worker VM's on port 8793. Remote logging is not set

[dstandish]

@juraa14 let me know if i understand this correctly.

you have airflow configured running celery.

each celery worker runs on a distinct VM.

webserver runs on distinct VM.

when task is running, webserver reads through log server.

when task is done, webserver does not read through logs server.

correct?

this makes sense. the current logic is to only attempt to read from the worker (over the log server running on celery)while the task is running and otherwise to just read local or remote.

i think in general the assumption is that celery workers cannot be relied upon to stick around forever. so you should either set up your logs to go to a folder shared by all of your services (e.g. celery, webserver, scheduler) or you should set up remote logging. doing otherwise is not a good practice because your logs can easily be lost. and since it's not a good practice, it's not expected that people will do it like this, and therefore we can optimize webserver performance by not attempting to read from the logs server in this case.

so my recommendation would be that you set up a shared drive for logs or enable remote logging (it's very easy to do)

let me know your thoughts

[juraa14]

You understood correctly
If it's the expected outcome, then yes I agree with you. Will set up remote logging probably in the near future. Thanks for the help, I'm closing this issue then.