Constant "The scheduler does not appear to be running" warning on the UI following 2.6.0 upgrade

[arjunanan6]

Apache Airflow version

2.6.0

What happened

Ever since we upgraded to Airflow 2.6.0 from 2.5.2, we have seen that there is a warning stating "The scheduler does not appear to be running" intermittently.

This warning goes away by simply refreshing the page. And this conforms with our findings that the scheduler has not been down at all, at any point. By calling the /health point constantly, we can get it to show an "unhealthy" status:

These are just approx. 6 seconds apart:

{"metadatabase": {"status": "healthy"}, "scheduler": {"latest_scheduler_heartbeat": "2023-05-11T07:42:36.857007+00:00", "status": "healthy"}}

{"metadatabase": {"status": "healthy"}, "scheduler": {"latest_scheduler_heartbeat": "2023-05-11T07:42:42.409344+00:00", "status": "unhealthy"}}

This causes no operational issues, but it is misleading for end-users. What could be causing this?

What you think should happen instead

The warning should not be shown unless the last heartbeat was at least 30 sec earlier (default config).

How to reproduce

There are no concrete steps to reproduce it, but the warning appears in the UI after a few seconds of browsing around, or simply refresh the /health endpoint constantly.

Operating System

Debian GNU/Linux 11

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-docker==3.6.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==10.0.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-microsoft-mssql==3.3.2
apache-airflow-providers-microsoft-psrp==2.2.0
apache-airflow-providers-microsoft-winrm==3.0.0
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-oracle==3.0.0
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployed on AKS with helm

Anything else

None more than in the description above.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[PhoenixChamb]

We are facing the same problem after upgrading Airflow from 2.5.3 to 2.6.0

[potiuk]

@arjunanan6 and @PhoenixChamb could you please increase the https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#job-heartbeat-sec to say 7 s and then increase it to 10 if it does not help (if you have default 5) and see if that decreases the probability of such an event and report it back here?

That would help us to be able to see if just increasing the default is a good idea because of increased resource usage in 2.6 or whether we have some other problem.

[arjunanan6]

@potiuk On it, I will report back shortly.

[arjunanan6]

@potiuk I'm afraid that did not really help the situation. I need more refreshes to see the warning, but it still appears 'incorrectly'.

[ptran32]

We also faced this issue after upgrading from Airflow 2.5.3 to 2.6.0, the scheduler runs for 2 minutes, then restarts

Scheduler logs: https://gist.github.com/ptran32/56703c86a854ec80bdae1d5195c182e7

We also notice an error on the liveness of the pod:

Liveness probe failed: Traceback (most recent call last): File "<string>", line 2, in <module> ModuleNotFoundError: No module named 'airflow.jobs.scheduler_job'

[arjunanan6]

As @ptran32 noted, we are also seeing unnecessary restarts for the scheduler because a livenessprobe failed. Interestingly, this was not a problem today, but something that started late last night - several hours after the upgrade.

But, I really cannot see any issues from the scheduler logs that would cause the probe to fail. As a temporary solution, I am trying out increasing the following values:

• scheduler.livenessProbe.timeoutSeconds to 30
• scheduler.livenessProbe.failureThreshold to 10

Hopefully, this results in no/fewer restarts.

[ptran32]

As @ptran32 noted, we are also seeing unnecessary restarts for the scheduler because a livenessprobe failed. Interestingly, this was not a problem today, but something that started late last night - several hours after the upgrade.

But, I really cannot see any issues from the scheduler logs that would cause the probe to fail. As a temporary solution, I am trying out increasing the following values:

• scheduler.livenessProbe.timeoutSeconds to 30
• scheduler.livenessProbe.failureThreshold to 10

Hopefully, this results in no/fewer restarts.

Hi,

if you are using Helm to deploy Airflow, the 3.6.0 version will cause the scheduler to restart in loop, not because of the timeout (at least in my case) but because the liveness check is not compatible with the new release

Scheduler logs after upgrade to 3.6.0

Liveness probe failed: Traceback (most recent call last): File "<string>", line 2, in <module> ModuleNotFoundError: No module named 'airflow.jobs.scheduler_job'

We fixed it by adjusting the liveness check command in the helm chart:

      livenessProbe:
          exec:
            command:
            - python
            - -Wignore
            - -c
            - |
              from typing import List
              from airflow.jobs.scheduler_job_runner import SchedulerJobRunner

I hope it helps.

[rcheatham-q]

I can confirm that the default liveness probe for the scheduler is failing. The helm chart v1.9.0 defines the liveness probe as follows:

CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint airflow jobs check --job-type SchedulerJob --local

However, executing that command on a live and functioning scheduler pod fails (I omitted the exec command so I could check the exit code):

airflow@airflow-scheduler-866db5c895-qt5pq:/opt/airflow$ CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR /entrypoint airflow jobs check --job-type SchedulerJob --local
No alive jobs found.
airflow@airflow-scheduler-866db5c895-qt5pq:/opt/airflow$ echo $?
1
airflow@airflow-scheduler-866db5c895-qt5pq:/opt/airflow$ 

I can confirm that the liveness check listed by @ptran32 does function, however it does not seem to directly detect if the scheduler job is running.

I also tested the example airflow jobs check commands listed in the CLI docs and both functioned the same.

It appears that the scheduler healthchecks in general are broken in version 2.6.0.

[ptran32]

@rcheatham-q that might help as a workaround for now: airflow-helm/charts#738