Airflow 2.7.1 can not start Scheduler & trigger

[ThuanDoHung]

Apache Airflow version

2.7.1

What happened

After upgrade from 2.6.0 to 2.7.1 (try pip uninstall apache-airflow, and clear dir airflow - remove airflow.cfg), I can start scheduler & trigger with daemon.
I try start with command, it can start, but logout console it killed.
I try: airflow scheduler or airflow triggerer :done but kill when logout console
airflow scheduler --daemon && airflow triggerer --daemon: fail, can not start scheduler & triggerer (2.6.0 run ok). but start deamon with webserver & celery worker is fine

Help me

What you think should happen instead

No response

How to reproduce

1. run airflow 2.6.0 fine on ubuntu server 22.04.3 lts
2. install airflow 2.7.1
3. can not start daemon triggerer & scheduler

Operating System

ubuntu server 22.04.3 LTS

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

Some questions regarding your report:

• Have you tried a clean install of Airflow 2.7.1 as well?
• Are you running in a Venv or in a system site package install?
• Can you please attach some logs to the problem? Especially what the logs of daemon processes are?
• Also can you describe how your service is usually instantiated? If you kill the terminal where you started it is normal that child processes are terminated as well. If you run in daemon mode they usually stay alive.

[ThuanDoHung]

• i run venv
• if start airflow triggerer then run and show log to console, but ctrl+c or close terminal triggerer killed.
• if start airflow trigger --daemon, i dont show any log, i dont known

[ThuanDoHung]

airflow triggerer --capacity 5 --log-file ./logs/triggerer/triggerer.log --pid ./triggerer.pid --daemon

---

____ |( )_______ / /________ __
____ /| |_ /__ / / __ / __ _ | /| / /
___ ___ | / _ / _ / _ / / // / |/ |/ /
// |// // // // _/____/|__/
[2023-10-10T00:03:58.394+0700] {triggerer_job_runner.py:171} INFO - Setting up TriggererHandlerWrapper with handler <FileTaskHandler (NOTSET)>
[2023-10-10T00:03:58.397+0700] {triggerer_job_runner.py:227} INFO - Setting up logging queue listener with handlers [<RedirectStdHandler (NOTSET)>, <TriggererHandlerWrapper (NOTSET)>]

--> any log on console, not show log in file

[jscheffl]

Sorry I can not understand you last comment completely.

• Can you please repeat and also add the verbose option via --verbose to the command?
• And can you also please post the logs of the scheduler as well?
• Have you tried a clean install of Airflow 2.7.1 as well?

[ThuanDoHung]

Hi,
I try:
Install new server (ubuntu server 22.04.3 LTS) and postgresql 16,
Install airflow:

wget https://github.com/apache/airflow/constraints-2.7.1/constraints-3.10.txt
pip3 install "apache-airflow[celery,cgroups,dask,statsd,virtualenv,redis]==2.7.1" --constraint "/home/constraints-3.10.txt"

then:

• config AIRFLOW_HOME in ~/.profile
• set up database on postgres and run airflow db migrate
• config database on airflow.cfg
• run:
  +/ airflow webserver --daemon --> ok, show webui on brower
  +/ airflow scheduler --daemon --> ok, show on cluster activity tab --> scheduler was Healthy
  +/ airflow triggerer --daemon --> NOT OK, show on cluter actity tab --> scheduler was Healthy, TRIGGERER UNHEALTHY
  i try:
  • airflow trigger --> cluster activity tab --> scheduler & triggerer was healthy but close terminal triggerer killed.

airflow triggerer --daemon, it's NOT OK:

airflow triggerer (without --daemon), it's OK:

run with -v (error same without -v), it's NOT OK

[ThuanDoHung]

I think if run triggerer on daemon, it not start gunicorn (i show gunicorn start on log file). if i run without daemon, i show starting gunicorn on console.

[calfzhou]

The same issue with 2.7.2, airflow triggerer -D not working anymore. In the error log file, i see that triggerer was kill by a term signal Handling signal: term right after worker booted.

[2023-10-13 12:15:23 +0800] [722477] [INFO] Starting gunicorn 21.2.0
[2023-10-13 12:15:23 +0800] [722477] [INFO] Listening at: http://[::]:8794 (722477)
[2023-10-13 12:15:23 +0800] [722477] [INFO] Using worker: sync
[2023-10-13 12:15:23 +0800] [722479] [INFO] Booting worker with pid: 722479
[2023-10-13 12:15:23 +0800] [722480] [INFO] Booting worker with pid: 722480
[2023-10-13 12:15:23 +0800] [722477] [INFO] Handling signal: term
[2023-10-13 12:15:23 +0800] [722479] [INFO] Worker exiting (pid: 722479)
[2023-10-13 12:15:23 +0800] [722480] [INFO] Worker exiting (pid: 722480)
[2023-10-13 12:15:23 +0800] [722477] [INFO] Shutting down: Master

[Bisk1]

I tested a couple of versions and can confirm it, the regression took place between 2.6.2 and 2.6.3 - in 2.6.3 (and later), after starting triggerer with -D option, airflow-triggerer.pid is not created, the process dies immediately.

[ThuanDoHung]

I tested a couple of versions and can confirm it, the regression took place between 2.6.2 and 2.6.3 - in 2.6.3 (and later), after starting triggerer with -D option, airflow-triggerer.pid is not created, the process dies immediately.

maybe can fix it or wait new version of airflow?

[Bisk1]

I think I isolated the code change that caused the problem - the problem starts to appear after this commit: d6cc9e4#diff-9cea7921268261a177e82c16fd5111f8d3252e3ca0267bdfb397c379c5d70857R353-R355

When I comment out these lines, the triggerer starts and the demon thread continues to run successfully.

@ThuanDoHung for now I'm trying to triage the bug, don't know how to fix it yet