Drop the LogTemplate DB model#69520
Conversation
04e801f to
81fa395
Compare
81fa395 to
1984f03
Compare
The ES/OpenSearch task handlers reached into the metadata DB directly to read the per-DagRun log template, which conflicts with the Airflow 3 execution model where task handlers shouldn't query the DB directly. Log filename/ID templates are now always read from live config instead of being pinned to the value in effect when a DagRun was created.
1984f03 to
351b9ad
Compare
|
The entire reason this exists is so that if the config changes the old logs are still viewable. This breaks that doesn't it |
This isn't true - we already handle this correctly since 3.0.0 by passing the log file suffix to the worker in ExecuteTaskWorkload; changes in 2.11 directly aren't available in 3.x either so that linked pr isn't really relevant |
Yes, but based on
I thought we can introduce the breaking change in
It's different part, not for the log file suffix, only the ES / OS consumed the LogTemplate DB model. IIRC, the ES / OS provider (mainly #53821) that works with Airflow 3 didn't respect the log_template model (both write and read side will fetch the latest config value instead of going through the LogTemplate model. When accessing the log_template property on TI, it hit the prohibited commit exception, so we workaround as directly accessing the conf.) at all (I will double check for this part) this is the reason why I raise the discussion to drop the LogTemplate DB model. Or another direction, I can make the OS and ES provider still respect the LogTemplate DB model (introducing LogTemplate accessor, or retrieve the LogTemplate at Execution API and set as the StartUpDetails, etc) |
|
LogTemplate is used well beyond just ES -- it's used to generate the log file suffix that is passed to workers where they write the local log file, and for reading old task logs back from S3/GCS etc. |
Thanks for the clarification. I see, then I will go through the flow again then restoring the LogTemplate for ES should be the correct direction, thanks. |
|
Dropping the LogTemplate DB model is the wrong direction, create #69688 to resolve the root cause. |
Why
The
LogTemplatemodel pinned historical values of[logging] log_filename_template/[elasticsearch] log_id_templateperDagRun, and task handlers read it viadag_run.get_log_template(session=...)— a direct metadata-DB dependency that conflicts with the Airflow 3 execution model and couples log handlers to an Airflow-core model just to read two strings already available from config.Airflow 2.11.1 already disabled per-run template retrieval by default for security reasons (#61880); this completes that direction on main by removing the model entirely.
What
LogTemplatemodel, thedag_run.log_template_idcolumn/FK,DagRun.get_log_template(), andsynchronize_log_template()with its call sites.FileTaskHandler._render_filename()now reads[logging] log_filename_templatefrom live config at render time.0126_3_4_0_drop_log_template.py(downgrade recreates the column and table); both directions usedisable_sqlite_fkeys(op)so the SQLitedag_runrebuild doesn't cascade-deletetask_instancerows.USE_PER_RUN_LOG_IDflag — defined but unused sinceElasticsearchRemoteLogIO, so dead-code cleanup only.hasattr(DagRun, "get_log_template")shim — per-run pinning must keep working on cores 3.0-3.3; on 3.4.0+ it self-disables and the handler falls back to[opensearch] log_id_templatefrom config.create_log_templatefixture version-adaptive so provider tests pass against all supported cores; update ES docs and add a significant-change newsfragment.Behaviour change (core 3.4.0+): changing
log_filename_template/log_id_templatenow applies immediately and retroactively to all DagRuns — there is no more per-run historical pinning, so logs written under a previous template are no longer reachable through the new one.Was generative AI tooling used to co-author this PR?