Status of testing of Apache Airflow 2.6.1rc3

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.6.1rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Prevent DagRun's start_date from reset (#30124) (#30125): @dimonchik-suvorov
  Linked issues:
  • DagRun's start_date updated when user clears task of the running Dagrun (#30124)
• Reenable clear on TaskInstanceModelView for role User (#30415): @eejbyfeldt
• Table for affected tasks in grid actions (#30633): @pierrejeambrun
• Fix backfill KeyError when try_number out of sync (#30653): @ephraimbuddy @sarvothaman @huozhanfeng @awbush
  Linked issues:
  • [Bug] Backfill job fails to run when there are tasks run into rescheduling state. (#17305)
  • Backfill crashes with "KeyError: TaskInstanceKey" when task has retries (#13322)
  • Add unittest for #17305 (#18806)
• Fix problem with displaying graph (#30765): @meujin
• Fix mappedStates case (#30916): @bbovenzi
• Improve task group UI in new graph (#30918): @bbovenzi
• Remove print from dag trigger command (#30921): @pankajastro
• Include node height/width in center-on-task logic (#30924): @bbovenzi
• Fix order_by request in list DAG rest api (#30926): @pankajastro
• Use the Task Group explicitly passed to 'partial' if any (#30933): @LuizArmesto
  Linked issues:
  • Tasks created using "dynamic task mapping" ignore the Task Group passed as argument (#30932)
• Add support for dynamic tasks with template fields that contain pandas.DataFrame (#30943): @tatiana
• Upgrade PIP to 23.1.2 (#30964): @potiuk
• Update incorrect config values (#30969): @ephraimbuddy
• Fix Pool schema OpenAPI spec (#30973): @pierrejeambrun
• Fix output for airflow providers get command (#30978): @pankajastro
• Fix unable to remove DagRun and TaskInstance with note (#30987): @herlambang
  Linked issues:
  • Unable to remove DagRun and TaskInstance with note (#30984)
• Fix missing models in API documentation. (#31021): @potiuk
  Linked issues:
  • Add Public Interface description to Airflow documentation (#28300)
• Move TaskInstanceKey to a separate file to fix circular import from kubernetes_helper_functions.py (#31033): @LipuFei
  Linked issues:
  • DB migration job fails with circular import (#30988)
• Fix issue in pre-importing modules in zipfile (#31061): @ephraimbuddy
• Update provider to apache-hive suffix. (#31068): @davidshur
  Linked issues:
  • [BUG] apache.hive extra is referencing incorrect provider name (#31067)
• Fix template error when iterating None value and fix params documentation (#31078): @herlambang @Limess
  Linked issues:
  • Exception when manually triggering dags via UI with params defined. (#31014)
• Fix Spacing for DagBag.dagbag_report()'s Output (#31095): @mtsadler-branch
• fix MappedTaskGroup import in taskinstance (#31100): @JCoder01
• Ensure that we check worker for served logs if no local or remote logs found (#31101): @dstandish @juraa14 @zachliu
  Linked issues:
  • Logs no longer shown after task completed CeleryExecutor (#31059)
  • Incorrect logic when reading from local log files (#30997)
• Move interleave_timestamp_parser config to section logging (#31102): @dstandish
• Fix unmasked env var on task detail view (#31125): @herlambang @chriscmorgan
  Linked issues:
  • Connection password values appearing unmasked in the "Task Instance Details" -> "Environment" field (#30089)
• Remove worrying log message about redaction caused by OpenLineage plugin (#31149): @ashb
• Ensure the KPO runs pod mutation hooks correctly (#31173): @jedcunningham
• Fix section name reference for api_client_retry_configuration (#31174): @dstandish
• Fix timestamp parse failure for k8s executor pod tailing (#31175): @dstandish
• Make sure that DAG processor job row has filed value in job_type column (#31182): @AmFlint
• Fix calculation of health check threshold for SchedulerJob (#31277): @potiuk

Thanks to all who contributed to the release (probably not a complete list!):
@pierrejeambrun @LipuFei @pankajastro @JCoder01 @davidshur @zachliu @meujin @LuizArmesto @Limess @AmFlint @juraa14 @jedcunningham @tatiana @bbovenzi @huozhanfeng @herlambang @sarvothaman @chriscmorgan @potiuk @eejbyfeldt @dstandish @awbush @mtsadler-branch @dimonchik-suvorov @ashb @ephraimbuddy

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[LipuFei]

Hi @ephraimbuddy , I've just tested #31033, but the DB job still fails due to another circular import. I found a fix for this. Can I directly make a new PR or should I also raise an issue for this?

UPDATE: I've created a PR to fix this #31204
UPDATE 2: #31204 was merged.

[herlambang]

UAT Test Result v2.6.1rc1

Dashboard Footer:
Version: v2.6.1
Git Version: .release:06e3aa3e40948efcb53e9320b78d6b33b875ad12

Fix unable to remove DagRun and TaskInstance with note (#30987) ✅

• open page /dagrun/list/
• edit 1 dag run, fill note field and save
• delete dag run previously edited
• dag run deleted
• open page /taskinstance/list/
• edit 1 task, fill note field and save
• delete task instance previously edited
• task instance deleted

Fix template error when iterating None value and fix params documentation (#31078) ✅

• Create dag with params

"array_param": Param(
    None,
    type=["null", "array"],
),

• Click dag's play button, select Trigger DAG w/ config
• Page loaded without error
• Form field array_param shown with empty textbox
• Generated Configuration JSON shows

{
    "array_param": null
}

Fix unmasked env var on task detail view (#31125) ✅

• Create variable with name contains secret
• Set environment variable with secret variable in an operator

run_this = BashOperator(
    task_id="masked_envvar",
    bash_command="echo 1",
    env={
        "ENVVAR_NAME": "{{ var.value.secret_service }}"
    },
)

• Run the dag and open task instance detail page
• Task Attributes env is masked

env	{'ENVVAR_NAME': '***'}

[ephraimbuddy]

Hi @ephraimbuddy , I've just tested #31033, but the DB job still fails due to another circular import. I found a fix for this. Can I directly make a new PR or should I also raise an issue for this?

UPDATE: I've created a PR to fix this #31204 UPDATE 2: #31204 was merged.

Thanks @LipuFei . I have created rc2 with your fix. Verify and let me know how it goes

[potiuk]

Checked PIP/docs to see if missing models are fixed - looks good

[LipuFei]

Hi @ephraimbuddy , I've just tested #31033, but the DB job still fails due to another circular import. I found a fix for this. Can I directly make a new PR or should I also raise an issue for this?
UPDATE: I've created a PR to fix this #31204 UPDATE 2: #31204 was merged.

Thanks @LipuFei . I have created rc2 with your fix. Verify and let me know how it goes

Checked #31033 and #31204 with 2.6.1rc2. Looks good! The migration job can run successfully.

Thanks! @ephraimbuddy

[potiuk]

Tested the #31277 behaviour. Works as expected now:

1. started airflow

2. both jobs check and UI show it as live

3. kill scheduler FORECEFULLY (i.e. with -9)

4. still alive
   5s

5. still alive
   10s

6. still alive
   ....

35 seconds:

The scheduler does not appear to be running. Last heartbeat was received 35 s ago.

Everything as expected

[pankajastro]

Tested #30921 #30926 #30978 looks good. thanks!

[ephraimbuddy]

2.6.1 has been released. Thank you all for testing this release candidate.