Templates not resolved in DatabricksNotebookOperator then in DatabricksWorkflowTaskGroup used

[a-baturin]

Apache Airflow Provider(s)

databricks

Versions of Apache Airflow Providers

apache-airflow-providers-databrick 6.9.0

Apache Airflow version

2.9.2

Operating System

Amazon Linux 2023

Deployment

Amazon (AWS) MWAA

Deployment details

No response

What happened

Templates/macros defined in DatabricksNotebookOperator inside DatabricksWorkflowTaskGroup are passed to Notebook as template text, not as an expected value.

What you think should happen instead

Resolution happens on task execution. But Databricks job is already created at that moment

How to reproduce

If you define notebook_params in DatabricksWorkflowTaskGroup templates will be resolved, if in DatabricksNotebookOperator inside DatabricksWorkflowTaskGroup - not resolved.
In provided part of DAG (unrelated removed) param1 correctly passed to notebook, but param2 passed as {{ ds }}

    with DatabricksWorkflowTaskGroup(
...
        notebook_params={"param1": "{{ ds }}"},
    ) as tg:
        task1 = DatabricksNotebookOperator(
...
            notebook_params={"param2": "{{ ds }}"},
        )

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Feel free to implement it if you would like to

[ajitg25]

Hey @potiuk , I was looking into this issue and considering a fix, but I'm unable to figure out how to test it with Databricks. Do we need an account to test there and setup the connection with our PAT in Databricks, or is there a way to test it using the Airflow Breeze local setup?

[potiuk]

You need databricks account (your own). We have no "shared" account to use. If you do not have it, better to look at other issues.

[SantanuBasuMNS]

Would be great to get a fix .

On first run :
with DatabricksWorkflowTaskGroup(
...
notebook_params={"param1": "{{ ds }}"},
) as tg:
task1 = DatabricksNotebookOperator(
...
notebook_params={"param2": "{{ ds }}"},
)

The above code snippet works perfectly

But on rerun it goes back to the same issue .
Also dag_run attributes are not working as expected , literal values gets passed to the DatabricksNotebookOperator unless they are defined in the DatabricksWorkflowTaskGroup.
But on rerun again the literal values are passed.

[a-baturin]

Unfortunately it's not completed
PR #46704 Fix repair inDatabricksWorkflowTaskGroup (probably, I did not test), not templates in DatabricksNotebookOperator