[FEAT] adds repair run functionality for databricks

[gaurav7261]

[FEAT] add repair run functionality in DatabricksRunNowOperator
This is useful in MULTI TASK mode, where operator will auto retry only failed tasks rather than relying on airflow retries that will create a new run

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[dirrao]

Can you add the test cases for the new functionality?

[gaurav7261]

sure @dirrao, adding them

[gaurav7261]

Hi @dirrao, added test cases to get latest repair id, I look for earlier test cases for method _handle_databricks_operator_execution, but seems like since it is a handle function, no test case is there, however added unit test case to get latest repair id and calling the repair run and recursive calling the same method. Let me know if that helps

[dirrao]

LGTM.
I would request data bricks expert to review this.

[eladkal]

cc @alexott

[pankajastro]

LGTM

[gaurav7261]

Hi @pankajastro , I got the issue, seems like I missed the comma in method definition

[gaurav7261]

Hi @pankajastro can you please review this

[gaurav7261]

Hi @dirrao , since repair run will work with DatabricksRunNowOperator only, test case was failing because same handle method was getting called for DatabricksSubmitRunOperator as well, can you please review again, can you share command to run all test locally if possible

[gaurav7261]

Hi @dirrao can you please rereview

[gaurav7261]

hi @dirrao @pankajastro , can you please approve for CI check

[dirrao]

Hi @dirrao , since repair run will work with DatabricksRunNowOperator only, test case was failing because same handle method was getting called for DatabricksSubmitRunOperator as well, can you please review again, can you share command to run all test locally if possible

Go through the below to setup the breeze.
https://github.com/apache/airflow/blob/main/BREEZE.rst

[dirrao]

LGTM

[gaurav7261]

Hi @dirrao , I think CI is pending

[gaurav7261]

Hi @dirrao , fixed static checks, can you please run CI

[potiuk]

Still some errors :( . I recommend installing pre-commit and running breeze static-checks --only-my-changes

[gaurav7261]

sure @potiuk , doing that thanks

[gaurav7261]

done @potiuk

[dirrao]

LGTM

[gaurav7261]

@dirrao can you approve so that CI will start cc @potiuk

[dirrao]

@dirrao can you approve so that CI will start cc @potiuk

@gaurav7261 I don't have permissions to trigger it. Let us wait for others to review and approve it.
CC @pankajastro / @potiuk

[gaurav7261]

fixing ci static cheque

[gaurav7261]

Hi @Taragolis i have fixed X|Y to int | None and also fixed the change that you mentioned, thanks for that, can you reapprove

[pankajastro]

I have approved ci 🤞

[gaurav7261]

@pankajastro should i remove log.warn to log.info ```Check if there are no deprecate log warn...........................................Failed

• hook id: python-no-log-warn
• exit code: 1

airflow/providers/databricks/operators/databricks.py:93: log.warn(```

[gaurav7261]

now only this error is there?

[gaurav7261]

@pankajastro goti it replaced log.warn from log.warning

[gaurav7261]

done @Taragolis , please trigger CI

[gaurav7261]

HI @pankajastro , can you trigger it, I have made changes as mentioned by @Taragolis

[gaurav7261]

Hi @Taragolis @pankajastro , comma miss was there log.warning( "%s but since repair run is set, repairing the run with all failed tasks", error_message, ), added it, please retrigger CI, it will pass hopefully 🤞

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.