Document supported databricks_retry_args usage for deferrable Databricks operators

[kosiew]

• Part of Callable deserialization causing error with DatabricksExecutionTrigger #64609

What does this PR do?

Follow up for #64960

Adds documentation clarifying which databricks_retry_args configurations are supported when using Databricks operators in deferrable mode.

The new documentation:

• Explains that databricks_retry_args must be serialization-safe because it is serialized across the trigger boundary when deferrable=True.
• Documents supported value types (plain Python primitives and collections of primitives).
• Provides examples of supported configurations, such as {"reraise": True}.
• Explains that retry count and delay should be configured through the dedicated retry_limit and retry_delay operator parameters.
• Documents unsupported configurations, including Tenacity strategy objects (stop_after_attempt, wait_incrementing, etc.) and arbitrary callables.
• Recommends using the non-deferrable Databricks operators when custom callable retry strategies are required.
• Adds a Databricks provider changelog entry describing the documentation update.

Does this PR introduce any user-facing change?

Yes. Documentation now explicitly describes the serialization requirements and supported shapes for databricks_retry_args in deferrable Databricks operators, helping users avoid unsupported retry configurations.

How was this patch tested?

No tests were added or modified. This PR contains documentation and changelog updates only.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  ChatGPT

[eladkal]

cc @moomindani for review

[moomindani]

Thanks for the follow-up — this is exactly the docs fix scoped at the end of the #64960 thread, and the structure of the two new sections is clean (heading levels, anchors, and code blocks all check out). However, two of the content claims do not match what the merged code actually does, and two files should not be in the diff:

1. The "Supported (serialization-safe and runtime-valid)" example {"reraise": True} is not runtime-valid: BaseDatabricksHook replaces (does not merge) the default retry_args, so stop/wait are lost and tenacity falls back to stop_never + wait_none() — an infinite, zero-backoff retry loop. Setting databricks_retry_args at all also makes retry_limit/retry_delay no-ops. Since tenacity strategy objects (the only way to set stop/wait inside retry_args) are exactly what deferrable mode rejects, the honest guidance is "do not use databricks_retry_args in deferrable mode; use retry_limit/retry_delay", rather than a supported example.
2. "will raise ValueError at task submission" overstates the timing: validation runs in the trigger constructors at defer time, after submit_run/run_now has already launched the Databricks run. (The #64960 thread did say "before any Databricks API call", but re-checking the merged code, the operator path submits first and defers second — operators/databricks.py:834-836.)
3. generated/provider_dependencies.json.sha256sum is a stale-base rebase artifact — please rebase onto current main and drop it.
4. The hand-added changelog entry should be removed (release-manager-maintained file; see inline).

Details inline.

---

Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting

[moomindani]

Small correction to one point in my review: generated/provider_dependencies.json.sha256sum was re-added to main by #68775 a few hours before I posted (the same accidental re-add pattern as #67080, after #68801 had removed it), so "a file that no longer exists upstream" is no longer accurate. The requested action is unchanged: rebase onto current main and keep this file out of this PR's diff.

---

Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting