[SPARK-56630][INFRA] Surface javadoc crash culprit in unidoc failure output

[cloud-fan]

What changes were proposed in this pull request?

Adds a diagnostic banner to the unidoc step in docs/_plugins/build_api_docs.rb. When build/sbt unidoc fails, the script now scans the captured sbt output and prints a framed summary naming the <Class>.html javadoc was generating when it died, the inferred source class to audit, and a one-paragraph hint about the usual scaladoc triggers.

Implementation:

• stream_and_capture tees sbt output to both stdout and target/unidoc-build.log (Ruby-only, no shell pipefail reliance).
• diagnose_unidoc_failure finds the last Generating .../<Class>.html... line before javadoc exited with exit code N and prints a culprit-pointer banner. ANSI colour codes are stripped before regex matching.
• When the failure mode doesn't match the mid-HTML-crash pattern (e.g. scaladoc failure, sbt env issue), the banner says so and points back to the full log.

Why are the changes needed?

Today, when javadoc hard-exits during unidoc HTML generation -- typically because of a specific scaladoc construct (e.g. wiki-style [[Class]] links or backtick-inline code refs) in an exposed Scala source -- the failing PR's CI log shows ~100 [error] lines on target/java/... files. Those errors are benign: they're genjavadoc-emitted Java stubs (static public abstract R apply(T1, T2, T3, T4)) that every PR produces, and javadoc always complains about them but normally still finishes. They are not the cause of the failure.

The actual signal is the last Generating .../<Class>.html... line before javadoc exited with exit code 1, which a developer has to find by hand in a multi-thousand-line log. The error reporting does not differentiate the benign noise from the real crash, so the failure consistently looks like it's "in" ErrorInfo.java / LexicalThreadLocal.java / similar, when it's actually in a Scala source that none of those names point to.

A recent example: PR #51419 hit this exact misdirection -- the log was full of errors on common/utils/target/java/... stubs, but the real culprit was a doc comment in CatalogV2Implicits.IdentifierHelper that triggered a hard exit during HTML generation. The diagnostic in this PR would have named that class directly.

Does this PR introduce any user-facing change?

No. CI-only output change visible in the unidoc step of the doc-gen job.

How was this patch tested?

• Dry-ran the parser logic against the captured failing log from PR [SPARK-52729][SQL] Add MetadataOnlyTable and CREATE/ALTER VIEW support for DS v2 catalogs #51419 -- it correctly extracts org/apache/spark/sql/connector/catalog/CatalogV2Implicits.IdentifierHelper.html as the crash class.
• The second commit on this branch (DO NOT MERGE: break a docstring to validate the unidoc diagnostic) intentionally reintroduces the same [[...]]+backtick-inline scaladoc pattern in CatalogV2Implicits.IdentifierHelper.asTableIdentifierOpt so that this PR's CI run actually exercises the new path. Once the banner fires and names that class on the failing CI run, that commit will be dropped from this PR.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

[cloud-fan]

The bait commit on this branch (a4b30e83a3d — DO NOT MERGE: break a docstring) successfully exercised the new diagnostic in CI on the latest run. Captured banner from the failing Run / Documentation generation job (log):

==============================================================================
Unidoc failed -- diagnostic summary
==============================================================================

  Javadoc crashed while generating: org/apache/spark/sql/connector/catalog/CatalogV2Implicits.IdentifierHelper.html
  Likely culprit: doc comment in org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper

  Javadoc can hard-exit (not just warn) on specific scaladoc
  patterns once they have been passed through genjavadoc --
  wiki-style `[[Class]]` / `[[method]]` links or inline-backticked
  code refs in the Scala source for the class above are common
  triggers. Start by auditing any recent doc-string changes in
  that source file.

  NOTE: the '[error]' lines above on files under
  target/java/... are benign genjavadoc stubs -- every PR
  emits them and they do not cause the exit. Ignore them.
==============================================================================

The banner correctly named CatalogV2Implicits.IdentifierHelper — the exact class the bait commit broke — instead of leaving the reader to scroll past ~100 benign [error] lines on target/java/... stubs. Reverting the bait commit next.

[HyukjinKwon]

Merged to master.