[SPARK-56666][INFRA] Reduce unidoc CI log noise with -Xdoclint:-missing and -verbose post-filter

[cloud-fan]

JIRA: https://issues.apache.org/jira/browse/SPARK-56666

What changes were proposed in this pull request?

Refines the unidoc javacOptions in JavaUnidoc / unidoc / javacOptions and the post-process stream filter in docs/_plugins/build_api_docs.rb so that the Documentation generation CI log is small enough to scan visually while still surfacing per-file error: reference not found diagnostics on broken {@link} references.

Builds on the -Xmaxerrs and -verbose insight from #55581 (SPARK-56630 follow-up): javadoc's default -Xmaxerrs 100 cap was hit by the ~100 inert genjavadoc-stub errors during source loading, so doclint never ran on the real sources, and the per-file error: reference not found diagnostics surfaced only with -verbose. That PR's flag set (-Xmaxerrs 999999, -Xmaxwarns 999999, -verbose) achieved the diagnostic goal but at a ~77K-line CI log per run.

This PR keeps the diagnostic visibility and brings the visible CI log down to ~4K lines (95% reduction), with four changes:

1. -Xmaxerrs 0 instead of -Xmaxerrs 999999. The 0 value is treated as unlimited by javadoc (locally verified) and reads cleaner than the magic number.

2. -Xdoclint:all + -Xdoclint:-missing (two separate flags, matching the existing Compile / doc / javacOptions pattern in SparkBuild.scala). Suppresses the missing doclint group at javadoc level: the ~22K no comment / no @param / no @return / no @throws warnings (each rendered as a 3-line block) that dominate the log on every Spark unidoc run. The two-flag form is load-bearing — bare -Xdoclint:-missing alone demotes other doclint groups (notably reference) to warning level, making broken {@link} non-fatal; the explicit -Xdoclint:all first keeps reference at error level. Locally verified.

3. Drop -Xmaxwarns 999999. Warnings don't fail CI; error visibility is governed by -Xmaxerrs, not -Xmaxwarns. javadoc's default cap of 100 is sufficient — shows a sample of any remaining warnings without flooding. Saves ~4K lines beyond -Xdoclint:-missing alone.

4. Post-filter -verbose progress lines from the build_api_docs.rb stream. -verbose itself stays (it is load-bearing for per-file error: reference not found emission per [SPARK-56630][INFRA][FOLLOWUP] Make unidoc surface real javadoc failures #55581), but its progress noise — Loading source file ..., [parsing started/completed], [loading /path/X.class], Generating /path/X.html — carries no diagnostic signal. The existing stream filter is extended with a verbose_line regex that drops these single-line progress entries from stdout. Saves ~13K lines.

Why are the changes needed?

Documentation generation CI logs were ~77K lines per run after SPARK-56630's flag set. That is large enough that scanning for diagnostics by eye is impractical, and grep-piping is the only reasonable workflow. Most of the volume is structural noise (genjavadoc stub errors, no comment warnings, -verbose progress markers) with no diagnostic signal. After this PR the log is ~4K lines on a real-failure run; the per-file error: reference not found diagnostics PR #55581 added are the dominant content.

Empirical breakdown of the reduction (verified end-to-end on this branch's earlier test commits with deliberately broken {@link} plants in both a real .java source and a Scala source):

State	Log lines	Vs baseline
PR #55581's flag set (baseline)	77K
Add -Xdoclint:all,-missing	22K	-71%
Drop -Xmaxwarns 999999	18K	-77%
Post-filter -verbose progress	~4K	-95%

All diagnostic targets remain visible: per-file error: reference not found for both Java sources and Scala sources via the genjavadoc stub.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Validated end-to-end on earlier (now reverted) test commits of this branch with planted broken {@link} references in both code paths:

• ColumnarMap.java (real Java source): {@link org.apache.spark.deliberately.NoSuchClass} and {@link ColumnVector#nonExistentMethod()}.
• Partition.scala (Scala source via genjavadoc): [[Partition.index]] — the wrong . separator that javadoc reads as inner-class lookup and fails to resolve. This is the case PR [SPARK-56630][INFRA][FOLLOWUP] Make unidoc surface real javadoc failures #55581's AGENTS.md note documents as the most common scaladoc-side cause of unidoc failure.

Both surfaced as per-file error: reference not found diagnostics in the CI log on the test commit, doc gen failed as expected, log size dropped to 3,977 lines, and zero Loading source file / [parsing started] / [loading X.class] / Generating *.html / no comment lines remained visible. See the test-result comment below for the full breakdown.

-Xmaxerrs 0 and the bare--Xdoclint:-missing demotion behavior were verified locally with standalone javadoc invocations on a minimal test file.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

[cloud-fan]

End-to-end test result

Validated on earlier branch commit a2bbe807909 with three deliberately broken {@link} references planted across one Java source and one Scala source (later reverted). The Documentation generation CI job ran and produced this log:

Doc gen status: FAILURE ✅ (broken refs are fatal)

Per-file diagnostics — all 4 visible:

[warn] /__w/spark/spark/core/target/java/org/apache/spark/Partition.java:3: error: reference not found
[warn] /__w/spark/spark/core/target/java/org/apache/spark/Partition.java:5: error: reference not found
[warn] /__w/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarMap.java:26: error: reference not found
[warn] /__w/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarMap.java:27: error: reference not found

The Scala plant [[Partition.index]] (with the wrong . separator) was correctly translated by genjavadoc into a stub {@link Partition.index}, and javadoc surfaced the resolution failure on the stub path core/target/java/org/apache/spark/Partition.java — exactly the failure mode #55581's AGENTS.md note documents.

Log composition:

Pattern	Count
Total lines	3,977
Loading source file	1 (a wrapped Unexpected javac output: Loading source file ... from sbt — different prefix, unfilterable)
[parsing started] / [parsing completed]	0
[loading X.class]	0
Generating *.html	0
no comment / no @param / no @return / no @throws	0 each
Final summary	4 errors / 2,408 warnings. (default 100 cap on warning printing)

Reduction journey:

State	Log lines	Vs baseline
#55581's full flag set (-Xmaxerrs 999999 -Xmaxwarns 999999 -verbose)	77K	baseline
+ -Xdoclint:all -Xdoclint:-missing	22K	-71%
+ drop -Xmaxwarns 999999	18K	-77%
+ post-filter -verbose progress lines	3,977	-95%

The current CI run on this PR's clean state (no plants) is expected to pass.

[cloud-fan]

The timeout issue is unrelated, thanks for review, merging to master!