Detach C++ node tests from ONNX on-disk data/node artifacts (onnx#7959) + retiring equivalence proof + corpus-collapse tripwire

[titaiwangms]

Motivation

ONNX onnx/onnx#7959 removes the on-disk onnx/backend/test/data/node/ corpus (targeted for ONNX 1.23), replacing it with on-the-fly in-memory generation. ORT's C++ onnx_test_runner reads that corpus from disk — after the deletion it would load 0 node tests and silently exit 0 (green CI on a corpus that no longer exists). This PR detaches ORT from the deleted artifacts ahead of the bump.

What this does (3 pieces)

1. Detach — a build-time materializer (tools/python/materialize_onnx_node_tests.py) regenerates the node corpus to disk from ONNX's surviving generator (collect_testcases), so the C++ runner keeps reading from disk unchanged. EP-agnostic (CPU/CUDA/QNN share one materialized tree).
2. Equivalence proof — a retiring test (onnxruntime/test/python/onnx_node_test_equivalence_test.py + tools/python/compare_node_test_corpora.py) proves the materialized corpus is byte-identical to the original (modulo a documented ULP band); it auto-skips once the ONNX on-disk oracle disappears post-Add transformers tool test to pipeline #7959.
3. Cause-agnostic tripwire — build-time --min-cases gate (FATALs the build on every MATERIALIZE=ON leg if generation < floor) + a runtime -m floor on the CPU node ctest, turning silent-green-on-empty into a loud red.

Key decisions

• numpy: cmake configure = HARD FATAL on off-pin numpy (build-env reproducibility across CI legs); Python materializer = SOFT WARN (standalone advisory). Different layers, different questions. The onnx pin stays HARD on both sides.
• QNN legs: repointed to the materialized tree + pinned onnx/numpy installed pre-build. Node-dir runs carry a low -m 1 collapse sentinel (NOT the 1500 build floor): -e qnn legitimately reduces the collected set to ~1529, so a 1500 runtime floor would be a false-red timebomb on the next opset bump — the strict count is enforced at build time via --min-cases, while -m 1 (zero false-red risk) catches a per-leg MATERIALIZE=OFF that still runs the node dir. The android leg's single-case cp -r source is repointed to the materialized tree (the old data/node source vanishes post-Add transformers tool test to pipeline #7959).

Testing

• Runtime tripwire empirically verified: empty/truncated corpus → runner FATAL (nonzero); full → exit 0; default (no -m) unchanged.
• Equivalence: 1799-case byte-probe (dir-set match; byte-identical modulo the documented ULP band).
• cmake configure verified to generate cleanly on latest main (ep_context + node-test regions coexist, no collisions).
• [needs-run] on a pinned-numpy CI box: the MATERIALIZE=ON inner path (actual materialization + the two add_test bodies) — this dev box is off-pin so the numpy gate FATALs by design.

Relates to: onnx/onnx#7959

---

Cross-consumer viability note (re: onnx/onnx#7959)

While detaching, we assessed all of ORT's node-test consumers. The in-memory generator approach behind #7959 works cleanly for single-version consumers (C++, C#, docs), but has a gap for consumers needing a historical multi-opset matrix:

• ORT's JS/web tests pull node data for opset 7–21 from 15 immutable rel-* release archives. Since onnx's generator is single-version, old opsets can't be regenerated from a new onnx.
• Those consumers stay green post-Add transformers tool test to pipeline #7959 only because released branches are immutable — there is no generator-based path to add a new post-Add transformers tool test to pipeline #7959 opset for them.

Net: workable for single-version consumers; the multi-version case would need a small per-version serialization utility or a documented migration note upstream.

[Copilot]

Pull request overview

This PR prepares ONNX Runtime’s C++ ONNX node-test coverage for ONNX PR onnx/onnx#7959, which removes the on-disk onnx/backend/test/data/node/ corpus. It introduces a build-time materialization step to regenerate the node-test artifacts to disk (so onnx_test_runner can keep consuming a directory), adds an equivalence oracle that auto-retires when the upstream oracle disappears, and adds “silent-green” collapse tripwires (build-time and run-time) across CMake, C++ runner, and Python test discovery.

Changes:

• Add Python tooling to materialize ONNX node tests from ONNX generators and (optionally) byte-compare vs the pre-#7959 on-disk oracle.
• Add CMake option + build/test wiring to generate the node corpus into the build tree, run it via onnx_test_runner, and (while available) run an equivalence ctest.
• Add “minimum discovered cases” tripwires to prevent silent pass on empty corpora; update QNN CI legs and Linux CI to consume the materialized tree.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/python/materialize_onnx_node_tests.py	New materializer + oracle comparator core for node-test corpus regeneration and equivalence classification.
tools/python/compare_node_test_corpora.py	Build-wired wrapper to compare an already-materialized corpus vs the on-disk ONNX oracle.
tools/ci_build/github/azure-pipelines/win-qnn-arm64-ci-pipeline.yml	Installs pinned onnx/numpy, enables materialization, repoints QNN node-test run to materialized tree with -m 1 sentinel.
tools/ci_build/github/azure-pipelines/linux-qnn-ci-pipeline.yml	Same as Windows QNN: install deps, enable materialization, repoint node-test run + sentinel floor.
tools/ci_build/github/azure-pipelines/android-arm64-v8a-QNN-crosscompile-ci-pipeline.yml	Enables materialization on host; repoints the single copied node test to materialized tree.
onnxruntime/test/python/onnx_node_test_equivalence_test.py	New retiring Python unittest to materialize into a tmpdir and compare vs oracle when present.
onnxruntime/test/python/onnx_backend_test_series.py	Adds a Python discovery-count floor to fail loud if node tests disappear from disk.
onnxruntime/test/onnx/main.cc	Adds -m minimum-tests option to onnx_test_runner to prevent silent-green on empty dirs.
cmake/onnxruntime_unittests.cmake	Adds onnxruntime_MATERIALIZE_ONNX_NODE_TESTS wiring: configure-time dependency gating, build-time materialization target, ctests, and oracle retirement.
cmake/CMakeLists.txt	Introduces the new opt-in dependent CMake option for node-test materialization.
.github/workflows/windows_qnn_x64.yml	Installs pinned onnx/numpy, enables materialization, repoints node-test run to materialized tree + -m 1 sentinel.
.github/workflows/linux_ci.yml	Enables materialization on a Linux CPU CI leg to exercise the build/test wiring.
.agents/skills/onnx-opset-bump-checklist/SKILL.md	Documents the onnx#7959 “silent-green” landmine and mitigation guidance in the opset bump checklist.

[titaiwangms]

Review synthesis (multi-model review team)

Strong, well-engineered PR. An end-to-end run of the materializer confirmed 1799 cases, byte-identical to the on-disk oracle, the 6 inverse-trig ULP cases reproduced exactly, and the writer is a faithful verbatim transcription of ONNX's cmd_tools.py::generate_data. One CI-breaking bug should be fixed before merge.

🔴 Critical (verified against ONNX source — blocks merge)

onnxruntime/test/python/onnx_backend_test_series.py — the new MIN_NODE_CASES gate fires on every full run.
ONNX's Runner calls _add_model_test(rt, "Node") with a capital "N" (onnx/backend/test/runner/__init__.py:82), but the override checks if kind == "node" (lowercase). _node_case_count therefore stays 0, so 0 < 1500 raises RuntimeError("discovered only 0 …") on every full backend-series run.
Fix: check model_test.kind == "node" (the TestCase.kind attribute is lowercase), or compare case-insensitively.

🟠 Major

1. Unwired consumers still read the on-disk data/node corpus — the PR repoints the QNN/CPU legs but misses:
   • csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests/runtest.sh:25 — symlinks data/node; becomes a dangling symlink → silent test skip once the pin passes Add transformers tool test to pipeline #7959.
   • js/scripts/prepare-onnx-node-tests.ts:53 — silently extracts nothing for a post-Add transformers tool test to pipeline #7959 opset version.
   • docs/python/conf.py:124 — hardcoded master URL will 404 the Sphinx docs build after Add transformers tool test to pipeline #7959 merges.
     Since the PR's goal is to detach ahead of the bump, these are in-scope gaps (they only trigger once the pin actually advances, but leave the detach incomplete).
2. Post-retirement schema-drift is caught by nothing. Once the equivalence oracle retires (post-Add transformers tool test to pipeline #7959), only the version-string gate + the count floor remain — neither detects a wheel whose compiled schema registry bakes a wrong opset (string matches, count unchanged). Acceptable given CI pip-pin discipline, but worth stating explicitly, or adding a cheap since_version cross-check anchor.

🟡 Minor

• min_cases checked before the kind == "node" filter (tools/python/materialize_onnx_node_tests.py) — the floor validates the collected count, not the written node count. Benign for 1.22.0 (all 1799 are node), but violates the guard's own invariant; check len(written).
• CMake STREQUAL vs Python _release_base — CMake requires an exact onnx version-string match and FATALs before the script runs, while the Python side intentionally accepts rcN/.dev of the same release base. Inconsistent policy on opset-bump / pre-release workflows.
• The 1500 floor is triple-duplicated — CMake single-sources its own 3 consumers from ORT_ONNX_NODE_MIN_CASES, but the two Python MIN_NODE_CASES literals are independent, coupled only by a comment. "Move in lockstep" is a convention, not a mechanism.
• _max_ulp_diff float16/bfloat16 branch — the uint16 sign-remap is dead code (values are always ≥0), and bfloat16 never reaches it (np.issubdtype(ml_dtypes.bfloat16, np.floating) is False → classified Class B first). Safe direction, but the docstring claiming bfloat16 is ULP-handled is misleading.
• Stale-corpus risk on incremental builds — add_custom_command DEPENDS only on the script + cmake/deps.txt, not the onnx wheel; reinstalling a different wheel without touching deps.txt leaves .stamp fresh and reuses the old corpus. Low impact in clean-build CI.

Nits

• ctest DEPENDS uses a target name (not a test name → no real runtime ordering; benign since the target is ALL).
• Dead _onnx_url_ver variable (CMAKE_MATCH_1 is used directly).
• C++ FATAL message cites "gotcha (p)" without the file path — add .agents/skills/onnx-opset-bump-checklist/SKILL.md.
• Duplicated "do not rename" comments on the input/output loops.
• Hardcoded onnx==1.22.0 / numpy pins in the ymls are drift-prone vs cmake/deps.txt + requirements.txt (verified they agree today).

---

Recommendation: fix the Critical ("Node" capitalization) — a one-liner that unblocks the Python leg — and address the Major unwired-consumer gaps (at least C# + the docs 404), since they undercut the "detach ahead of the bump" goal. The rest are polish/hardening.

_{🤖 Generated with multi-model review (readability / code / critical / deep / integration).}

[titaiwangms]

Re-review of the update

Verified the pushed fixes against source (and empirically for the numerical one). The blocking Critical is resolved and the core fixes are correct.

✅ Verified fixed

• 🔴 Critical — "Node" capitalization: now kind.lower() == "node" (onnx_backend_test_series.py:83). ONNX's Runner passes the capitalized display category "Node", so the old == "node" kept the counter at 0 and would have raised RuntimeError("discovered only 0") on every full backend-series run. Fixed.
• 🟡 min_cases counts written node cases: floor moved after the kind != "node" filter → if len(written) < min_cases (materialize_onnx_node_tests.py:279).
• 🟡 CMake onnx gate is now RC/.dev-aware: regex-strips to X.Y.Z base, mirroring the Python _release_base (onnxruntime_unittests.cmake:1585) — no longer stricter than the materializer.
• 🟡 ULP signed-zero / overflow: reworked to an unsigned sign-magnitude fold in Python-int (object) domain. Empirically confirmed: ±0.0 → 0 ULP (fp32 & fp64), adjacent → 1 ULP, neg/pos crossing → 2 ULP, bfloat16 → out-of-band sentinel 5 (>4 ⇒ forced Class B). Correct.
• Bonus follow-through (TestCase.cc): the new CPU onnx_test_node_materialized ctest now runs node tests on the CPU EP (previously QNN-only), so convinteger_with_padding + dft_(i)rfft* are correctly hoisted into the common GetBrokenTests initializer, mirroring the Python filter exclusions. Sound.

⚠️ Detach completeness — 3 non-C++ consumers still read the on-disk data/node (for a follow-up)

The C++ detach (the PR's stated scope) is complete and well-guarded. These remaining on-disk consumers were not migrated and leave the same silent-green landmine on other language legs. All are latent (fire only once the onnx pin/submodule/master actually passes #7959), so not blockers for this PR — flagging for a follow-up:

Consumer	Fires when	Impact
docs/python/conf.py:124	onnx master merges #7959 (independent of ORT's pin)	URL hardcodes master; urllib.request.urlretrieve on a 404 raises HTTPError and is not wrapped in try/except → Sphinx docs build hard-fails. Fires soonest — recommend pinning the URL to a fixed tag even in this PR.
csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests/runtest.sh:25	cmake/external/onnx submodule advances past #7959	macOS NuGet EndToEnd leg only (if IsMacOS); the data/node symlink goes dangling → no opset* dir → silent-green, zero C# model tests.
js/scripts/prepare-onnx-node-tests.ts:53	a post-#7959 opset is added to TEST_DATA_OPSET_VERSIONS	Extracts nothing from the archive → silent JS node-test coverage loss.

Remaining minors (non-blocking)

• Post-retirement, opset fidelity rests entirely on pip-pin discipline (version string + count floor; no executable schema-drift check once the equivalence oracle retires). Worth a one-line note.
• The 1500 floor is still triple-defined (CMake single-sources its own 3 consumers, but the two Python MIN_NODE_CASES literals are coupled only by comment).

Net: good to go for the C++ objective once the above follow-ups are tracked. Nice work on the fixes.

_{🤖 Multi-model review re-pass.}

[titaiwangms]

Thanks for the review! Dispositions:

• docs/python/conf.py — fixed in this PR: pinned the test_sigmoid model URL from onnx master to v1.22.0 (matching deps.txt), so the docs build stays green regardless of when Add transformers tool test to pipeline #7959 lands on onnx master (it fetches GitHub master directly, independent of our submodule pin).
• csharp EndToEndTests runtest.sh — analogous to the C++ detachment (single pinned opset, reuses the materializer); handling it in a stacked follow-up PR so it lands before the onnx->1.23 submodule bump, keeping this PR focused on the C++ objective.
• js prepare-onnx-node-tests.ts — deferred by design: it consumes 15 historical frozen opset archives from immutable rel-* branches, which Add transformers tool test to pipeline #7959 does not rewrite, so it stays green indefinitely; it only breaks if a post-Add transformers tool test to pipeline #7959 opset is manually added, which needs a different multi-version mechanism.
• Post-retirement opset fidelity rests on the pinned onnx version + the min-count tripwire once the equivalence oracle auto-retires.

Cross-consumer note for the #7959 viability question: the in-memory generator approach works cleanly for single-version consumers (C++, csharp, docs), but not for consumers needing a historical multi-opset matrix (our JS tests span opset 7-21), since onnx's generator is single-version and can't regenerate old opsets from a new onnx.

[titaiwangms]

Review team re-review (round 2, current state)

Ran the full 5-reviewer team (readability / code / critical / deep / integration) against the latest state. All round-1 fixes confirmed resolved:

• ✅ kind.lower() == "node" — verified against ONNX runner/__init__.py:82 (ONNX passes capitalized "Node"), so the counter now actually increments. This is a real correctness fix, not cosmetic.
• ✅ min_cases now derived from len(written) (post kind-filter).
• ✅ _max_ulp_diff is a correct ULP metric (matches the canonical Dawson formulation up to the additive signmask constant; ±0→0, adjacent→1, cross-sign→2).
• ✅ CMake RC/.dev regex effectively mirrors _release_base.
• ✅ TestCase.cc broken-list move drops no coverage — the node corpus was never exercised on non-QNN EPs before; CPU now runs it via the new ctest, so the global skip is required and mirrors the jsonc exclusions.

Still-open / new findings (none are merge-blockers)

🟠 Major — ULP classifier accepts finite↔∞ and ∞↔NaN as "Class A" (materialize_onnx_node_tests.py::_classify_tensor_diff).
Verified: max-finite fp32 vs +inf classifies as ('A', '1 ULP, max|d|=inf'). Raw-bit adjacency makes max-finite↔+inf = 1 ULP and +inf↔NaN(0x7f800001) = 1 ULP, so a structural finite→∞ (or ∞→NaN) output drift is silently blessed by the equivalence oracle despite max|d|=inf. Low real-world likelihood (retiring test, requires an actual finite/inf drift), but a valid robustness hole.
Cheap fix: in the Class-A branch, reject when np.isfinite(a) != np.isfinite(b), the NaN class differs, or max_abs is non-finite.

🟡 Minor — signed-zero false-RED. A pure +0/-0 diff hits np.array_equal(+0,-0) == True and returns Class B ("proto encoding drift") before ULP can collapse it, contradicting the ULP signed-zero doc claim.

🟡 Minor — stale IRFFT reason in onnx_backend_test_series_filters.jsonc:383-385. It still says ORT "does not implement IRFFT … outputs complex last dim=2", which is provably false per core/providers/cpu/signal/dft.cc:452-459 (IRFFT sets last dim=1). TestCase.cc was corrected and now claims "parity with jsonc" — but the jsonc twin still carries the old wrong reason. Please sync it.

🟡 Minor — two more residual on-disk data/node consumers not covered by this PR:

• .agents/skills/onnx-opset-bump-checklist/SKILL.md:471 — the checklist's own verification gauntlet runs onnx_test_runner -e cpu …/_deps/onnx-src/…/data/node, which breaks post-Add transformers tool test to pipeline #7959 (ironically added by this same PR).
• docs/python/examples/plot_load_and_predict.py:16 — blob/main/…/test_sigmoid link → 404 after retirement.

Pin/repoint both to the materialized dir or a tagged ONNX ref.

🟡 Minor — 1500 floor is now quadruple-defined (a 4th copy landed in onnx_node_test_equivalence_test.py), lockstep enforced only by comment with no cross-reference. ctest DEPENDS onnx_node_tests_materialized uses a build-target name in a CTest test-DEPENDS slot (benign since the target is ALL, but no real runtime ordering).

🔵 Nits — the kind.lower() rationale comment lives in __init__ rather than at the call site; the FATAL floor message omits the SKILL.md path.

Bottom line

No merge-blocker. The one item worth fixing here is the ULP finite/∞/NaN classifier gap (cheap, valid), plus the jsonc IRFFT reason-sync and the two extra doc/skill consumers.

[gramalingam]

Is the utility for materializing the node test cases worth upstreaming to onnx repo? (separately, not to block this PR.)

[tianleiwu]

Reviewed the node-test detach + materializer + tripwire. Overall this is a careful, well-documented change and the design (materialize from ONNX's surviving Python generators, keep the C++ runner reading from disk, add a cause-agnostic min-count tripwire, auto-retire the equivalence oracle post-#7959) is sound. The earlier automated-review findings look addressed on this head: the per-dataset arity guard in _write_case, the overflow-proof unsigned sign-magnitude fold in _max_ulp_diff, and the RC/.dev-aware release-base comparison in the cmake onnx gate are all present.

Two coupled points on the new CPU node ctest are worth confirming before merge (both left inline):

1. New CPU full-corpus coverage / broken-test parity. onnx_test_node_materialized runs the entire data/node corpus on the CPU EP for the first time. onnx_test_runner hard-fails on any case that fails on CPU but isn't in the C++ GetBrokenTests list, and this PR mirrors only convinteger + the 4 dft_* cases, while the Python onnx_backend_test_series_filters.jsonc excludes a much larger set.

2. -m floor vs. skipped-case count. The -m 1500 floor is checked against tests.size(), which excludes broken/disabled/training-domain cases, so the real headroom is smaller than the ~1799 -> 1500 reasoning implies, and the C++ and Python floors measure different populations.

The MATERIALIZE=ON py314 leg runs mode: test, so both would surface as CI-red rather than silently — but the PR description's own [needs-run] note says the MATERIALIZE=ON inner path (materialization + the two add_test bodies) hasn't executed on a pinned box yet. Running that leg's ctests once on-pin would de-risk the merge.

Minor / non-blocking: the numpy pin (2.2.6/2.4.2) is now duplicated across cmake, the 4 CI yaml files, and requirements.txt; the SKILL.md documents the lockstep requirement, but it remains three manual sync points.

[titaiwangms]

cc @tianleiwu

[tianleiwu]

Re-reviewed the current head (381ccd6). This is a thorough, well-documented change and my three earlier threads are now addressed — resolved them:

• python 3.10 literal: agreed it's a find_package lower bound, not a hard version, so 3.10 legs aren't blocked.
• CPU onnx_test_node_materialized broken-test parity: confirmed green on-pin (run 28893847416); future CPU-failing node tests surface as loud CI-red on the same leg.
• ORT_ONNX_NODE_MIN_CASES floor: the tightened comment now documents the two intentionally-different populations and the safe failure direction.

The materializer/compare core, the retiring equivalence guards (Python unittest + compare_node_test_corpora.py ctest), the configure-time onnx/numpy gates, and the -m runtime tripwire all look correct and internally consistent. One minor non-blocking nitpick left inline. LGTM once CI is green.

[titaiwangms]

I will address comments with another PR.