[Core] Partition parallel CPU DAGs into streams

[GopalakrishnanN]

Description

Automatically partition independent CPU paths into multiple logical streams for unconfigured ORT_PARALLEL sessions.

• Compute downstream CPU depth in reverse topological order and keep each node's deepest successor on the same stream.
• Open streams at independent roots and fan-outs, bounded by the configured inter-op pool capacity.
• Assign additional work to the least-loaded CPU stream once the cap is reached.
• Preserve sequential execution, explicit node partition files, and existing non-CPU device grouping.
• Use node indices directly for generated partitions so unnamed or duplicate node names do not affect stream assignment.
• Expose the actual thread-pool worker count instead of using the hybrid-CPU-inflated DegreeOfParallelism value as the stream cap.

The existing execution planner already inserts BarrierStep and TriggerDownstreamStep synchronization for cross-stream dependencies, so no executor changes are required.

Motivation and Context

The default DeviceBasedPartitioner groups nodes by OrtDevice::DeviceType. Consequently, an all-CPU graph receives one logical stream even in ORT_PARALLEL, serializing independent branches unless users provide a hand-authored session.node_partition_config_file.

This change exposes that branch parallelism automatically while retaining one stream for linear graphs.

Performance

Release CPU build, ORT_PARALLEL, intra-op=1, inter-op=4. Each condition used eight randomized process rounds with 80 iterations, the first five discarded, for 600 retained measurements. Confidence intervals use 20,000 bootstrap samples over process medians. The reference is an explicit single-stream configuration using the same binary.

DAG	Single stream	Automatic	Median speedup	95% CI
Balanced	1.767 ms	1.023 ms	1.726x	1.687-1.764
Imbalanced	1.777 ms	1.203 ms	1.477x	1.424-1.558
Repository SSD	167.745 ms	125.791 ms	1.334x	1.273-1.393
Linear control	1.707 ms	1.700 ms	1.004x	0.994-1.027

The SSD P95 improved from 208.362 ms to 157.970 ms.

Relationship to #29700

This is complementary to #29700. This PR changes graph planning to expose parallel CPU work; #29700 changes executor waiting and cancellation after streams have been planned. There is no changed-file overlap. The measurements above use the current executor and should be repeated after #29700 lands, particularly under high request concurrency.

Validation

• Release onnxruntime_test_all built successfully after rebasing onto current main.
• PlannerTest.*: 16 tests passed.
• The branch-heavy integration test compares sequential and parallel output sizes and full-output 128-bit hashes.
• Explicit partition configuration and linear-graph controls pass.
• Lintrunner/clang-format and git diff --check are clean.

Build configurations

Automatic multi-stream partitioning is limited to inference builds. Training-enabled builds (--enable_training) compute an additional memory-optimization plan (node_execution_order_in_training / CalculateProgramCounter in allocation_planner.cc) that assumes a single logical CPU stream, so the multi-stream branch is compiled out under ENABLE_TRAINING and those builds keep the original single-stream behavior. Without this gate, branch-heavy models such as SSD-MobileNetV1 trip the ProgramCounter::AddEnd assertion (sequential_execution_plan.h:60, "No matching 'start' entry") during session initialization, which previously failed the training CI leg. Reproduced and fixed against a local --enable_training build: pre-fix, SSD-MobileNetV1 at inter-op=8 fails with that exact assertion; post-fix it initializes and runs for inter-op in {2, 4, 6, 8, 12, 24}.

[Copilot]

Pull request overview

This PR enhances ONNX Runtime’s stream planning for ORT_PARALLEL sessions by automatically partitioning independent CPU-only DAG branches into multiple logical streams (while keeping linear CPU graphs on a single stream), without requiring a user-provided session.node_partition_config_file.

Changes:

• Extend the sequential-planner context to carry a max_num_streams cap and pass it into graph partitioning.
• Update the default DeviceBasedPartitioner to topology-aware split CPU paths across streams (bounded by inter-op concurrency), while preserving existing non-CPU device grouping and explicit partition config behavior.
• Add planner- and session-level tests validating automatic CPU branch partitioning and linear-graph behavior; add ThreadPool::WorkerThreadCount() to cap streams using actual worker threads (vs hybrid-inflated DegreeOfParallelism).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
onnxruntime/test/framework/allocation_planner_test.cc	Adds unit/integration coverage for CPU topology-aware partitioning, stream limits, and sequential-vs-parallel equivalence checks (size/hash).
onnxruntime/core/framework/session_state.cc	Computes and supplies max_num_streams to planner context based on inter-op thread pool worker count (+ caller).
onnxruntime/core/framework/allocation_planner.h	Adds GetMaxNumStreams() to planner context; extends IGraphPartitioner::PartitionGraph API to accept parallel/max-stream parameters.
onnxruntime/core/framework/allocation_planner.cc	Implements topology-aware CPU partitioning inside DeviceBasedPartitioner when auto-generating partitions.
onnxruntime/core/common/threadpool.cc	Implements ThreadPool::WorkerThreadCount() as NumThreads() (0 if null).
include/onnxruntime/core/platform/threadpool.h	Declares ThreadPool::WorkerThreadCount() API and documents semantics.

[GopalakrishnanN]

Good call. Relaxed the assertion to EXPECT_GT(parallel_result.num_streams, 1U) in 52830c5 so it validates the intended behavior (automatic >1 stream under ORT_PARALLEL) without pinning the exact count. The output size + 128-bit hash equivalence checks still guard correctness, so the test stays meaningful across benign graph/optimizer or future heuristic changes.

[xadupre]

0 or 1 if tp is null?