rustc: Support `--jobs` options for limiting parallelism in various parts of the compiler by petrochenkov · Pull Request #159675 · rust-lang/rust

petrochenkov · 2026-07-21T17:30:18Z

--jobs-frontend for the frontend, --jobs-backend for the backend, --jobs-linker for the linker, and -j/--jobs for the overall limit.
See the added docs for more details.
Part of rust-lang/compiler-team#1005.

Any suspicious option combinations are prohibited for now.
r? @bjorn3 @Zoxc

rustbot · 2026-07-21T17:30:21Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

petrochenkov · 2026-07-21T17:32:45Z

+/// `None` means everything is single-threaded and synchronization can be disabled.
+#[derive(Clone, Copy)]
+pub struct Jobs {
+    pub frontend: Option<NonZero<u8>>,


Suggested change

pub frontend: Option<NonZero<u8>>,

pub frontend: Option<NonZero<usize>>,

Probably can convert to usize after the range validation in parse_jobs_one is done, all the places that want this value accept usize.

View changes since the review

usize is used here now.

petrochenkov · 2026-07-21T17:33:59Z

+            check_upper_limit(backend, opt_name);
+            if zno_parallel_backend {
+                early_dcx
+                    .early_fatal("cannot use both `--jobs-backend` and `-Zno-parallel-backend`");


I think we can remove -Zno-parallel-backend right away in this PR for simplicity, I don't expect it to be used as much as -Zthreads.

View changes since the review

-Zno-parallel-backend is removed now (or rather always reports an error saying to use --jobs-backend=1).

petrochenkov · 2026-07-21T17:35:31Z

        compiler.arg(input_file);

-        // Use a single thread for efficiency and a deterministic error message order
-        compiler.arg("-Zthreads=1");


This is already the default, so it's not necessary (and it caused some option conflicts).

View changes since the review

It isn't the default when setting the rust.parallel-frontend-threads bootstrap config to a different value, right?

I don't see any logic turning bootstrap's rust.parallel-frontend-threads into -Zthreads for tests.

Compiletest's own --parallel-frontend-threads flag adds -Zthreads to UI tests, but it's not enabled by default.

Looks like one of the CI jobs does pass that flag:

rust/src/ci/docker/scripts/optional-x86_64-gnu-parallel-frontend.sh

Line 15 in b803f36

--parallel-frontend-threads="${PARALLEL_FRONTEND_THREADS}" \

This is an optional CI job, so if you break it, the merge would still work. Still it would be best to not break it unnecessarily.

Due to things like this I'll probably have to change the --jobs options from Opt to Multi right away.

-Zthreads=1 -Zthreads=2 works right now, but --jobs 1 --jobs 2 does not, as currently implemented.
Cargo also prohibits it - error: the argument '--jobs <N>' cannot be used multiple times, so I initially wanted to reproduce this more conservative behavior.

This is an optional CI job, so if you break it, the merge would still work.

FWIW, making it non-optional is my next immediate task.

I've added //@ ignore-parallel-frontend to a couple of tests causing option conflicts, test runs with --parallel-frontend-threads pass now.

bjorn3 · 2026-07-22T08:56:24Z

+            true => None,
+            false => match jobs {
+                Some(n) => n,
+                None => NonZero::new(32), // back compat with historical behavior


I don't think there was any cap when using a jobserver previously. The 32 tokens default is when there is no jobserver. And in any case I don't think the "The fixed number is used to have deterministic compilation across machines." comment is actually correct. Changing the max number of parallel threads doesn't affect determinism of the output at all. Only the amount of codegen units.

View changes since the review

I don't think there was any cap when using a jobserver previously.

Yes, if there's an external jobserver, then there's no static limit on main, only the dynamic limit from that jobserver.
But in that case --jobs-backend is not currently respected anyway (see the FIXME in write.rs).
I can keep the default "unlimited" behavior when addressing the FIXME (in a separate later PR).

And in any case I don't think the "The fixed number is used to have deterministic compilation across machines." comment is actually correct.

Yeah, that comment is removed now.

Added a variant to preserve the historical behavior precisely, even when the FIXME in write.rs is addressed.

bjorn3 · 2026-07-22T08:56:52Z

+            Opt,
+            "",
+            "jobs-backend",
+            "Limit on the number of parallel jobs used by backend",


Is this supposed to be respected when a jobserver exists?

View changes since the review

Yes, eventually (right now there's a FIXME about it in write.rs).
jobs_backend is the static upper limit, and the jobserver can limit work further within the 1 ..= jobs_backend range.

Zoxc · 2026-07-22T12:08:44Z

Why is -Zthreads kept given that --jobs-frontend seems to be a replacement?

I don't like the --jobs-frontend naming as the thread pool it limits is not intended nor is it actually limited to the frontend. A more accurate name would be a query system thread pool though it isn't strictly limited to that either, but it has to accommodate threads potentially blocking on queries which is notable.

petrochenkov · 2026-07-22T12:33:10Z

Why is -Zthreads kept given that --jobs-frontend seems to be a replacement?

I expect a large number of users of -Zthreads, so it makes sense to migrate more gradually and support the old name for some time.

I don't like the --jobs-frontend naming as the thread pool it limits is not intended nor is it actually limited to the frontend. A more accurate name would be a query system thread pool though it isn't strictly limited to that either, but it has to accommodate threads potentially blocking on queries which is notable.

Better naming suggestions are welcome.
The last notable work in the "frontend" pool (besides the "main thread", which also belongs to the pool) is a part of codegen (MIR -> LLVM IR lowering) in fn codegen_crate, after that the pool still exists, but is not really used for parallel work.
(That's what I tried to document in src/doc/rustc/src/command-line-arguments.md.)

rustbot · 2026-07-23T14:03:18Z

rustc-dev-guide is developed in its own repository. If possible, consider making this change to rust-lang/rustc-dev-guide instead.

cc @BoxyUwU, @tshepang

petrochenkov · 2026-07-23T18:14:23Z

All the comments are addressed.
@rustbot ready

rustbot · 2026-07-29T11:37:49Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

…arts of the compiler

bjorn3 · 2026-07-31T11:47:05Z

Implementation looks fine. And based on the MCP being accepted, I'm going to assume the CLI is fine too. Can still be adjusted before stabilization if necessary.

@bors r+

rust-bors · 2026-07-31T11:47:08Z

📌 Commit 2b4375d has been approved by bjorn3

It is now in the queue for this repository.

@bjorn3

rustc: Support `--jobs` options for limiting parallelism in various parts of the compiler `--jobs-frontend` for the frontend, `--jobs-backend` for the backend, `--jobs-linker` for the linker, and `-j`/`--jobs` for the overall limit. See the added docs for more details. Part of rust-lang/compiler-team#1005. Any suspicious option combinations are prohibited for now. r? @bjorn3 @Zoxc

…uwer Rollup of 21 pull requests Successful merges: - #160100 (Add "system" option to `override-allocator` directive) - #159675 (rustc: Support `--jobs` options for limiting parallelism in various parts of the compiler) - #159999 (Fix invalidation of stdlib in bootstrap when using non-LLVM codegen backends) - #160233 (Bubble bad path error while parsing field to avoid unecessary second error) - #160272 (rustc_metadata: Move native library search code to `rustc_codegen_ssa`) - #154202 (rustfmt: Format `cfg_select!`) - #158835 (rustc_passes: lint unused `#[path]` attributes on inline modules) - #159520 (Suggest `Vec<T>` instead of `[T]`) - #160034 (Move "macro only" check for `#[allow_internal_unsafe/unstable]` to attribute parser) - #160066 (rustc_middle: lint attribute cleanups) - #160085 (Remove various superfluous lint attributes) - #160113 (Coalesce `rustc_on_unimplemented` attributes and lint malformed filters) - #160119 (fix query cycle in `coroutine_hidden_types` for the next solver) - #160147 (tests: Remove `-Zthreads` options from tests in `ui/parallel-rustc`) - #160157 (Remove outdated comments from `va_list.rs`) - #160159 (More accurately check for interior mutability in `invalid_reference_casting` lint) - #160208 (rustdoc: Fix crash when trying to list attributes on an opaque type) - #160244 (Rename splat to rustc_splat in error messages) - #160246 (dont fire `unused_mut` on `&pin mut self`) - #160247 (Configure backport nominations for rustfmt) - #160274 (renovate: update lock files weekly)

…uwer Rollup of 22 pull requests Successful merges: - #160100 (Add "system" option to `override-allocator` directive) - #160220 (Refactor: shrink region ext traits) - #159675 (rustc: Support `--jobs` options for limiting parallelism in various parts of the compiler) - #159999 (Fix invalidation of stdlib in bootstrap when using non-LLVM codegen backends) - #160233 (Bubble bad path error while parsing field to avoid unecessary second error) - #160272 (rustc_metadata: Move native library search code to `rustc_codegen_ssa`) - #154202 (rustfmt: Format `cfg_select!`) - #159520 (Suggest `Vec<T>` instead of `[T]`) - #159710 (Add rustdoc/cargo PGO profiles to reproducible artifacts) - #160034 (Move "macro only" check for `#[allow_internal_unsafe/unstable]` to attribute parser) - #160066 (rustc_middle: lint attribute cleanups) - #160085 (Remove various superfluous lint attributes) - #160113 (Coalesce `rustc_on_unimplemented` attributes and lint malformed filters) - #160119 (fix query cycle in `coroutine_hidden_types` for the next solver) - #160147 (tests: Remove `-Zthreads` options from tests in `ui/parallel-rustc`) - #160157 (Remove outdated comments from `va_list.rs`) - #160159 (More accurately check for interior mutability in `invalid_reference_casting` lint) - #160208 (rustdoc: Fix crash when trying to list attributes on an opaque type) - #160244 (Rename splat to rustc_splat in error messages) - #160246 (dont fire `unused_mut` on `&pin mut self`) - #160247 (Configure backport nominations for rustfmt) - #160274 (renovate: update lock files weekly)

@bjorn3

Rollup merge of #159675 - petrochenkov:jjj, r=bjorn3 rustc: Support `--jobs` options for limiting parallelism in various parts of the compiler `--jobs-frontend` for the frontend, `--jobs-backend` for the backend, `--jobs-linker` for the linker, and `-j`/`--jobs` for the overall limit. See the added docs for more details. Part of rust-lang/compiler-team#1005. Any suspicious option combinations are prohibited for now. r? @bjorn3 @Zoxc

rustbot assigned bjorn3 Jul 21, 2026

petrochenkov commented Jul 21, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

bjorn3 reviewed Jul 22, 2026

View reviewed changes

Comment thread compiler/rustc_session/src/config.rs

bjorn3 reviewed Jul 22, 2026

View reviewed changes

Comment thread src/doc/rustc/src/command-line-arguments.md

petrochenkov force-pushed the jjj branch from a3f51aa to 483903a Compare July 23, 2026 14:03

rustbot added the A-rustc-dev-guide Area: rustc-dev-guide label Jul 23, 2026

This comment has been minimized.

Sign in to view

petrochenkov force-pushed the jjj branch from 483903a to c2159f4 Compare July 23, 2026 18:02

This comment has been minimized.

Sign in to view

petrochenkov force-pushed the jjj branch from c2159f4 to 2cd941d Compare July 23, 2026 20:06

This comment has been minimized.

Sign in to view

petrochenkov mentioned this pull request Jul 28, 2026

Promoting Parallel Front End rust-lang/rust-project-goals#121

Open

6 tasks

bjorn3 reviewed Jul 29, 2026

View reviewed changes

Comment thread tests/ui/compile-flags/jobs/jobs-fail-gate.rs

bjorn3 reviewed Jul 29, 2026

View reviewed changes

Comment thread compiler/rustc_codegen_ssa/src/back/link.rs Outdated

bjorn3 reviewed Jul 29, 2026

View reviewed changes

Comment thread compiler/rustc_codegen_ssa/src/back/link.rs Outdated

petrochenkov force-pushed the jjj branch from 2cd941d to 0dc7e46 Compare July 29, 2026 11:37

bjorn3 reviewed Jul 29, 2026

View reviewed changes

Comment thread compiler/rustc_codegen_ssa/src/back/link.rs Outdated

rustc: Support --jobs options for limiting parallelism in various p…

2b4375d

…arts of the compiler

petrochenkov force-pushed the jjj branch from 0dc7e46 to 2b4375d Compare July 29, 2026 12:33

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 31, 2026

JonathanBrouwer mentioned this pull request Jul 31, 2026

Rollup of 21 pull requests #160283

Closed

JonathanBrouwer mentioned this pull request Jul 31, 2026

Rollup of 22 pull requests #160284

Merged

rust-bors Bot merged commit 2eb8d70 into rust-lang:main Jul 31, 2026
13 checks passed

rustbot added this to the 1.99.0 milestone Jul 31, 2026

	pub frontend: Option<NonZero<u8>>,
	pub frontend: Option<NonZero<usize>>,

Uh oh!

Uh oh!

Conversation

petrochenkov commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 21, 2026

Uh oh!

petrochenkov Jul 21, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov Jul 21, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov Jul 21, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

bjorn3 Jul 22, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorn3 Jul 22, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

petrochenkov Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Zoxc commented Jul 22, 2026

Uh oh!

petrochenkov commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

petrochenkov commented Jul 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

rustbot commented Jul 29, 2026

Uh oh!

Uh oh!

bjorn3 commented Jul 31, 2026

petrochenkov commented Jul 21, 2026 •

edited

Loading

petrochenkov Jul 21, 2026 •

edited by rustbot

Loading

petrochenkov Jul 23, 2026 •

edited

Loading

petrochenkov Jul 21, 2026 •

edited by rustbot

Loading

petrochenkov Jul 21, 2026 •

edited by rustbot

Loading

bjorn3 Jul 22, 2026 •

edited by rustbot

Loading

bjorn3 Jul 22, 2026 •

edited by rustbot

Loading

petrochenkov Jul 22, 2026 •

edited

Loading

petrochenkov commented Jul 22, 2026 •

edited

Loading