Prevent macro expansion hang from exponential token growth

[vincenzopalazzo]

Summary

Fixes #95698

Recursive macro_rules! macros can produce exponentially growing token streams that hang the compiler. While the recursion depth limit (default 128) eventually catches infinite recursion, a macro that doubles its output per expansion would produce 2^128 tokens before the depth limit fires — the compiler hangs trying to parse this astronomical input through parse_tt.

This PR adds a new configurable macro_token_limit that caps the number of top-level tokens allowed as input to a single macro_rules! expansion:

• Default: 2^20 (~1 million) tokens
• Configurable via #![macro_token_limit = "N"], following the same pattern as #![recursion_limit = "N"]
• Only applies to macro_rules! (not proc macros)

Why 2^20 as the default?

Based on research across real-world Rust codebases:

Source	Largest legitimate macro invocation
rust-analyzer full analysis of its codebase	~30,672 tokens
Typical large derive/dispatch macros	5,000–100,000 tokens
Extreme case (rust-analyzer issue #10855)	~997,309 tokens
rust-analyzer's own token limit	2,097,152 (2^21)

The 2^20 default provides ~10x headroom over typical large macros. In the pathological case (exponential doubling from 3 tokens), the limit fires at recursion depth ~19 — producing an error in ~200ms instead of hanging indefinitely. The #![macro_token_limit] attribute provides an escape hatch for the rare legitimate case that needs more.

Reproducer (previously hung the compiler)

macro_rules! from_cow_impls {
    ($( $from: ty ),+ $(,)? ) => {
        from_cow_impls!($($from, Cow::from),+);
    };
    ($( $from: ty, $normalizer: expr ),+ $(,)? ) => {};
}

from_cow_impls!(u8, u16);

Now produces:

error: macro expansion token limit reached while expanding `from_cow_impls!`

Changes

File	Change
rustc_span/src/symbol.rs	Add macro_token_limit symbol
rustc_feature/src/builtin_attrs.rs	Register as ungated crate-level attribute
rustc_hir/src/attrs/data_structures.rs	Add MacroTokenLimit to AttributeKind
rustc_hir/src/attrs/encode_cross_crate.rs	Mark as No cross-crate encoding
rustc_attr_parsing/src/attributes/crate_level.rs	Add MacroTokenLimitParser
rustc_attr_parsing/src/context.rs	Register the parser
rustc_interface/src/limits.rs	Add get_macro_token_limit() with default
rustc_interface/src/passes.rs	Read attribute pre-expansion, pass to config
rustc_expand/src/expand.rs	Add macro_token_limit field to ExpansionConfig
rustc_expand/src/mbe/macro_rules.rs	Check token count in expand_macro
rustc_expand/src/errors.rs	New MacroInputTooLarge diagnostic
rustc_passes/src/check_attr.rs	Handle new attribute variant

Test plan

• New regression test tests/ui/macros/issue-95698-exponential-token-growth.rs
• x.py test tests/ui/macros/issue-95698-exponential-token-growth.rs passes (blessed)
• x.py check compiler/rustc_expand passes
• Full CI

[rustbot]

Some changes occurred in compiler/rustc_hir/src/attrs

cc @jdonszelmann, @JonathanBrouwer

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann, @JonathanBrouwer

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann, @JonathanBrouwer

[rustbot]

r? @jackh726

rustbot has assigned @jackh726.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 69 candidates
• Random selection from 10 candidates

[rustbot]

⚠️ Warning ⚠️

• There are issue links (such as #123) in the commit messages of the following commits.
  Please move them to the PR description, to avoid spamming the issues with references to the commit, and so this bot can automatically canonicalize them to avoid issues with subtree.
  • 03aa980

[rust-log-analyzer]

The job aarch64-gnu-llvm-21-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

##[endgroup]
Executing "/scripts/stage_2_test_set1.sh"
+ /scripts/stage_2_test_set1.sh
+ '[' 1 == 1 ']'
+ echo 'PR_CI_JOB set; skipping tidy'
+ SKIP_TIDY='--skip tidy'
+ ../x.py --stage 2 test --skip tidy --skip compiler --skip src
PR_CI_JOB set; skipping tidy
##[group]Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
---
To only update this specific test, also pass `--test-args cfg/suggest-alternative-name-on-target.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/cfg/suggest-alternative-name-on-target" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "incomplete_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error: unexpected `cfg` condition value: `arm`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:5:7
   |
---
LL | #![deny(unexpected_cfgs)]
   |         ^^^^^^^^^^^^^^^
help: `arm` is an expected value for `target_arch`
   |
LL - #[cfg(target_abi = "arm")]
LL + #[cfg(target_arch = "arm")]
   |

error: unexpected `cfg` condition value: `gnu`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:12:7
   |
LL | #[cfg(target_arch = "gnu")]
   |       ^^^^^^^^^^^^^^^^^^^
   |
   = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg.html> for more information about checking conditional configuration
help: `gnu` is an expected value for `target_env`
   |
LL - #[cfg(target_arch = "gnu")]
LL + #[cfg(target_env = "gnu")]
   |

error: unexpected `cfg` condition value: `openbsd`
##[error]  --> /checkout/tests/ui/cfg/suggest-alternative-name-on-target.rs:19:7
   |

[rust-bors]

☔ The latest upstream changes (presumably #154958) made this pull request unmergeable. Please resolve the merge conflicts.

[jackh726]

CI isn't passing. And there needs to be many more tests.

In general, I think this needs lang approval, since it's a new attribute.

View changes since this review

[rust-bors]

☔ The latest upstream changes (presumably #155416) made this pull request unmergeable. Please resolve the merge conflicts.

[jackh726]

Just checking in from the lang team triage meeting:

Seems like the the team is on board with doing this as an experiment, which means this will need to be feature-gated.

Three things brought up by @joshtriplett:

• We should probably be thinking about having better diagnostics for this (like we do for const fn) so that people can figure out what macros are taking a long time. That may be good to do, whether or not we also do this change.
• As an implementation detail (and we should be testing this): unless the feature gate is enabled, we should not have some macro token limit (otherwise if this does break someone, they will need nightly to fix that).
• A question was raised about whether this is only on the crate, or if you could write this on particular macros. From the current PR, it seems like the former - but there is some question of if we want to latter too (like, e.g. how we let you override lints).

Reviewer notes:

• This will need many tests; including feature gated behavior, testing that writing the attribute works for stopping smaller macros, etc.
• For the experiment, the original issue should be closed; instead, a new tracking issue should be opened and linked to
• Would be good to split this into multiple PRs; likely, e.g., an initial one just with the feature gate and tests being added.
• Testing is going to be a bit difficult: I don't think the MCVE from the linked issue is great, because that's going to take a while to fail (probably).