Use depth-first traversal for macro parsing by bal-e · Pull Request #159808 · rust-lang/rust

bal-e · 2026-07-24T03:35:25Z

This PR changes the control flow in rustc's declarative macro parser to use a depth-first traversal instead of a breadth-first traversal. It builds upon a series of preparatory changes (#158577, #158894, #158974, #158976) that removed reliance on breadth-first traversal (usually with some performance improvements).

There are several motivations for this change.

It introduces an important invariant (the sort order in backtrack) needed for optimizing MatcherPos::matches (turning it into a single, non-Rc'd field on TtParser). This will be implemented in a later PR.
It reduces reliance on the heap. The backtrack stack is only accessed when an mp fails to match. While it introduces the seen_tokens list, this list is kept quite small, and it can be optimized further (e.g. only adding tokens to it if !backtrack.is_empty()).
It should offer a more predictable workflow for the CPU, possibly showing a greater cycle-count improvement than instruction-count.

This is a perf-sensitive PR, I'd appreciate it if someone can start a perf run.

r? @nnethercote. Commits are individually reviewable. I'm happy to split out some of the earlier commits into separate PRs.

`has_no_remaining_items_for_step()` is only meaningful for breadth-first traversal during parsing; it needs to be removed or adjusted to make depth-first traversal possible. This has a small effect on error-reporting behavior. In one case in the `assert-trailing-junk.rs` test, `$(,)` (at the end of the LHS) is matched against `blah`. This leads to three possible `MatcherLoc`s being investigated in the last step: the sequence start, the `,`, and EOF (in this order). Before this commit, the sequence start is reporting as `remaining_matcher`; when it is reached, there is nothing else in `cur_mps`. The `,` is ignored because the EOF is also in `cur_mps`, and the EOF is ignored because it is EOF. After this commit, the `,` is reported as `remaining_matcher`. This is the only change, and I think the new behavior is more useful; it will prioritize the contents of a sequence over a "sequence start" `MatcherLoc`. The `Display` impl for `MatcherLoc` incorrectly noted that "sequence start" is not used in diagnostics. With this change, it is less likely to be used, but I've left in a FIXME to investigate that thoroughly.

This commit reduces `Tracker`'s reliance on `Parser`. `Parser` can only be relied on for information about the furthest match; this is not a problem for BFS because all mps are at the same input position. But in DFS, mps will have varying input positions. Now the diagnostics tracker will collect every token observed from the parser. In the next commit, this will be used in `ambiguity()`. `tests/ui/macros` passes.

With this commit, `CollecTrackerAndEmitter::ambiguity()` relies on the `tokens` field (added in the last commit) instead of the `parser` parameter. It identifies ambiguity by finding the earliest position where ambiguity occurred. This nicely crosses the bridge from BFS to DFS -- in BFS, the position of ambiguity is obvious, but in DFS, there could be multiple ambiguities at different positions and the earliest one needs to be prioritized.

`check_for_ambiguity()` repeated some of the work done by `parse_tt()` (specifically, processing mps from `cur_mps`). The previous flow for metavar/EOF matching was: - During `parse_tt_inner()`: - During `match_one()`: - Check for a match, e.g. with `nonterminal_may_begin_with()`. - If `checking_for_ambiguity`, fail. - Call `check_for_ambiguity()`: - Drain everything in `cur_mps`. - If anything matched successfully, fail. - Finish processing the mp, e.g. `Parser::parse_nonterminal()`. The new flow is: - During `parse_tt_inner()`: - During `match_one()`: - Check for a match, e.g. with `nonterminal_may_begin_with()`. - Store the mp in `maybe_ambig_mp`. - If something is already there, fail. - Drain everything in `cur_mps`. - If `maybe_ambig_mp` is set: - If anything matched successfully, fail. - Finish processing the mp, e.g. `Parser::parse_nonterminal()`. This is quite similar to the structure before I started making changes, e.g. via `bb_mps`. In the new structure, it also handles EOF. `tests/ui/macros` passes.

rustbot · 2026-07-24T03:35:29Z

Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @nnethercote (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Kobzol · 2026-07-24T04:38:10Z

@bors try @rust-timer queue

Use depth-first traversal for macro parsing

bal-e · 2026-07-24T05:44:48Z

The bug was a bit more involved than I realized, but I managed to retain the old behavior (at least as far as the UI tests are concerned). In addition (the last two commits), I simplified the control flow around parse_tt() / parse_tt_inner() / etc. and updated the top-level docs.

Previously, `remaining_matcher` was picked based on implementation dependent ordering of mps. With this commit, it is deterministically selected as the furthest-along mp with the furthest-along loc.

Needed so I can add another field to `MatcherPos` without growing it.

This will form the basis for backtracking. Note that it is cleared after meta-variable parsing; since meta-variables do not admit ambiguity, and `cur_mps` is cleared by `check_for_ambiguity()` when a meta-variable is about to be parsed, no mps could possibly exist to refer to older tokens from the input.

This will be used for backtracking. At the moment, all mps in `cur_mps` have the same input position, and all mps in `next_mps` have the same input position (exactly one more than that in `cur_mps`).

`tests/ui/macros` passes!!!

Instead of pushing and popping `backtrack` all the time, try to return an mp for immediate use.

rust-bors · 2026-07-24T06:43:52Z

☀️ Try build successful (CI)
Build commit: cb2bbd4 (cb2bbd46e2ed7c8222822175981471a02f2d3a83)
Base parent: 76c35a1 (76c35a14a45897a366c11657ecbbe9ec4153ad17)

rust-timer · 2026-07-24T07:23:54Z

Finished benchmarking commit (cb2bbd4): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.2%, 1.4%]	50
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	1
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.2%]	9
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.2%]	6
All ❌✅ (primary)	0.4%	[-0.3%, 1.4%]	59

Max RSS (memory usage)

Results (secondary -3.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.8%	[-5.0%, -2.6%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (primary 4.4%, secondary 14.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	4.4%	[1.2%, 8.0%]	28
Regressions ❌ (secondary)	14.8%	[2.5%, 28.7%]	15
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	4.4%	[1.2%, 8.0%]	28

Binary size

Results (primary -0.0%, secondary -0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.0%	[-0.0%, -0.0%]	4
Improvements ✅ (secondary)	-0.0%	[-0.1%, -0.0%]	16
All ❌✅ (primary)	-0.0%	[-0.0%, -0.0%]	4

Bootstrap: 488.274s -> 487.61s (-0.14%)
Artifact size: 387.66 MiB -> 387.60 MiB (-0.02%)

bal-e · 2026-07-24T07:27:59Z

Oh, yeesh.

nnethercote · 2026-07-24T07:41:14Z

Instruction counts are moderately bad, cycles and wall-times are significantly worse :(

nnethercote · 2026-07-24T09:18:02Z

@bors try @rust-timer queue

Use depth-first traversal for macro parsing

rust-bors · 2026-07-24T11:25:21Z

☀️ Try build successful (CI)
Build commit: ff100f1 (ff100f1f5cb432915bad69142842d57f9a9710cd)
Base parent: 29e68fe (29e68fe2295f8fc2feb52b8cb0b61a055842fdcf)

rust-timer · 2026-07-24T12:32:23Z

Finished benchmarking commit (ff100f1): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.5%]	9
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.7%	[-2.0%, -0.2%]	35
Improvements ✅ (secondary)	-1.5%	[-5.0%, -0.2%]	33
All ❌✅ (primary)	-0.5%	[-2.0%, 0.5%]	44

Max RSS (memory usage)

Results (secondary 1.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.0%	[2.1%, 5.9%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-5.1%	[-5.1%, -5.1%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (primary 2.3%, secondary 8.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.2%, 2.4%]	2
Regressions ❌ (secondary)	8.5%	[2.8%, 15.1%]	14
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[2.2%, 2.4%]	2

Binary size

Results (primary -0.0%, secondary -0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.0%	[-0.0%, -0.0%]	4
Improvements ✅ (secondary)	-0.0%	[-0.1%, -0.0%]	16
All ❌✅ (primary)	-0.0%	[-0.0%, -0.0%]	4

Bootstrap: 488.104s -> 488.233s (0.03%)
Artifact size: 387.69 MiB -> 387.68 MiB (-0.00%)

bal-e · 2026-07-24T12:34:31Z

A lot more tenable, but cycle counts are still regressing. I seem to be on the right track and I can think of some ways to improve it further :)

nnethercote · 2026-07-24T21:01:32Z

Looking forward to the improved version :)

@rustbot author

rustbot · 2026-07-24T21:01:36Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

arya dradjica added 5 commits July 21, 2026 09:52

Remove return value from match_one()

888a320

rustbot assigned nnethercote Jul 24, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 24, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 24, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 24, 2026

Auto merge of #159808 - bal-e:depth-first-macro-parsing, r=<try>

cb2bbd4

Use depth-first traversal for macro parsing

bal-e force-pushed the depth-first-macro-parsing branch from f2fc1eb to 5413c09 Compare July 24, 2026 05:42

arya dradjica added 8 commits July 24, 2026 08:24

Deterministically pick remaining_matcher

da78b53

Previously, `remaining_matcher` was picked based on implementation dependent ordering of mps. With this commit, it is deterministically selected as the furthest-along mp with the furthest-along loc.

Use u32s for indices into [MatcherLoc]

00be6b5

Needed so I can add another field to `MatcherPos` without growing it.

Track input position in MatcherPos

436e129

This will be used for backtracking. At the moment, all mps in `cur_mps` have the same input position, and all mps in `next_mps` have the same input position (exactly one more than that in `cur_mps`).

Use depth-first traversal for macro parsing

6683197

`tests/ui/macros` passes!!!

Store current mp in local variable

c07eb9f

Instead of pushing and popping `backtrack` all the time, try to return an mp for immediate use.

Merge backtracking logic into parse_tt()

b5dbe2c

Update macro_parser.rs docs for DFS

1859c23

bal-e force-pushed the depth-first-macro-parsing branch from 5413c09 to 1859c23 Compare July 24, 2026 06:25

This comment has been minimized.

Sign in to view

rustbot added the perf-regression Performance regression. label Jul 24, 2026

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 24, 2026

wip: wrangle perf, try 1

807fd41

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 24, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 24, 2026

Auto merge of #159808 - bal-e:depth-first-macro-parsing, r=<try>

ff100f1

Use depth-first traversal for macro parsing

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 24, 2026

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 24, 2026

bal-e mentioned this pull request Jul 26, 2026

Tracking issue (ish) for some declarative macro optimizations #159951

Open

23 tasks

Uh oh!

Uh oh!

Conversation

bal-e commented Jul 24, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 24, 2026

Uh oh!

This comment has been minimized.

Kobzol commented Jul 24, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

bal-e commented Jul 24, 2026

Uh oh!

rust-bors Bot commented Jul 24, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jul 24, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

bal-e commented Jul 24, 2026

Uh oh!

nnethercote commented Jul 24, 2026

Uh oh!

nnethercote commented Jul 24, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jul 24, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jul 24, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

bal-e commented Jul 24, 2026

Uh oh!

nnethercote commented Jul 24, 2026

Uh oh!

rustbot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bal-e commented Jul 24, 2026 •

edited by rustbot

Loading