Skip to content

perf: skip tokenizer trivia bookkeeping without trivia#713

Merged
lwshang merged 2 commits into
masterfrom
perf-4-tokenizer-no-trivia
Mar 15, 2026
Merged

perf: skip tokenizer trivia bookkeeping without trivia#713
lwshang merged 2 commits into
masterfrom
perf-4-tokenizer-no-trivia

Conversation

@sasa-tomic

@sasa-tomic sasa-tomic commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Overview
Reduce parser work on tokenizer paths that do not capture trivia.

Requirements
Keep comment and doc-comment behavior unchanged when trivia capture is enabled.

Solution
Skip comment and line-tracking bookkeeping when the tokenizer runs without trivia capture. This trims successful DID and argument parsing on the hot path without changing the trivia-enabled path.

Considerations
Parser behavior with trivia capture is unchanged, and the parser test suite still passes. Series-level benchmark context is tracked in #710.

Avoid comment and line-tracking work when parsers do not capture trivia. This trims successful DID and arg parsing on the hot path without changing the trivia-enabled behavior.

Refs #710
@sasa-tomic sasa-tomic requested a review from a team as a code owner March 13, 2026 11:39
@github-actions

github-actions Bot commented Mar 13, 2026

Copy link
Copy Markdown
Name Max Mem (Kb) Encode Decode
blob 4_224 4_207_487 2_122_432
btreemap 73_856 531_975_925 12_984_691_964
nns 192 2_021_253 5_663_359 ($\textcolor{green}{-0.01\%}$)
nns_list_proposal 1_216 7_013_446 ($\textcolor{red}{0.06\%}$) 64_312_599 ($\textcolor{green}{-0.00\%}$)
option_list 64 716_007 21_888_381
text 6_336 4_204_384 7_877_759
variant_list 64 710_989 20_528_739
vec_int16 16_704 123_694_298 633_364_072
  • Parser cost: 16_745_623 ($\textcolor{green}{-1.90\%}$)
  • Extra args: 2_872_191
Click to see raw report
---------------------------------------------------

Benchmark: blob
  total:
    instructions: 6.33 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.21 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 2.12 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 13.52 B (no change)
    heap_increase: 1154 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 531.98 M (no change)
    heap_increase: 159 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 12.98 B (no change)
    heap_increase: 995 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 2.87 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 25.27 M (-1.27%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    calls: 1 (no change)
    instructions: 16.75 M (-1.90%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 2.02 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 5.66 M (-0.01%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns_list_proposal
  total:
    instructions: 71.33 M (0.01%) (change within noise threshold)
    heap_increase: 19 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 7.01 M (0.06%) (change within noise threshold)
    heap_increase: 5 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 64.31 M (-0.00%) (change within noise threshold)
    heap_increase: 14 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 22.61 M (no change)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 716.01 K (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 21.89 M (no change)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 12.08 M (no change)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.20 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 7.88 M (no change)
    heap_increase: 33 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 21.24 M (no change)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 710.99 K (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 20.53 M (no change)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 757.06 M (no change)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 123.69 M (no change)
    heap_increase: 261 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 633.36 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 9 | regressed 0 | improved 0 | new 0 | unchanged 9]
    change:   [max +3.71K | p75 0 | median 0 | p25 0 | min -325.31K]
    change %: [max 0.01% | p75 0.00% | median 0.00% | p25 0.00% | min -1.27%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 9 | regressed 0 | improved 0 | new 0 | unchanged 9]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 9 | regressed 0 | improved 0 | new 0 | unchanged 9]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
Successfully persisted results to canbench_results.yml

@lwshang lwshang merged commit 1cb9813 into master Mar 15, 2026
11 checks passed
@lwshang lwshang deleted the perf-4-tokenizer-no-trivia branch March 15, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants