Look into using es regex

[Keats]

See https://github.com/ridiculousfish/regress and https://github.com/slevithan/oniguruma-to-es which shiki uses.
Worse perf but no C deps

[Keats]

There's also https://github.com/fancy-regex/fancy-regex which doesn't support every onig thing right now

[nik-rev]

It would be amazing to get rid of C dependencies - I'd really like to try and replace syntect in Typst with giallo but Typst wants to remain pure rust, so any transitive C dependencies are a blocker.

I previously tried adding tree-sitter integration to Typst, but that didn't end well (mostly due to incompatible WASM runtimes, but also in part because of the C dependencies that tree-sitter requires)

[Keats]

See fancy-regex/fancy-regex#162 there are still a few things missing from fancy-regex since there are 41 grammars that won't work (including typst ironically). I won't have the time myself to test with regress but if someone wants to try I would definitely take a PR, asumming performance is roughly similar.

[nik-rev]

See fancy-regex/fancy-regex#162 there are still a few things missing from fancy-regex since there are 41 grammars that won't work (including typst ironically). I won't have the time myself to test with regress but if someone wants to try I would definitely take a PR, asumming performance is roughly similar.

If you include grammars with backreferences, by removing these lines from the test:

// Skip patterns with backreferences
if super::has_backreferences(pattern) {
    continue;
}

Then 79/244 languages fail instead of just 41/244

Why are backreferences not included in the original test? fancy-regex claims to support them, wonder why it fails here?

  Pattern: \)\2(\3)"
  Error: Error compiling regex: Invalid back reference to group 3

  Pattern: \)\2(\3)"
  Error: Error compiling regex: Invalid back reference to group 3

[Keats]

Why are backreferences not included in the original test? fancy-regex claims to support them, wonder why it fails here?

Because they are not valid regex at compile time. The way textmate works is that for example in a BeginEnd pattern, the end regex pattern can reference the matches from the begin regex. So in practice that regex in your post is never compiled, giallo will replace those \2, \3 etc by the matches from the start regex. Since they are not valid regex, they can be ignored when testing for compatibility - they only become valid at runtime.