implement matching on byte strings

[vi]

Currently as far as I see regexes here are only for Unicode text. But they can be used to parse binary files as well (to to parse a mixture of binary and text).

For example, how can one implement strings --all using regexes in Rust?

[BurntSushi]

This is an interesting question. When building regex, I never even considered the possibility of running a regex on something that wasn't Unicode. If you're matching against, say, &[u8], then how are Unicode structures inside the regex interpreted? For example, \pN or even \w (which is Unicode aware).

For example, how can one implement strings --all using regexes in Rust?

Would a lossy conversion to a Unicode string work? Seems like it should.

[vi]

In Python, when maching over byte array the regex itself is also a byte array (there is nice b"123" syntax for it).

When matching over byte buffer, I may want to capture some binary blob delimited by text, like in BEGIN_BINARY_BLOB\n(.*)\nEND_BINARY_BLOB.

[BurntSushi]

I don't follow. Sorry. I'm not talking about the encoding of the regex, I'm talking about the definition of its Unicode aware character classes. For example see http://doc.rust-lang.org/regex/regex/index.html#perl-character-classes-(unicode-friendly) and http://doc.rust-lang.org/regex/regex/index.html#matching-one-character.

[BurntSushi]

I think the best solution to this---if I understand correctly---is to do lossy Unicode conversion before handing it off to a regex.

If that is for some reason not acceptable, then another way to go about this would be to provide an alternate implementation of Input that restricts characters to single bytes. The problem is: how are non-ASCII single bytes handled? I think the Char type will have to become polymorphic.

It seems doable, but I'm skeptical of whether it's worth it. I'll leave this open for now.

[vi]

Lossy Unicode conversion can break binary data that we may want to extract using the regular expression.

One alternative is to convert from some 8-bit encoding like cp1251, do the regex then convert the result back to cp1251. This way the binary data should be preserved, but it's hacky and not efficient.

The main idea is regular expressions are not only for handling text. Without a special easy syntax for byte buffers like Python's b"123\x00\x00\xFF" they would be obviously are less convenient.

[BurntSushi]

Rust has byte string literals.

This does seem doable. Inputs to the various matching functions would need to become polymorphic (i.e., accept either a &str or a &[u8]), and input handling within the engines would need to be polymorphic too.

[BurntSushi]

I've been thinking about this lately, because I'm currently working on converting the matching engines to be byte based instead of character based. This means that, e.g., . matches Unicode scalar values, which means it would be effectively useless on anything other than UTF-8 encoded data. For example, say I wanted to search for .*?myname.*?. If the regex engine claims to be able to run on binary data, then those dots will foil your intention because they won't consume arbitrary bytes---only valid UTF-8 encodings of scalar values. I contend that this is actually a pretty valid use case, because it means a regex can operate on any data that contains ASCII, which is reasonably common in the wild in my experience. (e.g., latin-1.)

"OK, so add a flag that makes . match bytes instead of characters." Well, that's problematic, because the entire existing API is focused around strings, which must be UTF-8 encoded. If a . can successfully match any byte, then the byte boundaries returned are no longer guaranteed to be on UTF-8 boundaries. Whoops.

So this feature requires some serious thought at the API level.

[vi]

Missing byte regexes can invite hacks.

Imagine a sequence: somebody wants to parse apart some piece of data. At first he just uses strings and regexes, everything works.

But later he adds features/refactors and realizes that this should be byte buffer, not a string. So regexes should sometimes now extract a string, but other times extract a byte buffer.

Case 1: byte regexes supported. User processed with refactoring, using byte regexes. Nicer, more secure.

Case 2: only string regexes supported. User aborts refactoring, letting everything to be a string (with some escaping or just latin-1 everywhere). Uglier, less secure.

[BurntSushi]

@vi I don't think I'm questioning the utility of byte based regexes. The question isn't, "Why should we want them?" The question is, "How do we do it?" My other comments have described some of the issues.

[vi]

Just a parallel set of structs/traits won't do? Like Java's InputStreamWhatever and ReaderWhatever.

[BurntSushi]

@vi That would double the size of today's API.

[flying-sheep]

i think it would be useful for fast operations on (mainly) ASCII buffers, e.g. ASCII-based protocols and gene sequence data (which is often stored as ASCII sequence of GTACGTACGATCGATCG...)

byte regexes wouldn’t know anything about unicode and operate bytewise, e.g. . would match any byte (except maybe \n which would be the canonical newline byte like in python bytestrings)