I've run into a problem where a UTF-8 encoded string is parsed by Simdjson.parse and one of the resulting strings is encoded in ASCII-8BIT. I can reproduce this like so:
# run with ruby --encoding=UTF-8 if UTF-8 isn't your system default.
x = '{"m":" – "}' # note the non-ASCII character in the value
puts x.encoding # => #<Encoding::UTF-8>
y = Simdjson.parse(x)
puts y['m'].encoding # => #<Encoding:ASCII-8BIT>
It seems like the encoding of the output strings should remain the same as the encoding of the input string, right? I'm not sure if this is an issue that belongs here or in the main simdjson repository but I appreciate you taking a look either way.
I've run into a problem where a UTF-8 encoded string is parsed by
Simdjson.parseand one of the resulting strings is encoded in ASCII-8BIT. I can reproduce this like so:It seems like the encoding of the output strings should remain the same as the encoding of the input string, right? I'm not sure if this is an issue that belongs here or in the main simdjson repository but I appreciate you taking a look either way.