Why not separate string types for each encoding?

[conrad-watt]

I haven't been part of the conversations surrounding this specific proposal, so apologies if I'm retreading previous discussions. I'm not specifically hoping to effect a concrete change to the proposal with this question - I mainly want to understand the design better.

The operations on the stringref type already seem rather opinionated in the actual encodings supported - (e.g. encode_wtf16 and encode_wtf8). Moreover, the current proposal introduces separate view types for wtf16 and wtf8. Was a design considered where there are separate stringref_wtf16 and stringref_wtf8 types instead of a "unified" stringref type? IIUC this would eliminate the need for separate view types (except perhaps stringview_iter).

[wingo]

Heya @conrad-watt -- good question. Some attempts to answer it :)

Firstly, a goal was to not add to the number of string implementations on the web. (Of course one may argue about whether that is a good goal or not.) stringref_wtf8 would be another string implementation, which couldn't be exposed to JS as-is. I didn't seriously consider the possibility of having two separate types, so I might not have fully thought out the implications here.

As regards e.g. stringref.encode_wtf8 and stringref.encode_wtf16 -- I think this is an unrelated question. When we think of bulk operations like this, it's essentially the same cost to use WTF-8 or WTF-16 -- on-the-fly transcoding is very cheap. And you will want them both -- even in Java you might well want to encode a string as UTF-8 or WTF-8 for various purposes, for example for sending a value over the wire. The place where you want differentiation is whenever you need to denote a position in a string without copying out the bytes to a specific encoding: indexed access, chunked processing, slices, and iteration. All of those operations are punted to the views.

I have the impression though that if possible, a platform should have just one string type. The network-effect results are better, when it comes time to compose modules and components into a system. I know that there is no magic bullet for inter-language composition but strings are a useful "interoperation MVP" between otherwise unrelated components. There is a core of relatively encoding-independent cases (eq? comparison, create from encoding, write encoding, concat), together with simply passing around a string without inspecting it, that you can usefully and cheaply provide irregardless of whether the string is stored as WTF-8 or WTF-16.

Additionally I don't think we should be constrained to think of a JS string as a WTF-16 string. All JS implementations now use WTF-16, true, but it's conceivable that some might switch to WTF-8 internally at some point, which would leave us in a weird state -- presumably V8 (or whatever) would use JS strings as stringref_wtf16, which would be WTF-8 underneath.

Finally, I suspect that having two separate types ends up being the same as having a base stringref and views, in a way. Let's imagine there would be stringref_wtf8.new_from_stringref_wtf16 and vice-versa, to allow for impedance-matching. So, probably in a JS-hosted wasm implementation you may implement that as breadcrumbs over underlying WTF-16 storage. Vice-versa for WTF-16 over WTF-8 on e.g. wasmtime. Given that you'd feel incentives to optimize the reverse case, to strip the wrapper off instead of adding a wrapper, then it seems like reflecting this implementation in the type/object structure is probably a useful thing to do.

[conrad-watt]

Thanks for the thoughtful answer!

Finally, I suspect that having two separate types ends up being the same as having a base stringref and views, in a way.

I could certainly imagine in a world with stringref_wtf8 and stringref_wtf16, there would be incentives for the engines to implement both using the same underlying storage format.

If we believe that engines will want "one true internal string representation" rather than actually separately storing strings from stringref.new_wtf8 as wtf8 and strings from stringref.new_wtf16 as wtf16, I can understand how the current design is arrived at.

[jakobkummerow]

I would add two points to @wingo's reply:

(1) Having two types of strings would create a risk of bifurcation of the module ecosystem: to be able to send strings to each other, two modules would have to agree on which type of string to use for this purpose. Having a single string type makes it obvious what the canonical string type is.

(2) By remaining as "high-level"/opaque as possible wrt. actual encoding, we can keep conversions to an opportunistic minimum: suppose in a world of explicitly choosing between stringref_utf8 and stringref_wtf16, point (1) was addressed by an emerging de-facto standard of using the wtf16 variant on public APIs for interop purposes. Then it could happen that two modules that each prefer to use utf8 strings internally have to perform the required conversions just at their boundary. By having a single stringref type, it's entirely feasible for engines to perform re-encodings lazily, which in this case would have the effect of avoiding all conversions. (This is similar in spirit to Interface Types' approach of lifting/lowering adaptor functions that are meant to be inlined-and-eliminated when possible.)
In short: the design aims to give module producers the flexibility and expressiveness they need to conveniently accommodate their respective source languages, while giving (both web and non-web) engines maximum implementation freedom and a high achievable performance ceiling.

[wingo]

continuation -> WebAssembly/stringref#3