Text format: field names

[tlively]

There's some discussion of this on #333, but it deserves its own issue.

On one hand, it would be nice for debugging and readability if the same field name could be reused across different struct types, but on the other hand this kind of dependent look up would be more complicated than how the text format currently treats names and might require further complications if we ever add unannotated accessor instructions in the future.

[kripken]

Can you please explain a little more about the complexity here?

Naively, when seeing (struct.get $Type 7 the parser needs to understand that this means field number 7 in type $Type, so to find the type of the field being read a lookup is needed using the type. And (struct.get $Type $foo seems to have the same complexity, that is, it also needs to do an operation that figures out information about the field from the combination of type + identifier. That is, a dependent look up is needed either way?

I also don't follow the point about unannotated accessor functions. Those annotations are only in the binary format, aren't they? But this discussion is about the text format.

[tlively]

When field names have to be globally unique, the parser only has to bind $foo to an index (e.g. 7) in the index space of fields. Wherever $foo appears where a field index is required, the parser blindly replaces it with 7. The parser never has to know what type we are indexing into at all. Actually looking up the type definition and indexing into it isn't necessary until validation.

(This simplification does not affect the complexity of Binaryen's parsers because they collect type information as they parse rather than as a separate step.)

For the point about unannotated accessor functions, consider a hypothetical instruction struct.get_unannotated that in the binary format has a field index immediate but no type annotation immediate. You would expect the text format for this instruction to look like struct.get_unannotated $foo where $foo is a field name. This Just Works if $foo is globally unique because it can be parsed as struct.get_unannotated 7, but if different types can contain $foo fields at different indices, then either we force parsers to track type information so it knows which $foo this is or we complicate the text format by inserting extra type information that isn't present in the binary.

[kripken]

I see, thanks. For comparison, I believe all production binary parsers parse+validate at the same time, so if text parsers did the same then neither of the problems you mentioned would be an issue, I think? Is it unreasonable for some reason to expect them to do so?

My perspective on this is that we toolchain developers spend a lot of time debugging using wat files. So any work that goes into making easier debugging there will pay dividends over time for anyone writing a compiler or toolchain for wasm, and that number of people is larger than the number that write wat parsers I think. But I may be missing something.

[tlively]

@rossberg, I don't think supporting a two-level namespace would actually require mixing parser phases. Instead, we could maintain a two-level mapping of type index -> name -> field index that we would collect when we parse type definitions. This is more complex than the name -> index mapping maintained for everything else, but not by much, and it doesn't require any type information or anything like that.

The full syntax for specifying a field index could be something like <fieldidx> ::= u32 | (field <typeidx> id) and we could have abbreviations such that e.g. struct.get t:typeidx f:id expands to struct.get t (field t f). This would be future-compatible with unannotated instructions as well.

Yes, this is more complicated than a single global namespace, but it's also much less surprising for folks expecting field names to behave in a familiar way.

[rossberg]

@tlively, yes, the mixing of phases would come in if we did not have type annotations for the struct type in the text format. Then the parser has no information to go on, and would need to do full typing. Likewise, this would completely screw up the phasing in the spec, with the text format becoming undefinable independent of validation.

The idea for a possible mitigation was to still require the types in the text format, and merely drop them from the binary, and I believe that's roughly what you sketch above. But as @conrad-watt observed during the meeting, that would just shift the problem elsewhere, and e.g. break round-tripping between the formats, since a binary-to-text disassembler would not have enough information to reconstruct these annotations -- unless it also does full type validation.

So for practical purposes, I think dependent field names are truly incompatible with dropping type annotations, since they always entangle at least one of the directions from/to text format with type validation.

[tlively]

the mixing of phases would come in if we did not have type annotations for the struct type in the text format.

I agree that this would be bad, which is why I sketched a solution above where all the necessary information is syntactically present in the fieldidx.

since a binary-to-text disassembler would not have enough information to reconstruct these annotations -- unless it also does full type validation.

Such a disassembler could simply choose to emit a u32 field index instead of a name. We already do not preserve the textual format of indices when round tripping through a binary. The benefits of having a two level namespace for test writing and debugging in now dramatically outweigh the downside of being unable to trivially recover a useful name for a field index when disassembling with hypothetical future instructions.

[jakobkummerow]

As the author and maintainer of a binary-to-text disassembler (which, indeed, currently doesn't track the value stack because it doesn't need to), I don't think it would be overly burdensome to add value stack tracking to it -- certainly not so much so that that'd qualify as "true incompatibility".

I also generally value user friendliness higher than (reasonable amounts of) tooling implementation complexity. It is fair to expect the lowest layers of a technology stack to work a little harder in order to make higher layers' lives easier.

[rossberg]

@tlively, isn't a disassembler an essential part of a debugger? And isn't that exactly where you want symbolic names to show up? It always having to type the entire function first, even if it just wanted to display the current instruction, strikes me as a rather un-assembly value proposition, and could impact user experience in different ways.

[tlively]

Even with a flat namespace for fields, disassemblers won't be able to produce meaningful names without extra type information. Given a field index (and nothing else), the disassembler will be able to come up with a list of every single field name associated with that index across all defined types, but it won't know which one to pick without the exact same information it would need if we had a two-level namespace.

Unless you're thinking that a flat namespace would allow for only a single field name to be associated with each possible index, the same way it works for other index spaces? Surely not, though, because that would be next to useless.

[tlively]

@rossberg, it would be great to settle this issue. Talking to toolchain partners, they universally expressed surprise that we would consider using a single-level namespace, and although they would be able to work around that, they would prefer the two-level namespace.

Since the single-level namespace wouldn't solve any usability problems, even in a hypothetical future with unannotated instructions, and since we have a possible syntax that is forward-compatible with that hypothetical future, are there any other issues that would block us from using a two-level namespace?

[rossberg]

Yeah, I will have to figure out how to spec & implement this in the least intrusive manner. May not get to that in the next couple of weeks.