"The parser"

[nikomatsakis]

It'd be worth covering

• where the lexer is
• how the parser works
• conventions around error recovery
• - What is this eat function?
  • It consumes a token, and queues up the next token as lookahead
• How do you handle identifiers and keywords?
• What about constructing spans?
  • (quoting Niko from below) I have to bring this back into cache, but in general there is this last_span thing that stores the span of the last token, and many things follow the idiom of saving the "lo" point of the span, parsing some stuff, then extracting the "hi" point and combining them. This would be used to make a span that encompasses, for example, an entire trait definition (the lo point would come from the trait keyword, but the end point comes after having parsed a bunch of other things).
• What is this crazy pretty printer?
  • It dumps out the AST in human readable form.
  • We have tests that will parse each file in run-pass, pretty print it, and then test that the resulting pretty printed file type checks, so you have to implement this.
• Something like this: "The parser" #13 (comment) and "The parser" #13 (comment)

[Michael-F-Bryan]

I'd be interested in helping out with the parser section. I haven't made any contributions to the Rust parser specifically, but I've played around with syn and proc macros and also done a fair amount of parsing for other projects (toy projects, custom file formats at work, etc).

Do you know if there's anyone with experience in the parser that I'd be able to bounce questions off?

[nikomatsakis]

@Michael-F-Bryan you can certainly bounce questions off of me; I would say the main owner of the parser is @petrochenkov

[nikomatsakis]

Copying some stuff I wrote in #21 that seems relevant:

What things would someone need to know when either working on the parser or trying to use the parser

Hmm, good question. I imagine it'd be good to look at a few PRs that modified the parser to see what they had to do. Actually, it might be worth just including a link to those PRs in the guide to help people get oriented.

rust-lang/rust#45047 is such a PR, and here is a link to some of the relevant content.

Some things immediately jump out as worth explaining:

• What is this eat function?
  • It consumes a token, and queues up the next token as lookahead
• How do you handle identifiers and keywords?
  • Actually, this PR didn't have to do anything around that. I'd have to find a good example (cc @petrochenkov -- thoughts?)
• What about constructing spans?
  • I have to bring this back into cache, but in general there is this last_span thing that stores the span of the last token, and many things follow the idiom of saving the "lo" point of the span, parsing some stuff, then extracting the "hi" point and combining them. This would be used to make a span that encompasses, for example, an entire trait definition (the lo point would come from the trait keyword, but the end point comes after having parsed a bunch of other things).
• What is this crazy pretty printer?
  • It dumps out the AST in human readable form.
  • We have tests that will parse each file in run-pass, pretty print it, and then test that the resulting pretty printed file type checks, so you have to implement this.

[Michael-F-Bryan]

I'm surprised how familiar a lot of this is! Quite a lot of parsers for toy languages I've written in Rust are effectively a state machine which incrementally consumes tokens (eat()). Your parser keeps track of where it's at (last_span) and then uses that to create a Span so you can figure out where an AST node originally came from. The "union" operation for joining spans is then a logical extension of this for when you compose larger nodes (e.g. a function body) from a bunch of smaller ones (each statement in the function).

I'm interested to find out how libsyntax deals with errors and tries to typecheck bits of your program even if you've got a syntax error elsewhere. My understanding is you break the tokens up into token trees, allowing you to "checkpoint" your progress at known points. Then if one particular token tree fails you still try to pass the rest of the (incomplete) AST to later stages... Is that somewhat close to how rustc does things?

[nikomatsakis]

@Michael-F-Bryan

I'm surprised how familiar a lot of this is!

Yeah, it's your basic recursive descent parser. Nothing too fancy. =)

I'm interested to find out how libsyntax deals with errors and tries to typecheck bits of your program even if you've got a syntax error elsewhere.

This is somewhat ad-hoc, but actually one of the things I would like documented. @nrc is the one who put in a lot of that, and they may be able to comment a bit, though I think @estebank has also put in some effort here. One particular pattern I recall that @estebank has used a few times is that they "checkpoint" the state of the parser and run "tentative" parses that are then discarded -- this is used to give better error messages, basically saying something like "it looks you were writing a fn here, you need to add this keyword".

In short, though, we make a 'best effort' to recover and build up some sort of tree. We wind up reporting an error but then passing this tree back up to the rest of the code, which continues as usual. I'm not sure if there are any kind of "error nodes" in the AST, I didn't see any in a quick search, but I would expect that to appear -- we certainly use that later on. e.g. the HIR (constructed from the AST, not directly from the parser) has a node TyErr that represents an erroneous type.

[Michael-F-Bryan]

I'm moving across a conversation I was having with @mark-i-m about how much we want to explain parser basics.

Should the parsing chapter go into more detail about lexing and parsing or rely on external sources for the basics?

I was asking myself this exact question when I wrote the start of the parser chapter. Should we add a small note up the top saying we assume people know how a basic recursive descent parser works and what tokenizing/lexical analysis is? The idea being this is a book about rustc internals, not a book an introduction to parsing.

There is already loads of good quality material on basic parsers on the internet, a couple paragraphs at the top of the chapter probably wouldn't be able to do it justice.

I agree that we shouldn't try to teach parsing here, but given that I don't expect most people to know basic parsing, I worry that it would discourage contributions... Perhaps we can

• Add a high level overview of the algorithm and point to a few solid resources for learning in detail
• Give some key term definitions
• Tie them all back to the code

What do you think?

---

Sounds like a good idea. I was thinking we could say something like this:

Rust syntax is specified by a grammar (link) which is essentially a list of rules where each rule specifies how a piece of the language is written (e.g. a crate contains multiple items, an item may be a function declaration, a function has a bunch of statements, a statement is a ...), with each rule being written in terms of other rules or terminals (the base case, typically tokens).

Generally speaking, for each grammar rule there will be one parser method. In this way we can translate a token stream into an AST by recursively calling the appropriate method.

For people who already know how parsers work it's essentially recursive descent 101, but something like that is probably complete enough to give people the general idea, but short enough to not be off topic.

You could then tie all of this back to rustc by inspecting a sample code snippet (e.g. an if statement) and then showing what would be called when parsing it. This probably also shows how you deal with ParseSess and CodeMaps.

[mark-i-m]

Copying over Niko's comment from #26 :

I agree that we shouldn't try to teach parsing here, but given that I don't expect most people to know basic parsing, I worry that it would discourage contributions... Perhaps we can

I think I agree with both of you. I don't think we want a lot of introductory material; a few links don't hurt, but not too much. But I think there's a third way, though it may take some iteration to get there: To some extent, I think you can serve both audiences by doing a kind of "walk through" of the code.

In other words, e.g. to explain tokenizing, we might point to the token data structure and give some source showing how it would be divided into tokens (we can always link to wikipedia or something too). This way, if you know what a token is, you learn about the Rust-specific parts of it. If you don't know what a token is, you can just understand it as this Rsut data structure and later learn about the more general form.

Similarly I imagine we can say something like "Rust has a recursive-descent parser" (where we link to wikipedia) and then walk through how it would parse some small example, showing a few key functions (eg., the one that parses a type). If you're not familiar with recursive descent, this will basically give you the idea, but if you are, then you'll learn about the names of key concepts in the code.

[mark-i-m]

Triaged: not a whole lot of change. There was some discussion earlier with @chrissimpkins, @matklad and @Centril (who is taking some time off atm). Perhaps Chris can give an update?