Type ascription syntax can subtly change the behaviour of macros

[jonas-schievink]

In the wild: phsym/prettytable-rs#11

This program prints "1" on stable and beta, but "0" on nightly:

macro_rules! cell {
    ($value:expr) => { 0 };
    ($style:ident : $value:expr) => { 1 };
}

fn main() {
    println!("{}", cell!(t : T));
}

Originally reported by @hoodie

[petrochenkov]

If I'm not mistaken, the macro future proofing RFC should make macros like this illegal?
cc @pnkfelix

[pnkfelix]

Well honestly I'm not sure; that RFC was tightening up rules for a single rewrite rule; I'm not sure how a we handle/check interactions between multiple rules. ..

[durka]

Even with the future-proofing, FOLLOW(ident) is "any token" so this would not be prevented. To fix this : would need to be removed from FOLLOW(ident) (and FOLLOW(block) as well), but that would be another breaking change :( . Of course type ascription is also a breaking change as evidenced by this issue, so it's one against the other. At least perhaps a lint could be added to catch this situation where type ascription influences macro expansion.

[pnkfelix]

I think removing : from FOLLOW(ident) is a bit stronger than necessary. Or at least, maybe we should be thinking more about validating the set of rewrite rules as a whole set of composed alternatives, rather than thinking about them in isolation. (This would likely still introduce breaking change, but hopefully not as significant as throwing more things into the follow set would be.)

[durka]

I guess another (distasteful) option is changing the type ascription syntax so that it would be covered by the extant future-proofing. For example allowing expr-but-not-ident, so ident: ty is an error but (ident): ty is ascripted. [edit: would still break a macro with a pattern like ( $e:expr ): $t:ty but that should be rare and is much harder to prevent]

[hoodie]

Is there a way to know that the second part describes a type?

[durka]

@hoodie what do you mean? Macros are expanded before name resolution and type checking so AFAIK they can only really look at syntax.

[hoodie]

Yes you are right, I thought of that too right after I posted. This just means that the colon character is to be avoided in macros.

[pnkfelix]

validating the set of rewrite rules as a whole set of composed alternatives

To be clear, what I meant by this is that we might be able to walk across all of the rules in parallel, maintaining multiple cursors into the rules, rather than separately with a single cursor, and have the FOLLOW set for each token depend upon where all the cursors are located. Right now the FOLLOW(ident) is the whole universe of tokens. But what I'm thinking is that if you have one cursor pointing at an expr and another at an ident (representing the idea that the macro expander could eventually be trying to decide which rewrite arm to use), then the FOLLOW for that ident would be much more restricted, perhaps e.g. FOLLOW(ident) = FOLLOW(expr) in that context (or maybe it would need to be even more restricted, I'm not sure).

[pnkfelix]

In the meantime, its possible that a workaround could be to just reorder the rewrite arms in the macro definition itself so that the case with expr comes after the case with ident : ...

[DanielKeep]

@pnkfelix I'm afraid macro_rules! is too dumb for that to work:

macro_rules! cell {
    ($style:ident : $value:expr) => { 1 };
    ($value:expr) => { 0 };
}

fn main() {
    println!("{}", cell!(t : T));
    println!("{}", cell!(42));
}

produces the following on stable:

<anon>:8:26: 8:28 error: expected ident, found 42
<anon>:8     println!("{}", cell!(42));
                                  ^~

In cases like this, expr always has to come before ident, or it just won't work.