Error index test integration needs to be more robust

[alexcrichton]

We've been plauged today with problems related to the error index tests. Today the way these tests work are:

• During compiling a crate with error codes, extra diagnostic information is emitted to a tmp/ directory.
• Later the error_index_generator program runs, processing this diagnostic information and generating error-index.md
• Then rustdoc is used to test this markdown file.

This setup is brittle for a few reasons:

• Running the error index generator doesn't actually depend on the compilations of each crate, so you just have to conveniently sequence it after compiles have finished to get it to actually work.
• The probing logic of the error index generator is not precise, it simply slurps up anything that looks like diagnostics. This means that if there's a bad or stale diagnostic file it'll get slurped up nonetheless.
• Emitting files always has fun complications when you run processes concurrently, and this in the past forced us to fix this emission for multi-host builds (e.g. the nightlies)

The compiler should not jettison json files out the back when compiling and instead the error index generator should process the diagnostics directly. This sidesteps sequencing problems, stale problems, etc, and should be more robust.

[Mark-Simulacrum]

I believe that these problems still exist today. I feel like with today's error diagnostics being generated through a Rust plugin (as far as I can tell), this is quite hard to change.

@alexcrichton Do you feel like the right approach here would instead be to change the error format that currently exists in Rust to be stored in something like a directory tree and then loaded (perhaps by the same plugin as now) into Rust code? This tree could then also be parsed by the error index generator separately. If not, what do you think needs to be done to resolve this issue?

[alexcrichton]

What I think should be done to solve this:

• Add a build script to the error index test generator and/or error index generator which probes for diagnostic.rs scripts throughout the repository.
• Add definition of the register_long_diagnostics! and reigster_diagnostics macro in the error index generator.
• include! all diagnostics, using the local deifnition of registering diagnostics.

That way this doesn't rely on any artifacts or compilations at all and goes straight to the source.

[Mark-Simulacrum]

#35284 notes that some long diagnostics aren't even registered properly and can't be used with rustc --explain today. I've closed that issue in favor of this, but something to keep in mind.

[venkatagiri]

@Mark-Simulacrum I would like to work on this.
I was giving it a try using steps provided by @alexcrichton here.
Code I tried is https://gist.github.com/venkatagiri/c6fdf659f55963c0bbd2210289baca4f

Trying to compile this, am getting below error. Copying the diagnostics into the main works. But, importing them through include! is failing.

Am not sure what I am doing wrong.

Full error is

error: macro expansion ignores token `register_diagnostic` and any following
  --> t.rs:25:15
   |
25 |             $(register_diagnostic! { $code, $description })*
   |               ^^^^^^^^^^^^^^^^^^^
   |
note: caused by the macro expansion here; the usage of `register_long_diagnostics!` is likely invalid in expression context
  --> ./diagnostics.rs:1:1
   |
1  | / register_long_diagnostics! {
2  | |     E001: r##"hello"##,
3  | |     E002: r##"world"##,
4  | | }
   | |_^

note: trace_macro
  --> t.rs:29:5
   |
29 |     include! { "./diagnostics.rs" }
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: expanding `register_long_diagnostics! { E001 : r##"hello"## , E002 : r##"world"## , }`
   = note: to `register_diagnostic ! { E001 , r##"hello"## } register_diagnostic ! {
           E002 , r##"world"## }`
   = note: expanding `register_diagnostic! { E001 , r##"hello"## }`
   = note: to `println ! ( "code is {}" , stringify ! ( E001 ) ) ;`
   = note: expanding `println! { "code is {}" , stringify ! ( E001 ) }`
   = note: to `print ! ( concat ! ( "code is {}" , "\n" ) , stringify ! ( E001 ) )`
   = note: expanding `print! { concat ! ( "code is {}" , "\n" ) , stringify ! ( E001 ) }`
   = note: to `$crate :: io :: _print (
           format_args ! ( concat ! ( "code is {}" , "\n" ) , stringify ! ( E001 ) ) )`

error: aborting due to previous error(s)

[Mark-Simulacrum]

I suspect the problem is that I think include! can only expand to one expression or one item; perhaps we could somehow facilitate that? I don't think it'll be easy, unfortunately :/

[venkatagiri]

@alexcrichton The steps you mentioned here didn't work out. Any other suggestions on how to solve this one?

[alexcrichton]

@venkatagiri does #[path = "..."] mod foo work if include! doesn't?

[venkatagiri]

@alexcrichton Changed to #[path = "diagnostics.rs"] mod diags; and it is still failing but with a different here. Not sure how to fix it. Code here

vagrant@rust-lang error_index $ cargo run
   Compiling error_index v0.1.0 (file:///home/vagrant/repos/error_index)
register_long_diagnostics! { E001 : r##"hello"## , E002 : r##"world"## , }
register_diagnostic! { E001 , r##"hello"## }
println! { "code is {}" , stringify ! ( E001 ) }
error: macros that expand to items must either be surrounded with braces or followed by a semicolon
 --> <println macros>:3:16
  |
3 | ) => ( print ! ( concat ! ( $ fmt , "\n" ) , $ ( $ arg ) * ) ) ;
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

print! { concat ! ( "code is {}" , "\n" ) , stringify ! ( E001 ) }
error: expected one of `!` or `::`, found `(`
 --> <print macros>:2:25
  |
2 | $ crate :: io :: _print ( format_args ! ( $ ( $ arg ) * ) ) ) ;
  |                        -^ unexpected token
  |                        |
  |                        expected one of `!` or `::` here

error: Could not compile `error_index`.

[alexcrichton]

Ah sorry, I'm running out of ideas :(

[Mark-Simulacrum]

I think if you change https://gist.github.com/venkatagiri/c6fdf659f55963c0bbd2210289baca4f#file-main-rs-L10-L17 to the following by adding a block around the macro body it'll work.

    macro_rules! register_diagnostics {
        ($($code:tt),*) => {{
            $(register_diagnostic! { $code })*
        }};
        ($($code:tt),*,) => {{
            $(register_diagnostic! { $code })*
        }};
    }