Specify rust toolchain explicitly, document how to change it

[alamb]

Which issue does this PR close?

• closes Buildable release builds #14479

Rationale for this change

Currently DataFusion's CI runs with whatever the most recent release is. This ensures DataFusion always works with the latest rust release but causes some non trival churn:

Magically broken Releases

Currently whenever a new rust version is released we often scramble to fix the CI (for new clippy lints for example).

Some recent examples

• Fix clippy for Rust 1.84 #14065
• Apply clippy fixes for Rust 1.83 #13596
• Fix clippy errors for Rust 1.80 #11654

Old releases don't pass CI cleanly

As @findepi observes on #14479, always using the most recent Rust release means that older releases no longer cleanly pass CI tests

What changes are included in this PR?

We can solve this by specifying a specific version of Rust using rust-toolchain.toml
https://rust-lang.github.io/rustup/overrides.html

We use this in influxdb_iox and it works well for avoiding surprise brekages

The downside is that to use a newer version of rust with DataFusion will require a PR to update the config file

Changes:

1. Add a rust-toolchain.toml file
2. Add instructions on how to update it

Are these changes tested?

by CI
(testing)

Are there any user-facing changes?

Hopefully a more controlled experience

[findepi]

Does this apply to CI only, or is rust-toolchain read by cargo too?

[alamb]

I think it applies by default if you just run cargo locally if that is what you are asking

So unless someone overrides the setting by default local development environments would use the toolchain version specified by this file

[alamb]

I plan to merge this PR over the next day or two unless there are any additional comments

[alamb]

Let's give it a try

[alamb]

Thanks everyone for the reviews

[findepi]

Now cargo will auto-update my local toolchain.

$ cargo test --test sqllogictests  -- map.slt --complete
info: syncing channel updates for '1.84.1-aarch64-apple-darwin'
info: latest update on 2025-01-30, rust version 1.84.1 (e71f9a9a9 2025-01-27)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
...

very cool!