Starcoder2 model - bis

[RaymondLi0]

The Starcoder2 model, adapted from Mistral. All changes are done through options, so Mistral itself is still supported.
Main changes:

• Use layer norm
• Use standard MLP
• Add biases (optional)
• Change (default?) tokenizer class
  *Embedding and residual dropout

It does not support absolute embeddings, so can't support Santacoder or Starcoder

Starcoder2-3B model: https://huggingface.co/bigcode/starcoder2-3b

Todo:

• Forward changes from [Core generation] Adds support for static KV cache #27931, [CLeanup] Revert SDPA attention changes that got in the static kv cache PR #29027 (and future changes from Feb. 19) (in a future PR?)
• Documentation
• Copyright
• Point to starcoder2 checkpoint
• Other minor things (see todos)

@younesbelkada @ArthurZucker @jlamypoirier

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

In #29228 I mention that static cache is not a blocker for the PR 😉

[younesbelkada]

Thanks ! LGTM once we add the docs !

[ArthurZucker]

Let's go!

[ArthurZucker]

this is fairly short. Let's add that the main difference with mistral is dropout, as the authors would be nice to explain how much this influenced training for example.

[younesbelkada]

yeah makes sense, I will take care of that after the official release

[ArthurZucker]

Thank you for the PR @RaymondLi0. cc @loubnabnl and @lvwerra, next time let's make sure doc and paper links are fully completed before we merge, we require this for every model / organisation regardless of the release date 😉 let's make this an exception