Generate: remove deprecated public decoding functions and streamline logic 🧼

[gante]

What does this PR do?

🧼 🧼 🧼

Calling the internal decoding functions as part of our public API was scheduled for removal in v4.41 (the next release). Its motivation was flexibility and conciseness: having multiple public interfaces for the same functionality forced us to add repeated logic in many places, increasing every time we added a new decoding method.

Due to this removal from the public API, a few things were changed/removed as a logical consequence:

1. No more documentation/examples in the internal decoding functions. We have a page with examples to call the internal decoding methods from generate;
2. The arguments to the decoding functions are no longer optional -- the decoding functions are exclusively called from generate. As such, we can remove a lot of boilerplate (x = x if x is not None else self.generation_config.x).
3. No more tests regarding the internal decoding methods -- already removed in this PR

---

Tests ran locally:

• generate doctests (pytest --doctest-modules src/transformers/generation -vv)
• generate integration tests (RUN_SLOW=1 py.test tests/generation/test_utils.py -vv)
• cache integration tests (RUN_SLOW=1 py.test tests/test_cache_utils.py -vv) -- same failures as in main
• llama slow tests (RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv)
• whisper slow tests (RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv) -- same failures as in main

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

Tests on all models are passing 🙌 (the failing pipeline test seems unrelated, and passing locally on my end)

[zucchini-nlp]

Yaaay, thanks for clean-up! Looks so much nicer

[gante]

functions not public -> no docs -> remove link to the docs

[gante]

(see comment on L1434)

[gante]

Instead of pulling information from generation_config to pass around through model_kwargs, let's use generation_config directly.

A single object to hold all generation parameterization.

[gante]

These were being passed through model_kwargs before

[gante]

This one was not needed -- contrastive search does not sample

[gante]

On most decoding methods eos_token_ids doesn't need to be passed -- it was used when the decoding method was called directly and stopping_criteria was not passed.

However, beam methods still need it.

[gante]

function not public -> let's remove the example (preparing its inputs will be more challenging now, as we no longer have API guarantees)

[gante]

In the main generate body, we set pad_token_id to eos_token_id in this situation -- this exception will never be reached

[gante]

ping @ArthurZucker -- ready for review :)

[ArthurZucker]

Late review but very nice cleanup sir! 🤗

[ArthurZucker]

theoretically some of these can be taken from the config / the generation config if it inherits them. But nit

[gante]

Yes! That's a good idea

[gante]

Reran slow tests locally, all seems good 👍