Add support to use config dtype in HybridChunkedCache

[vivekkhandelwal1]

What does this PR do?

This PR fixes the issue with the HybridChunkedCache initialisation since it does not consider the dtype present in the model config. Here:

transformers/src/transformers/cache_utils.py

Line 1806 in b949747

self._dtype = dtype

[vivekkhandelwal1]

Hi @ydshieh @Rocketknight1! It would be great if you can find some time to review this change.

[Rocketknight1]

cc @gante

[vivekkhandelwal1]

Hi @gante, can you please review this PR?

[gante]

Hi @vivekkhandelwal1

I don't think this is the right solution: config.torch_dtype may be out of sync with the model's actual dtype, e.g. when we manually cast a model with model.to(torch.XX) after loading it.

Would you be able to expand the description of the issue you're seeing? I might be able to provide solutions 🤗