Added separate limits for saving full checkpoints and model weights

[GitGautamHub]

What does this PR do?
This PR introduces separate limits for saving full checkpoints and model weights in the Trainer class. Currently, save_total_limit applies to both, which can lead to inefficient storage management. This change allows users to:

Retain intermediate model weights for analysis without storing full checkpoints unnecessarily.

Ensure that the latest full checkpoint is available for resuming training.

Fixes Issue
Fixes #37195

Motivation
The current implementation of save_total_limit does not differentiate between full checkpoints and model weights. This can be problematic when users require frequent weight saves but only need a limited number of full checkpoints. Introducing separate limits enhances flexibility and optimizes storage usage.

Changes Made
Updated _rotate_checkpoints() in trainer.py to support independent limits for full checkpoints and model weights.

Introduced save_checkpoint_limit and save_model_weights_limit as configurable parameters.

Updated relevant documentation and method docstrings.

Tests
Due to local environment issues, I was unable to run the tests locally.

The CI/CD pipeline should validate the changes.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[Rocketknight1]

cc @muellerzr @SunMarc here and to the original issue #37195

[Aravind-11]

Hi @GitGautamHub , what's the status on this? I can take this up but it looks very complicated since these changes could affect a lot of other files. I don't know if i'm right. Would like your inputs on this.