Skip to content

Allow quantizers to work with a state dict on meta device#37133

Closed
Cyrilvallez wants to merge 3 commits into
mainfrom
fix-quantizer
Closed

Allow quantizers to work with a state dict on meta device#37133
Cyrilvallez wants to merge 3 commits into
mainfrom
fix-quantizer

Conversation

@Cyrilvallez

@Cyrilvallez Cyrilvallez commented Mar 31, 2025

Copy link
Copy Markdown
Member

What does this PR do?

Following #35926 and #37086, make the quantizers work with a state dict on meta device.
See #37086 for the issue at hand

Currently, will lead to inconsistencies and params initialized with random weights

cc @SunMarc @MekkCyber is there any way to replace params with only param_value instead of doing it for a whole layer at once with the state_dict for hqq and bnb 4bits? I believe this should be possible, but would require to know a bit more about internals of each lib for me.
Otherwise if truly not possible we can add an exception for these 2 types of quantizers, but it would be so much simpler if we can simply standardize and just replace one param after the other, instead of doing a full layer at once

@github-actions github-actions Bot marked this pull request as draft March 31, 2025 10:15
@github-actions

Copy link
Copy Markdown
Contributor

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@Cyrilvallez Cyrilvallez marked this pull request as ready for review March 31, 2025 10:16
@github-actions github-actions Bot requested review from MekkCyber and SunMarc March 31, 2025 10:17
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc

SunMarc commented Mar 31, 2025

Copy link
Copy Markdown
Member

run slow: bnb

@github-actions

Copy link
Copy Markdown
Contributor

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: []
quantizations: ['quantization/bnb'] ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants