Skip to content

Fix check_quantized_param method when param_value is a safetensors slice#40425

Open
radulescupetru wants to merge 3 commits into
huggingface:mainfrom
radulescupetru:bugfix/check_quantized_param_as_slice
Open

Fix check_quantized_param method when param_value is a safetensors slice#40425
radulescupetru wants to merge 3 commits into
huggingface:mainfrom
radulescupetru:bugfix/check_quantized_param_as_slice

Conversation

@radulescupetru

Copy link
Copy Markdown

What does this PR do?

Fixes a case where when loading a quantized model from safetensors with tensor parallelism enabled (device_mesh!=None) , check_quantized_param(...) method was failing as it was expecting a torch.Tensor as input for param_value and it was supplied with a slice.
More details: #40361

Fixes #40361

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@SunMarc @S1ro1

@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: bnb, eetq_integration, fbgemm_fp8, finegrained_fp8, higgs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Quantized model loading failing with custom device mesh

1 participant