Fix the seamless_m4t cannot work on Gaudi

[yuanwu2017]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Rocketknight1]

cc @IlyasMoutawwakil I think? Feel free to ping me if this should go to someone else!

[IlyasMoutawwakil]

@yuanwu2017 can be more specific about what you're fixing here ?

[yuanwu2017]

@IlyasMoutawwakil The torch.gather cannot work correctly on Gaudi with seamless_m4t. The patch is only for tracking this issue. I will closed it.

[yuanwu2017]

git clone https://github.com/yuanwu2017/llm-dbg.git
PT_HPU_LAZY_MODE=1 run_text-to-speech.py
Before:

After:

[yuanwu2017]

@IlyasMoutawwakil Can we merge this patch first? The root cause is that Gaudi torch.gather cannot support input of int64 or long . It seems slow to fix this bug from the Gaudi pytorch. After they fix, I will remove the workaround.

[yuanwu2017]

@yao-matrix

[IlyasMoutawwakil]

@yuanwu2017 you can patch it in the is_torch_hpu_available method, the same way we patched masked_fill_ :

transformers/src/transformers/utils/import_utils.py

Line 840 in 37367c7

def patched_masked_fill_(self, mask, value):

the transformers modeling code can't afford to be bloated with accelerator specific patches.

[yuanwu2017]

Done.

[yuanwu2017]

Test passed.

[yuanwu2017]

Done.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

sorry I missed this.

You might consider open an issue to track the patched stuffs and remove them once things are fixed (as mentioned in the added comments)

[yuanwu2017]

Done.