[WIP]Add openpangu_dense model

[yzy1996]

What does this PR do?

This PR introduces inference support for the openPangu-Embedded model. By integrating the inference implementation directly into the Transformers library, we aim to ensure seamless compatibility and enable continuous updates alongside future open-source releases.

This implementation requires torch_npu as a dependency. I'd like to confirm whether introducing this NPU-specific dependency aligns with the community's guidelines and acceptance criteria for the Transformers library. Any guidance on this matter would be greatly appreciated.

I welcome any feedback on potential improvements or additional changes that would enhance this implementation. Please feel free to share your suggestions!

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. @ArthurZucker

[Rocketknight1]

Hmm, it seems like this model has the same architecture as Llama, but just adds an NPU operation in attention. I'm not sure we need a whole architecture for that! cc @Cyrilvallez for attention, @ivarflakstad for NPU

[yzy1996]

Thanks for the reply! Due to precision and performance gaps on NPU, we'll gradually add more NPU operators (beyond just attention ops). Although we have the same architecture as Llama, we'd like to add a separate model to keep it updated.

[Cyrilvallez]

Hey @yzy1996! Indeed, as @Rocketknight1 mentioned, modeling code should be agnostic to the device used, so we don't really want special conditional branching based on accelerator type. You can however register custom attention function and masking to use with Llama if you follow what is shown here https://huggingface.co/docs/transformers/en/attention_interface, which will allow you to perform npu-specific optimizations!

[ArthurZucker]

Related to #39238 for NPU we'd love to just have proper kernels support!

[zheliuyu]

Related to #39238 for NPU we'd love to just have proper kernels support!

We are excited to announce NPU support for kernels!

• Add support for NPU kernelize/layers kernels#155
• [Merged through #155] Add support for npu kernelize kernels#146

Example of Usage:

• Add RMSNorm kernels for npu #41098

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, openpangu_dense