Add the GeLU activation from pytorch with the tanh approximation

[jlamypoirier]

Fixes #21344. See that issue for more details.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as gelu_fast? If that is the case, I would prefer we just replace the current gelu_fast with this when PyTorch is 1.12 or above.

[jlamypoirier]

Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as gelu_fast? If that is the case, I would prefer we just replace the current gelu_fast with this when PyTorch is 1.12 or above.

The results are similar but there are still rounding errors, see my analysis in the related issue #21344. I would also be in favor of replacing the existing implementation / using it as default, but I would introduce small numerical differences in some models, is that a problem?

[sgugger]

Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/
So let's go with a new activation. Maybe gelu_pytorch is a better name?

[jlamypoirier]

Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/ So let's go with a new activation. Maybe gelu_pytorch is a better name?

Wouldn't it cause confusion with the default pytorch implementation? That one is currently named "gelu". (And the one named "gelu_python").

Also should I add an explicit pytorch version check?

[sgugger]

Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the PyTorch version is too low, then put that function in the mappinh.

[jlamypoirier]

Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the PyTorch version is too low, then put that function in the mappinh.

Made a class to match the other activations, and raising a NotImplementedError (I don't think an ImportError is the best here since the function exists in earlier versions.) Also added to test_get_activation.

[sgugger]

Thanks for iterating! I just have one last comment on the error raised.

[sgugger]

Suggested change

	raise NotImplementedError(
	raise ImportError(

[jlamypoirier]

All fixed, this should be ready to merge once the tests pass.

[sgugger]

Failure is unrelated so merging. Thanks again for your contribution!