Skip to content

Add set_to_none=False for optimizer in workflow #2595

Description

@yiheng-wang-nv

After Pytorch 1.7, Optimizer adds a parameter set_to_none. When setting to True, it may increase the performance.
Some references:

https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.zero_grad.html

https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions