Round and sign straight-through-estimators C operators.

[igolan]

Description

Implemented sign and round straight-through-estimators operators in C.
Straight-through-estimators have derivative of 1 everywhere instead of 0 everywhere, this is required for quantized training.

Checklist

Essentials

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the my best knowledge, examples are not affected by this change.

Changes

• contrib.round_ste() including test and API doc
• contrib.sign_ste() including test and API doc

Comments

N/A

[sxjscience]

@szhengac @xidulu I think the stochastic version of straight-through estimator is related to the distribution package.

[igolan]

@sxjscience , just to clarify, it's not a stochastic version of round/sign, it's just round and sign with derivative of 1 (instead of 0).

[sxjscience]

@igolan Yes, just to ping the other people to see if we could merge the efforts.

[xidulu]

@sxjscience
You are right, straight through estimator could be useful when performing hard assignment on relaxed discrete variable. (e.g. https://arxiv.org/pdf/1611.01144.pdf section 2.2).