Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add hyperparameter principles #435

Closed
wants to merge 1 commit into from
Closed

add hyperparameter principles #435

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions training_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,11 @@ CLOSED:

By default, the hyperparameters must be the same as the reference.

Hyperparameters by default must be as constrained as possible for the benchmark to be fair and focused on system performance, not algorithmic tricks.
Changes to hyperparameters are allowed when both:
1. the proposed hyperparameter change has demonstrated to reduce/improve samples to convergence on the reference (on some portion of the batch size range, if not the whole range), AND
2. the proposed hyperparameter change has reasonable evidence of industry adoption.

Hyperparameters include the optimizer used and values like the regularization norms and weight decays.

The implementation of the optimizer must match the optimizer specified in the Appendex: Allowed Optimizer. The Appendex lists which optimizers in the popular deep learning frameworks are compliant by default. If a submission uses an alternate implementation, the submitter must describe the optimizer's equation and demonstrate equivalence with the approved optimizers on that list.
Expand Down
Loading