Mini-batches, momentum, Adam, schedules, and the hyperparameter search that finds the run.
9h · 12 lessons · 1 challenge
Before this: Making Deep Networks Train
Plain gradient descent is not what trains a modern network. This module builds the alternatives in the order they were invented and for the reasons they were: mini-batches for throughput, exponentially weighted averages for noise, then momentum, RMSProp and Adam as three answers to the same problem.
Then the tuning: which hyperparameters matter, on what scale to sample them, and how batch normalization changes the answer.
Five training runs, one learning rate each. Find the sweet spot before you run out.
Worth 450 XP.
