Floating point, overflow, the log-sum-exp trick and the reason your loss became NaN.
4h · 6 lessons · 1 challenge
Before this: Probability
Nothing in the source courses teaches this, and everything in them depends on it. Softmax overflows. Cross-entropy takes the log of a zero. A sum of a million small numbers loses its tail. Mixed precision halves your memory and quietly changes your answer.
This module is ours, and it is here because a NaN at step 4000 is otherwise unexplainable.
A training run produces NaN at step 4000. Find the operation that did it, from the tensors alone.
Worth 350 XP.
