The bottleneck, the attention that removed it, and the architecture that dropped recurrence entirely.
11h · 13 lessons · 1 challenge
Before this: Word Embeddings
Attention is not a trick added to encoder-decoder models; it is the answer to a specific failure, and this module shows the failure first. A fixed-size context vector cannot carry a long sentence, and attention is what replaces it.
Then self-attention, multi-head attention, and the transformer built from them -- positional encoding, the block, and the whole network assembled from the paper.
The pieces of a transformer block, out of order. Assemble it, with the residuals and norms in the right places.
Worth 500 XP.
