A demo version of my notebook: English captions enter an encoder, attention decides what to look at, and the decoder produces French one token at a time.
Multi30k
paired caption dataset
LSTM + attention
custom encoder-decoder
BLEU + heatmaps
evaluation and inspection
Live translation trace
notebook-derived exampleEnglish source tokens
Encoder states
French decoder output
Currently explaining un: the darker English tokens receive more attention for this French output step.
Part-by-part walkthrough
Each step has a simple explanation, a short source snippet, and a talk track you can use when presenting it.
3. Attention
Attention prevents the decoder from relying on one fixed sentence vector. At each generated word, it scores the encoder outputs, masks padding, normalizes with softmax, and creates a context vector.
How I would explain it
The heatmap below is the story: for each French word, the model decides which English states deserve attention right now.
attention_scores = torch.bmm(
encoder_outputs,
weights.unsqueeze(-1)
).squeeze(-1)
masked_scores = attention_scores.masked_fill(mask == 0, float("-inf"))
attention_weights = F.softmax(masked_scores, dim=-1)
context = torch.sum(encoder_outputs * attention_weights.unsqueeze(-1), dim=1)Attention lab
This heatmap is an interactive reconstruction of the attention logic from the workbook. It makes the mechanism explainable without needing to run the training notebook live.
Example sentence
Attention scoring
Bilinear attention adds a learned projection before scoring, so the alignment can become sharper than a raw dot product.
Decoding playground
The decoder produces a probability distribution. The decoding strategy turns that distribution into the next token. Changing the policy changes how deterministic or exploratory the output feels.
Candidate distribution
What this demonstrates
The page is deliberately built around explanation: not just what the code does, but why each choice changes translation behavior.
Encoder-decoder LSTM, source and target tokenizers, output projection.
Dot-product versus bilinear scoring, padding masks, softmax, context vectors.
Greedy, top-k, and top-p as different ways to sample next-token probabilities.
Early stopping on validation BLEU, checkpoint testing, and heatmap inspection.