Attention-based models — shared thought trail

Brief history

Attention emerged in recurrent encoder–decoder models as a way to select the most relevant parts of an input sequence for each output step, rather than relying only on one fixed representation. The original design was RNN cross-attention, not the self-attention architecture later used by Transformers.[1] In 2017, Ashish Vaswani and colleagues made attention the Transformer’s primary computational mechanism.[2]

Original mechanism

At each decoding step, the model:

1. compares the current output state with the encoder’s input representations to produce attention scores;
2. normalizes those scores into weights;
3. takes a weighted sum of the input representations to form a context vector; and
4. uses that context to help predict the next output.

Because the alignment was soft and differentiable, it was learned jointly through backpropagation rather than supplied explicitly.[1]

Lite feature snapshot

• Dynamic context: a new input summary for each output step
• End-to-end learning: attention and prediction are trained together
• Soft alignment: weights indicate which input positions matter most
• Architectural distinction: originally cross-attention within an RNN; later Transformers centered computation on attention