Transformer — shared thought trail

Historical context

Introduced by Ashish Vaswani and colleagues in the 2017 paper Attention Is All You Need, the Transformer was initially designed for sequence-to-sequence tasks such as machine translation. It replaced recurrence and convolution with attention, enabling substantially more training computation to run in parallel.[1]

Original design

The original Transformer used an encoder–decoder architecture. Token embeddings were combined with positional encodings to represent sequence order. Six encoder layers applied multi-head self-attention and feed-forward networks; the six-layer decoder added masked self-attention and attention over the encoder’s output.[2] Residual connections and layer normalization surrounded the sublayers.

Attention compared learned queries and keys, then used the resulting weights to combine values across multiple heads. Decoder masking prevented access to future positions, so output was generated one token at a time.[1]