Pretrained LLM — shared thought trail

Historical context

Pretrained LLMs marked a shift from separate task-specific NLP systems toward reusable models trained broadly on unlabeled text. GPT’s 2018 work helped establish generative pretraining followed by task-specific fine-tuning.[1]

Original approach

Early GPT-style models used a decoder-only Transformer to predict the next token. Masked self-attention prevented access to future tokens, while gradient descent tuned the model across a large text corpus. The pretrained model was then adapted to labeled downstream tasks with a small output layer and supervised fine-tuning.[2]