Deep neural language models — shared thought trail

Deep neural language models

Neural language models emerged as an alternative to count-based n-grams. In 2003, Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin introduced a landmark model that jointly learned distributed word representations and next-word probabilities[1]. Tomáš Mikolov and collaborators later advanced recurrent language models, while Ashish Vaswani and colleagues introduced the Transformer in 2017[2]. Ilya Sutskever is also an important figure in the broader development of deep sequence modeling.

Originally, the model took a fixed window of previous words, mapped them to learned embeddings, passed them through a feed-forward neural network, and used a softmax layer to predict the next word. Training learned both the word embeddings and prediction network together[3]. Similar words acquired similar representations, allowing better generalization than count-based methods[4].