Transformers Explained: The Discovery That Changed AI Forever

Feedforward networks struggled to retain context across a sentence, while RNNs processed words one at a time and could lose learning signals over long sequences. In the 1990s, Sepp Hochreiter and Jürgen Schmidhuber proposed LSTMs, whose gates control what information to keep, update, or forget. GPU acceleration and larger datasets made LSTMs practical for NLP in the early 2010s.

Early sequence-to-sequence translation models compressed an input sentence into one fixed-size vector, which made long or complex sentences difficult to represent. In 2014, Bahdanau, Cho, and Bengio showed how a decoder could attend to an encoder’s hidden states and learn alignments between input and output words. Neural translation then competed with leading statistical systems, and Google Translate adopted a neural sequence-to-sequence architecture around this period.

Attention improved alignment, but RNNs still processed tokens sequentially, limiting parallel computation and slowing training as sequences grew. In 2017, Google researchers introduced the Transformer, which uses self-attention to update separate token embeddings and process a sequence in parallel. The presenter says it was faster than RNNs and more accurate on machine translation benchmarks.

Researchers explored encoder-only models such as BERT and decoder-only autoregressive models such as OpenAI’s GPT series. GPT models trained on larger datasets became the basis for today’s large language models, including those used in ChatGPT and Claude. Earlier NLP models were generally tailored to individual tasks, before large-scale autoregressive training made models more general-purpose.