Transformer (architecture)

Transformer is a neural network architecture created by Google in 2017 (paper 'Attention Is All You Need') that serves as the foundation for all modern language models — both generative (GPT-4, Claude, Gemini) and embedding-based (BERT, Jina).

The key innovation is that the attention mechanism allows the model to analyze relationships between ALL words in a sentence simultaneously, rather than reading sequentially. This means the word 'bank' has different vectors in the sentences 'bank on the river' (riverbank) and 'bank downtown' (financial institution) — solving Word2Vec's problem. BERT (2018) is a bidirectional Transformer (reads both ways), while GPT is a unidirectional Transformer (generates text left to right).

For SEO: Transformer-based embedding models understand context, enabling semantic search, clustering, and cannibalization detection. In practice, you don't need to understand Transformer mathematics — it's enough to know that thanks to it, embeddings understand CONTEXT, not just words.