Transformers and Language Modeling
Practical AI for Biomedical Researchers
Valentina Sora
A Refresher: Vectors and Matrices
A vector is a list of numbers, written as a column. This one has three.
It is also an arrow in space: each number says how far to go along one axis (the coloured steps).
A matrix transforms vectors. This one turns the whole space a quarter turn around the vertical axis: its columns are where the axes' unit arrows land.
Multiplying moves the arrow, and the space with it: the same steps, taken along the turned axes, land on Rv = (−1, 3, 2).
What is GPT?
Generative
The model is made to generate new text.
Pretrained
The model has been trained on massive amounts of data and can be further trained (“fine-tuned”) for specific tasks.
Transformer
This refers to the specific architecture used in the model, the “transformer”.
Next Word (or “Token”) Prediction
“The answer to Life, the Universe, and Everything is 42”
The Importance of Context
“The cat sat on the mat because it was tired.”
Context is necessary to correctly parse the grammatical, syntactical, and semantic meaning of the components of a sentence.
To capture it, we need direct connections between all positions in the sentence. The attention mechanism provides them: it was initially proposed for computer vision [1] and then applied to language.
In plain words, attention is a context-dependent reweighting of information, or how much each position in the sentence should “look” at any other position to infer its own meaning.
[1] Larochelle, Hugo, and Geoffrey E. Hinton. “Learning to combine foveal glimpses with a third-order Boltzmann machine.” Advances in neural information processing systems 23 (2010).
Transformers
Transformers work by iteratively providing the answer to the question “what is the most probable next token (word, punctuation, etc.) that follows the input processed so far?” [1]
[1] Vaswani, Ashish, et al. “Attention is all you need.” Advances in Neural Information Processing Systems 30 (2017).
Transformer Architecture
Embedding
The embedding procedure converts each element of the input sentence (a “token”) into a vector.
Transformer blocks
They mix the information from the different positions in the sentence.
Output probabilities
They determine the likelihood of each possible next token (word, punctuation, etc.).
The Embedding Space
Each token's embedding is a point in a space of 768 dimensions. Here, it is seen along three directions: the two along which these words differ most, and the one from singular to plural.
Words with related meanings sit close together: fantasy, Star Wars, Star Trek, and other science fiction each gather in their own region.
Directions carry meaning, too: the arrows from each word to its plural all point roughly the same way.
We can even do arithmetic with meanings: sword − knight + Jedi lands closest to lightsaber, out of all 50,257 tokens in the vocabulary.
The Importance of Tokenizers
They define the model’s input alphabet (and whether “unknown words” exist). [1]
They control sequence length, meaning that they influence compute costs and context usage. [2] [3]
They encode modeling assumptions about what units matter. [4]
[1] Sennrich, Rico, Barry Haddow, and Alexandra Birch. “Neural machine translation of rare words with subword units.” Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers). 2016.
[2] Xue, Linting, et al. “ByT5: Towards a token-free future with pre-trained byte-to-byte models.” Transactions of the Association for Computational Linguistics 10 (2022): 291-306.
[3] Clark, Jonathan H., et al. “Canine: Pre-training an efficient tokenization-free encoder for language representation.” Transactions of the Association for Computational Linguistics 10 (2022): 73-91.
[4] Kudo, Taku, and John Richardson. “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.” arXiv preprint arXiv:1808.06226 (2018).
The Challenges of Tokenizers
Tokenization bias exists across languages and scripts. [1]
They can be brittle to small input changes (typos, casing, spacing, Unicode normalization). [2]
Numbers, dates, and structured strings can be difficult to meaningfully tokenize. [3]
Tokenization can change “meaning” in ways you don’t intend. [4]
[1] Petrov, Aleksandar, et al. “Language model tokenizers introduce unfairness between languages.” Advances in Neural Information Processing Systems 36 (2023).
[2] Chai, Yekun, et al. “Tokenization falling short: On subword robustness in large language models.” Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
[3] Spathis, Dimitris, and Fahim Kawsar. “The first step is the hardest: Pitfalls of representing and tokenizing temporal data for large language models.” Journal of the American Medical Informatics Association 31.9 (2024): 2151-2158.
[4] Haslett, David A. “Tokenization changes meaning in large language models: Evidence from Chinese.” Computational Linguistics 51.3 (2025): 785-814.
Why Attention? Meaning Depends on Context
Before the first block, “force” is the same vector in every sentence: its embedding knows nothing about the words around it.
Block by block, it moves toward its meaning in each sentence: the three Star Wars sentences travel together, and so do the physics and the army ones.
Each block makes two moves: attention (purple arrows) brings in information from the other words; the MLP (blue arrows), which sees each word alone, builds on it.
Real GPT-2 values for blocks 0–6 of 12, in 9 sentences ending in “force”. Positions are relative to the sentences' average at each block; the axes separate the three meanings.
The Limits of Transformers
Attention is computationally expensive (it scales quadratically with the context window). [1]
Retrieval accuracy drops sharply when relevant information is in the middle of long inputs, even for models nominally supporting long contexts. [2]
“Unlearning” specific information (because of factuality issues, privacy concerns, etc.) is very difficult. [3] [4]
Interpretability is still unsolved. [5]
Hallucinations: confident, plausible-sounding, but wrong outputs. They may be an innate feature of transformer-based architectures. Reasoning models may avoid them, but at a high computational cost (tokens in the chain-of-thought prompt). [6] [7]
[1] Vaswani, Ashish, et al. “Attention is all you need.” Advances in Neural Information Processing Systems 30 (2017).
[2] Liu, Nelson F., et al. “Lost in the middle: How language models use long contexts.” Transactions of the Association for Computational Linguistics 12 (2024): 157-173.
[3] Yao, Yuanshun, Xiaojun Xu, and Yang Liu. “Large language model unlearning.” Advances in Neural Information Processing Systems 37 (2024): 105425-105475.
[4] Wu, Ruihan, et al. “Evaluating deep unlearning in large language models.” arXiv preprint arXiv:2410.15153 (2024).
[5] Räuker, Tilman, et al. “Toward transparent AI: A survey on interpreting the inner structures of deep neural networks.” 2023 IEEE conference on secure and trustworthy machine learning (SaTML). IEEE, 2023.
[6] Peng, Binghui, Srini Narayanan, and Christos Papadimitriou. “On limitations of the transformer architecture.” First conference on language modeling. 2024.
[7] Xu, Ziwei, Sanjay Jain, and Mohan Kankanhalli. “Hallucination is inevitable: An innate limitation of large language models.” arXiv preprint arXiv:2401.11817 (2024).
Newer Transformer Architectures
Rotary positions
Queries and keys are rotated by their position, instead of adding a position vector (Llama 3). Attention then depends on how far apart two tokens are, not on where they sit, and no fixed-size position table is needed.
Grouped-query attention
Several query heads share one key and value head, so less memory is needed to store the past tokens.
Sliding-window attention
Most layers only attend to recent tokens (Gemma 3), while a few see everything.
Latent attention
Keys and values are compressed before being stored (DeepSeek V3, Kimi K2).
Mixture of experts
There are many MLPs (“experts”), of which a router uses a few per token (DeepSeek V3).
Hybrid recurrent layers
Most layers keep a fixed-size running memory instead of attention (Qwen3-Next).
Content taken from the Ahead of AI blog by Sebastian Raschka
Beyond Language: Transformers for Scientific Applications
Structural Biology
AlphaFold (and later versions) performed atomic-accuracy 3D structure prediction from amino-acid sequence, the breakthrough that “solved” a 50-year-old grand challenge. [1]
Computational Chemistry
ChemBERTa and the SMILES Transformer proved useful for molecular property prediction. [2] [3]
Math
AlphaGeometry pairs a transformer architecture with a deduction engine. It solved 25 of 30 recent International Mathematical Olympiad geometry problems. [4]
[1] Jumper, John, et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596.7873 (2021): 583-589.
[2] Chithrananda, Seyone, Gabriel Grand, and Bharath Ramsundar. “ChemBERTa: large-scale self-supervised pretraining for molecular property prediction.” arXiv preprint arXiv:2010.09885 (2020).
[3] Honda, Shion, Shoi Shi, and Hiroki R. Ueda. “SMILES transformer: Pre-trained molecular fingerprint for low data drug discovery.” arXiv preprint arXiv:1911.04738 (2019).
[4] Trinh, Trieu H., et al. “Solving olympiad geometry without human demonstrations.” Nature 625.7995 (2024): 476-482.