A vector with no context

Text enters as a sequence of token ids. A token is a piece of a word, sometimes a whole word, sometimes a space and a prefix. An embedding table turns each id into a vector, a list of a few thousand numbers. The same id always starts from the same list. The token for bank is the same vector beside a river and beside a loan.

The block exists to make them different, using the tokens around them. That difference is the contextual representation. The vector stays the same length. The numbers in it now depend on the rest of the sequence.

A transformer stacks identical blocks. Each one reads the vectors the previous block wrote, and writes an update. Two sublayers do the writing. Attention moves information between positions. The feed-forward layer, with SwiGLU inside it, rewrites each position on its own.

Linear layers

A linear layer is a matrix multiplication, plus a bias the model can learn.

y = Wx + b

W is a learned table. Each row asks the input vector one question, and the answer is a weighted sum of the input's numbers. A wide layer asks many questions at once. A narrow one compresses.

The projections that build queries, keys, and values are linear layers. So is the projection that mixes attention heads back together. So is the embedding. Most of the file size of a model is these matrices.

Stacked linear layers, with nothing between them, collapse. Two matrices in a row multiply into one matrix. Depth drawn that way is still a single mixing. The rest of the block exists so the next linear layer sees a vector a matrix on its own would not produce: a vector gated by a nonlinearity, or weighted by other tokens.

Attention

Attention lets one position build a new vector from the values at other positions. The weights of that mix are computed from the tokens themselves.

Each position produces three vectors, through three linear layers. The query is what this position is looking for. The key is the label it offers. The value is the content it hands over if it is chosen.

The score between two positions is their dot product: large when the two vectors point the same way. Scores are divided by the square root of the key's length, so a long vector does not produce huge scores by default and flatten the softmax. Softmax turns a list of scores into positive weights that sum to one. The output is those weights applied to the values. That is the scaled dot-product attention in Attention Is All You Need (Vaswani et al., 2017).

Attention(Q, K, V) = softmax(QKT / √dk) V

The same operation runs in several heads. Each head has its own projections, so one head can track agreement, another a name and the pronoun that follows it, another a bracket that still needs to close. The heads are concatenated and passed through one more linear layer.

In a language model, a position may look at earlier positions only. The vector at that point is a summary of the sequence so far. The rule is the causal mask from the decoder in Vaswani et al., a pattern applied to the scores before the softmax. Part 3 trains against exactly this summary. Predict the next token from it.

The cost follows the mask. Every query is compared with every key it is allowed to see, so the work grows with the square of the sequence length. Part 5 is what you do about that cost when the model is answering one token at a time.

RoPE

A bag of attention weights still has a gap. Nothing yet says whether a word stood before the verb or after it. The model needs position.

Su et al. (2021) write position into the query and the key by rotating them. This is rotary position embedding, RoPE. Take the numbers of a query two at a time. Each pair is a point in a plane. RoPE turns that point by an angle that grows with the token's index. A token at position m is turned by m times a base angle θ. The key at position n is turned by nθ.

The rotation is fixed. The original design has no learned weights in it. Early pairs spin quickly as the index grows. Later pairs spin slowly. A short-range pattern and a long-range pattern can each have a pair of dimensions that lines up for them.

After both vectors are rotated, their dot product depends on the gap m − n, and on the vectors themselves. Attention matches content, and it matches content at a distance. Swap two tokens and the distances change, so the mix changes, even though the embedding table gave each id the same start.

Because the signal lives in a rotation, a model can be stretched to sequences longer than the ones it trained on by rescaling those frequencies. The representation does not depend on a position vector that was only learned up to some maximum index.

SwiGLU

Attention has mixed the neighbors in. The feed-forward layer then works at each position alone. This is where most of the parameters sit, and where the network gets a nonlinearity.

Shazeer (2020) defines SwiGLU, the gated form used in current language models. The gate is Swish: small negative numbers pass through in a smooth, reduced way, and large positive numbers pass almost unchanged. The gate and the values are two different linear projections of the same input. They are multiplied element by element. A third projection maps the result back to the width of the residual stream. Touvron et al. (2023) use this block in LLaMA.

SwiGLU(x) = (SiLU(xWgate) ⊙ xWup) Wdown

The multiplication is the selection. Where the gate is near zero, that feature is dropped. Where the gate is large, the feature passes. Attention selected tokens. SwiGLU selects features. Part 4 adds a third selection, over whole feed-forward experts, in this same slot.

Three matrices would cost more than the two-matrix feed-forward they replaced. Touvron et al. set the inner width to (2/3) × 4d, two thirds of the 4d width PaLM used, so the parameter count stays close. The gate is paid for by a narrower middle.

What one block writes

Before attention, and again before the feed-forward, the vector is rescaled. Zhang and Sennrich (2019) define RMSNorm: divide by the root mean square of the numbers, then multiply by a learned gain on each coordinate. Touvron et al. put that normalization on the input of each sublayer. A linear layer behaves more steadily when its input stays on a familiar scale.

Each sublayer adds its result onto the vector that entered it. The block writes a change, and the incoming representation stays available for the next layer. After a stack of blocks, the vector at a position is the original embedding plus every change the blocks decided to make.

That sum is the contextual representation. This token, given these neighbors, at these distances.

What to take with you

Read a hidden state as a mixture. Linear layers propose it. Attention chooses the tokens that contribute, and RoPE makes that choice sensitive to order and distance. SwiGLU chooses the features that continue. The residual path is why a later layer can still see an early one.

When a model surprises you, the surprise is in those choices. A word changed meaning because attention weighted different neighbors, or because a gate opened a different feature. The rest of this series is what you do once a network can build that vector. Edit a scene, train the choices, skip most of the parameters, and serve the result on hardware you actually have.

Sources

  1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762, 2017.
  2. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864, 2021.
  3. Noam Shazeer. GLU Variants Improve Transformer. arXiv:2002.05202, 2020.
  4. Hugo Touvron and colleagues. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023. RoPE, SwiGLU, and RMSNorm in one widely used block.
  5. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. arXiv:1910.07467, 2019.