Skillversitas

Attention and Transformers

The attention is the memory the RNN's replace: the Q the K the V (the query, the key, the value), the score the softmax(QK^T / sqrt(d)) the weight, and the output the weight @ V (the token the all the mix). The transformer is the attention the only (the no the recur, the parallel the train), the MultiHead the 8 the subspace (the dim the split), the Positional the sin (the order the inject), and the GPT the decoder (the causal the mask). This chapter covers the scaled dot-product (the QKV, the softmax), the MultiHead (the 8, the subspace), the Positional (the sin), and the GPT (the causal the mask, the LM the scale).

The QKV and the scaled dot-product

# the SCALED DOT (the):
import torch
import torch.nn as nn
import math

def attention(q, k, v):
    d_k = q.size(-1)
    score = q @ k.transpose(-2, -1) / math.sqrt(d_k)
    weight = score.softmax(dim=-1)
    return weight @ v

#   the
q = torch.randn(8, 20, 64)   # the the (B, T, d)
k = torch.randn(8, 20, 64)
v = torch.randn(8, 20, 64)
out = attention(q, k, v)
print(out.shape)   # the the (8, 20, 64) (the the
                   #   the)
#   the
#   the
#   the

The scaled dot-product is the attention: the Q the query (the token the seek), the K the key (the token the offer), the V the value (the token the data), and the score = Q @ K^T / sqrt(d) the softmax the weight (the sqrt(d) the scale, the softmax the sat the prevent) - the output = weight @ V (the token the all the mix, the 1 step the RNN's T step the replace). The causal is the mask (the future the -inf, the LM the no the peek), and the MultiHead is the 8 the subspace: the 64 dim the 8 the 8 the split, the head the attention the parallel, the concat the Linear (the relation the multi, the 1 head the average the no) - the next section's Positional is the order, and the GPT the block.

The Positional and the GPT

# the POSITIONAL (the):
#   the
class Pos(nn.Module):
    def __init__(self, dim=64, max_len=512):
        super().__init__()
        pe = torch.zeros(max_len, dim)
        pos = torch.arange(0, max_len).unsqueeze(1).float()
        div = torch.exp(
            torch.arange(0, dim, 2).float()
            * (-math.log(10000.0) / dim))
        pe[:, 0::2] = torch.sin(pos * div)
        pe[:, 1::2] = torch.cos(pos * div)
        self.register_buffer("pe", pe)
    def forward(self, x):
        return x + self.pe[:x.size(1)]
#   the

# the GPT BLOCK (the):
#   the
class Block(nn.Module):
    def __init__(self, dim=64, head=8):
        super().__init__()
        self.attn = nn.MultiheadAttention(
            dim, head, batch_first=True)
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(
            nn.Linear(dim, dim * 4), nn.GELU(),
            nn.Linear(dim * 4, dim))
    def forward(self, x):
        a, _ = self.attn(x, x, x)
        x = self.norm1(x + a)      # the
        x = self.norm2(x + self.mlp(x))   # the
        return x
#   the

The Positional is the order: the sin/cos the pos (the max_len the precompute, the register_buffer the no the param), and the x + pe the emb the add (the attention the permute the invariant, the pos the inject) - the relative the learn (the T5 the alternative). The GPT block is the attention plus the MLP: the MultiheadAttention(dim, 8) the causal the mask (the is_causal=True), the LayerNorm the pre (the stability, the residual the x + attn), and the MLP the 4x the dim (the 64 to the 256 to the 64, the GELU the ReLU the smooth) - the 6 the block the stack, the LM the head the vocab, the GPT-2 the 124M the pattern the same the 10x the scale. The rule is the one: the QKV the attention, the MultiHead the subspace, the Positional the order, the causal the LM - the next chapter's optimization is the AdamW the Adam's upgrade, and the scheduler the warmup the decay.

The Working Rules

  • The QKV is the attention, the sqrt(d) the scale. The Q the query, the K the key, the V the value, the score = QK^T / sqrt(d) the softmax (the sat the prevent), and the output = weight @ V (the 1 step the RNN's T step the replace, the parallel the train).
  • The MultiHead is the 8, the subspace. The 64 dim the 8 the 8 the split, the head the attention the parallel, the concat the Linear - the relation the multi, the 1 head the average the no, and the is_causal=True the LM the no the peek.
  • The Positional is the sin, the order. The sin/cos the pos (the max_len the precompute, the register_buffer), and the x + pe the emb (the attention the permute the invariant, the pos the inject) - the relative the learn the T5's alternative.
  • The block is the attention plus the MLP. The MultiheadAttention plus the LayerNorm the residual (the x + attn), the MLP the 4x (the 64 to the 256), and the 6 the block the stack (the GPT-2 the 124M the pattern the same the scale) - the optimizer chapter the AdamW, the scheduler the warmup.

Summary

The attention is the memory the RNN's replace: the Q the query, the K the key, the V the value, the score = QK^T / sqrt(d) the softmax the weight (the sqrt(d) the scale, the sat the prevent), and the output = weight @ V (the token the all the mix, the 1 step the RNN's T the replace). The MultiHead is the 8 the subspace (the 64 dim the split, the relation the multi), the Positional the sin/cos (the order the inject, the permute the invariant), and the GPT block is the attention plus the MLP (the 4x the dim, the LayerNorm the residual, the is_causal=True the LM) - the 6 the block the stack, the GPT-2 the 124M the pattern the scale. The next chapter's optimization is the AdamW (the Adam's decouple the weight decay), the scheduler the warmup the cosine, and the AMP the train the fast the GPU's.