位置埋め込みが重要な理由 — 開発者のためのAPE、RPE、RoPEの解説

Self-Attention can compare every token with every other token. But there is a catch. By itself, it does not know the order of tokens. That is a serious problem because “dog bites man” and “man bites dog” use the same words but mean completely different things. Core Idea A Transformer needs two kinds of information: what the token is where the token is Token embeddings provide the “what.” Positional embeddings provide the “where.” This matters because attention without position is order-blind. It can compare tokens, but it does not naturally know which token came first. The Key Structure A simple positional embedding flow looks like this: Token Embedding + Positional Information → Input Representation For Absolute Positional Embedding: E = X + P Where: X = token embedding P = positional embedding E = final input representation More compactly: Transformer input = meaning vector + position signal Different positional methods change how the position signal is injected. Pseudo-code View Basic positional injection: tokens = tokenize(text) x = embedding(tokens) position = positional_embedding(token_positions) input_representation = x + position For attention-based position methods: q = project_query(x) k = project_key(x) q = apply_position(q) k = apply_position(k) attention_scores = q @ k.T APE usually modifies the input embedding. RPE usually modifies the attention score. RoPE usually modifies Query and Key. That difference is the whole story. Concrete Example Compare these two sentences: dog bites man man bites dog The token set is the same: dog, bites, man But the order changes the meaning. Without positional information, Self-Attention sees token relationships but has no built-in sequence order. With positional information, each token representation includes location. So “dog” at position 1 is different from “dog” at position 3. This is why positional encoding is not optional. It is required for language understanding. APE: Absolute Positional Embedding Absolute Positional Embedding assigns a vector to each position index. Position 1 has one vector. Position 2 has another vector. Position 3 has another vector. Then the model adds that position vector to the token embedding. Example: Token embedding: X = [0.2, 0.5] Position embedding: P = [0.1, -0.2] Final representation: E = [0.3, 0.3] APE is easy to understand. It says: this token is at this exact position Why APE Is Useful APE is simple. It is easy to implement. It works well when sequence lengths stay close to what the model saw during training. Implementation-wise, it is just: x = token_embedding + position_embedding That makes it cheap and clean. But the simplicity has a cost. APE treats position as a fixed index. If the model sees much longer inputs than it was trained on, unseen positions can become unreliable. That makes APE weaker for long-context extrapolation. RPE: Relative Positional Embedding Relative Positional Embedding focuses on distance. Instead of asking: What position is this token at? It asks: How far apart are these two tokens? This is often more natural for language. A subject and verb may appear at different absolute positions. But their relative distance and direction still matter. A simplified RPE attention score looks like this: Aᵢⱼ = (QᵢKⱼᵀ + Rᵢ₋ⱼ) / √d Rᵢ₋ⱼ represents the relative position between token i and token j. This means position directly affects attention. Concrete RPE Example Suppose: QᵢKⱼᵀ = 12 Rᵢ₋ⱼ = 4 √d = 4 Then: Aᵢⱼ = (12 + 4) / 4 = 4 Without the relative term: Aᵢⱼ = 12 / 4 = 3 So the distance relationship increased the attention score. That is the intuition. RPE lets the model say: This token is more relevant because of where it is relative to me. RoPE: Rotary Positional Embedding Rotary Positional Embedding takes a different path. It does not add a position vector to the input. It rotates Query and Key vectors based on position. The core idea: position becomes rotation A 2D rotation matrix looks like this: Rθ = [[cosθ, -sinθ], [sinθ, cosθ]] If you rotate [1, 0] by 90 degrees: [1, 0] → [0, 1] RoPE applies this idea across Query and Key dimensions. Different positions get different rotations. Then attention scores naturally include relative position. Why RoPE Works Well RoPE uses absolute position to rotate Q and K. But when Q and K are compared, the score depends on their relative position difference. The key relationship is: (RθⁱQ)ᵀ(RθʲK) = QᵀRθʲ⁻ⁱK This means the attention score contains j - i. That is the relative distance. So RoPE gives you a useful combination: absolute-position injection + relative-position behavior This is why RoPE became popular in modern LLMs. APE vs RPE vs RoPE APE: adds position vectors to token embeddings simple and cheap good for fixed or known sequence lengths weaker for long-context extrapolation RPE: adds relative distance information to attention scores directly models token-to-token distance flexible for variable lengths can complicate attention implementation RoPE: rotates Query and Key vectors by position makes relative distance appear inside attention memory-efficient works well with modern long-context LLMs The key difference: APE = where am I? RPE = how far are we? RoPE = rotate Q/K so distance appears in attention Implementation Perspective If you are reading Transformer code, look at where position enters the model. APE usually appears near the embedding layer: x = token_embedding + position_embedding RPE usually appears inside attention score computation: scores = q @ k.T + relative_position_bias RoPE usually appears after Q and K projection: q = apply_rope(q, positions) k = apply_rope(k, positions) scores = q @ k.T This is the developer shortcut. Find the injection point. Then you know which positional method the model uses. Naive vs Practical View Naive view: Positional embedding just tells the model token order. Practical view: Positional design affects long-context behavior, caching, memory, and attention quality. Naive mindset: add positions run attention Practical mindset: choose how position enters attention consider context length consider extrapolation consider KV Cache compatibility consider implementation complexity This matters because positional encoding is not a small detail. It changes how the model behaves when the context becomes long. Why This Matters Again Short inputs can hide positional weaknesses. Long-context models expose them. If positional information does not extrapolate well, the model may become unstable outside its training length. This is why modern LLMs care so much about RoPE variants and long-context scaling. The position method affects whether a model can reliably handle long prompts, code files, documents, and conversations. Important Conditions and Limits APE is easy but tied to absolute indices. RPE is expressive but can complicate attention computation. RoPE is efficient and practical, but still needs careful scaling for very long contexts. Also: Positional embeddings do not create reasoning by themselves. They only give attention a way to use order. The model still needs training to learn useful patterns. Takeaway Self-Attention needs positional information because it is order-blind by default. APE adds absolute position to embeddings. RPE adds relative distance to attention scores. RoPE rotates Query and Key vectors so relative position appears naturally. The shortest version: Positional Embedding = the order signal that makes attention understand sequence structure If you understand where position enters the model, you understand the difference between APE, RPE, and RoPE. Discussion When learning Transformer internals, which positional method feels most intuitive to you? APE, RPE, or RoPE? Originally published at zeromathai.com. Original article: https://zeromathai.com/en/advanced-positional-embeddings-en/ GitHub Resources AI diagrams, study notes, and visual guides: https://github.com/zeromathai/zeromathai-ai Self-Attention can compare every token with every other token. But there is a catch. By itself, it does not know the order of tokens. That is a serious problem because “dog bites man” and “man bites dog” use the same words but mean completely different things. Core Idea A Transformer needs two kinds of information: what the token is where the token is Token embeddings provide the “what.” Positional embeddings provide the “where.” This matters because attention without position is order-blind. It can compare tokens, but it does not naturally know which token came first. The Key Structure A simple positional embedding flow looks like this: Token Embedding + Positional Information → Input Representation For Absolute Positional Embedding: E = X + P Where: X = token embedding P = positional embedding E = final input representation More compactly: Transformer input = meaning vector + position signal Different positional methods change how the position signal is injected. Pseudo-code View Basic positional injection: tokens = tokenize(text) x = embedding(tokens) position = positional_embedding(token_positions) input_representation = x + position For attention-based position methods: q = project_query(x) k = project_key(x) q = apply_position(q) k = apply_position(k) attention_scores = q @ k.T APE usually modifies the input embedding. RPE usually modifies the attention score. RoPE usually modifies Query and Key. That difference is the whole story. Concrete Example Compare these two sentences: dog bites man man bites dog The token set is the same: dog, bites, man But the order changes the meaning. Without positional information, Self-Attention sees token relationships but has no built-in sequence order. With positional information, each token representation includes location. So “dog” at position 1 is different from “dog” at position 3. This is why positional encoding is not optional. It is required for language understanding. APE: Absolute Positional Embedding Absolute Positional Embedding assigns a vector to each position index. Position