2026 Frontier LLM Architectures: MLA, iRoPE, Hybrid Sliding Attention, and mHC
Technical comparison of DeepSeek V3.2, Gemma 4 26B-A4B MoE, Llama 4, Gemini 3, and Qwen3: KV-cache bytes-per-token math at 128K context, Per-Layer Embeddings, and Manifold-Constrained Hyper-Connections.
On this page (12 sections)
- The 2026 Architecture Landscape
- Why KV-Cache Geometry Dominates Long-Context Serving
- Four Paths to Long-Context Efficiency
- 1. Multi-head Latent Attention (MLA) + DeepSeek Sparse Attention (DSA)
- 2. Hybrid Sliding-Window + Global Unified-KV Attention & Per-Layer Embeddings (Gemma 4)
- 3. iRoPE: Interleaved Position and NoPE Layers (Llama 4)
- 4. Early Multimodal Fusion
- Expert Routing and Input-Stem Stability
- DeepSeek’s mHC Paper: Stabilizing Multi-Stream Residuals
- Why Unconstrained Hyper-Connections Diverge at Scale
- Projecting Onto the Birkhoff Polytope
- Practical Selection Rules for Engineering Teams
Every flagship language model released over the past eighteen months uses sparse Mixture-of-Experts (MoE) at scale. On that single point, the major research labs agree. Once you open the model configs and inspect how each architecture manages long-context memory, positional generalization, and residual stability, their engineering bets diverge sharply.
DeepSeek projects key-value tensors into a compressed latent subspace and pairs it with a learned sparse token indexer. Meta alternates rotary and position-free layers to stretch context past millions of tokens. Alibaba standardizes on clean Grouped-Query Attention with on-demand reasoning toggles. And in the Gemma 4 family, open-weight design splits memory optimization across two fronts: interleaving 1,024-token sliding windows with unified-KV global layers on the 26B-A4B MoE backbone while introducing Per-Layer Embeddings (PLE) on compact edge checkpoints.
Meanwhile, DeepSeek’s Manifold-Constrained Hyper-Connections (mHC) paper targets an assumption untouched since the original 2017 Transformer: the single-stream x + f(x) residual highway itself.
The 2026 Architecture Landscape
Here is how the leading open-weight and frontier model families compare across total capacity, per-token active compute, and attention topology:
| Model | Total / Active | Active % | Attention Mechanism | Expert Topology |
|---|---|---|---|---|
| DeepSeek V3.2 | 685B / 37B | 5.4% | MLA + DSA (O(Lk)) | 256 routed, 8 + 1 shared |
| Llama 4 Maverick | 400B / 17B | 4.3% | iRoPE (RoPE / NoPE) | 128 routed, 1 + 1 shared |
| Llama 4 Scout | 109B / 17B | 15.6% | iRoPE (10M context) | 16 routed experts |
| Qwen3-235B-A22B | 235B / 22B | 9.4% | GQA (4 KV heads) | 128 routed, 8 active |
| Gemma 4 26B-A4B | 25.2B / 3.8B | 15.1% | 5:1 Sliding + Global K=V | 128 routed, 8 + 1 shared |
| Gemma 4 E4B (Edge) | ~8B / 4B eff. | Edge dense | Sliding + Global + PLE | Dense + Per-Layer Embeds |
| Gemini 3 Pro | ~2-4T* / ~150-200B* | ~5-8%* | Sparse / Multimodal | Native early-fusion MoE |
*Gemini 3 parameter counts are undisclosed; ranges reflect public industry estimates. Anthropic’s Claude Opus 4.5 also remains closed-weight, though its 80.9% SWE-bench Verified performance underscores heavy post-training investment in multi-step tool and code verification.
Resident VRAM vs Active FLOPs
Activation ratios govern decode latency, whereas total parameter counts still govern GPU memory residency. Whether DeepSeek V3.2 activates 5.4% of its weights or Gemma 4 26B-A4B activates 15.1% (3.8B per token), every one of the 128 or 256 routed experts must remain resident in HBM because the router can select a different subset on the very next token.
Why KV-Cache Geometry Dominates Long-Context Serving
During autoregressive decoding at 128K tokens (L = 131,072), model weights are static, while the key-value cache grows linearly with sequence length and batch concurrency. Without architectural intervention, KV state quickly exceeds the memory footprint of the weights themselves.
Comparing analytical KV-cache footprints across the four dominant attention designs at a full 128K-token sequence (131,072 tokens) makes the engineering motivation behind 2026 architectures unmistakable:
| Attention Layout | Reference Config | BF16 Bytes / Tok | 128K Seq (BF16) | 128K Seq (FP8) |
|---|---|---|---|---|
| Standard MHA | 80L, 64 KV heads, d=128 | 2,621,440 B | 320.00 GiB | 160.00 GiB |
| Grouped-Query (GQA) | Llama 3.3 70B (80L, 8 KV, d=128) | 327,680 B | 40.00 GiB | 20.00 GiB |
| Latent (MLA) | DeepSeek V3.2 (61L, 512+64 dim) | 70,272 B | 8.58 GiB | 4.29 GiB |
| 5:1 Hybrid (vLLM Uniform) | Gemma 4 26B-A4B ((8, 256) paged KV) | 42,560 B* | 5.20 GiB | 2.60 GiB |
5:1 Hybrid + Native K=V | Gemma 4 26B-A4B (25 local, 5 K=V) | 11,840 B* | 1.45 GiB | 0.72 GiB |
*Analytical calculations at L = 131,072: Gemma 4 26B-A4B’s 25 local sliding layers cap cached tokens at W = 1,024 (a fixed 200.0 MiB in BF16 or 100.0 MiB in FP8). Under vLLM’s uniform (8, 256) paged KV allocator (modeled in gemma4_serving.sizing and our interactive GPU VRAM, KV-Cache & Serving Cost Explorer), all 30 layers allocate 8 KV heads at d=256 with separate K and V buffers, yielding 5.20 GiB BF16 / 2.60 GiB FP8 at 128K (420.00 MB FP8 at 16K). With native heterogeneous K=V sharing kernels (2 KV heads, d=512, single unified buffer on the 5 global layers), global growth drops 4x further to 1.45 GiB BF16 / 0.72 GiB FP8 at 128K (180.00 MB FP8 at 16K).
Look at the spread across that table: a single 128K conversation under naive 80-layer Multi-Head Attention would consume 320 GiB of HBM, half of an 8xH100 node for one user. GQA cuts that by 8x (40.00 GiB). DeepSeek’s MLA shrinks a 685B frontier model’s 128K cache down to 8.58 GiB in BF16 (4.29 GiB in FP8), and Gemma 4’s hybrid ring-buffer schedule holds a 128K FP8 session to 2.60 GiB under vLLM’s uniform paged allocator or 0.72 GiB with native heterogeneous K=V kernels.
Four Paths to Long-Context Efficiency
1. Multi-head Latent Attention (MLA) + DeepSeek Sparse Attention (DSA)
Introduced in DeepSeek-V2 and refined across DeepSeek R1 and DeepSeek V3.2, Multi-head Latent Attention replaces per-head key and value caches with a shared low-rank joint compression vector.
Instead of caching num_heads * head_dim keys and values in every layer, MLA projects token hidden states down into a 512-dimensional compressed latent vector (d_c = 512). Because Rotary Position Embeddings (RoPE) are position-dependent and cannot be absorbed cleanly into static matrix weights without recomputation, DeepSeek decouples position into a small 64-dimensional shared RoPE key (d_R = 64). Each of V3.2’s 61 layers stores just 512 + 64 = 576 elements per token (1,152 bytes in BF16, 576 bytes in FP8), absorbing the up-projection matrices into query and output heads during inference.
In December 2025, V3.2 layered DeepSeek Sparse Attention (DSA) directly on top of MLA to attack quadratic prefill compute (O(L^2) -> O(Lk)). A lightweight MLP Lightning Indexer, distilled during a 2.1B-token warm-up stage to approximate dense attention scores, ranks historical positions and selects only the top k = 4,096 token keys per head for full attention scoring. At 128K context, scoring 4,096 keys instead of 131,072 cuts attention FLOPs by 32x while preserving gold-medal reasoning performance on IMO and IOI benchmarks.
2. Hybrid Sliding-Window + Global Unified-KV Attention & Per-Layer Embeddings (Gemma 4)
While MLA compresses the width of every layer’s cache, Gemma 4 attacks the depth and length dimensions simultaneously without requiring custom latent-decompression kernels.
In google/gemma-4-26B-A4B-it (25.2B total parameters, 3.8B active across 30 decoder blocks), layers follow a 5:1 local-to-global schedule:
- 25 Local Sliding-Window Layers: Each local block attends strictly within a 1,024-token rolling window (
8KV heads,head_dim = 256). Once prompt length passes 1,024 tokens, local KV buffers stop growing altogether and overwrite evicted slots in a constant-size ring buffer. - 5 Global Full-Attention Layers: Every sixth block attends across the full sequence (up to 256K tokens) using unified key-value sharing (
K = V) across2KV heads (head_dim = 512) and proportional RoPE (p = 0.25). Reusing keys directly as values halves the global projection buffer with negligible perplexity impact.
In our Open-Weight Agent Serving with Gemma 4 & vLLM benchmark (modeled in gemma4_serving.sizing and /lab/), vLLM’s uniform (8, 256) paged KV allocator drops a 16,384-token FP8 session cache from 1,920.00 MB (if all 30 layers cached full context) down to 420.00 MB (a 78.1% reduction, or 2.60 GiB at 128K), allowing two 24 GB NVIDIA L4 GPUs (g2-standard-24) to host both the 26.5 GB FP8 MoE weights and 29 concurrent 16K agent sessions. Dedicated heterogeneous K=V kernels shrink those 5 global layers 4x further, bringing the theoretical FP8 cache down to 180.00 MB at 16K (0.72 GiB at 128K).
For on-device edge tiers (Gemma 4 E2B and E4B, where E denotes effective active parameters), the Gemma 4 technical report pairs 512-token local sliding windows with Per-Layer Embeddings (PLE). Standard transformers map each token ID through a single input embedding matrix and rely on wide FFN layers to refine features across depth. PLE adds a parallel lookup pathway that injects compact, dedicated token-conditioned embedding vectors into every individual decoder layer. Because table lookups require O(1) indexing rather than dense matrix multiplications, PLE lets 2B- and 4B-effective models carry substantially richer lexical and factual conditioning on laptops and mobile NPUs without increasing per-token MAC operations.
3. iRoPE: Interleaved Position and NoPE Layers (Llama 4)
Meta’s Llama 4 herd approaches ultra-long context (up to 10M tokens in Scout) by rethinking where positional decay belongs.
Applying RoPE on every transformer block imposes distance attenuation across all heads, which complicates length extrapolation well beyond pre-training sequence windows. iRoPE alternates standard RoPE blocks with NoPE (“No Position Embedding”) full-attention blocks, paired with inference-time attention temperature scaling:
- RoPE blocks anchor local word order, syntax, and nearby relative offsets.
- NoPE blocks score content similarity purely by key-query inner product regardless of whether matching evidence sits 200 tokens or 2 million tokens back in the stream.
4. Early Multimodal Fusion
Across Gemini 3, Llama 4, and Gemma 4, early multimodal fusion has displaced bolted-on late-stage vision adapters. Rather than freezing a text LLM and training a cross-attention bridge near the output head, modern checkpoints project image patches, audio frames, and text subwords into a unified token stream early in the network. Cross-modal self-attention operates across depth, improving spatial grounding on dense engineering diagrams, UI screenshots, and multi-page PDF tables.
One practical fine-tuning consequence surfaces immediately in code: multimodal backbones often wrap vision linear layers in specialized range-clamping modules (such as Gemma 4’s 163 Gemma4ClippableLinear layers). As we show in Part 3 of our Vertex AI series, LoRA fine-tuning recipes on multimodal MoEs must anchor target_modules regexes strictly to model.language_model.layers so PEFT does not crash on vision towers or destabilize sparse router gates.
Expert Routing and Input-Stem Stability
Within sparse MoE blocks, routing topologies balance capacity against load-balancing fragility:
- Shared + Fine-Grained Routed Experts (
DeepSeek V3.2andGemma 4 26B-A4B): Both pair 1 always-active shared expert with top-8 routing across a fine-grained pool (256experts in DeepSeek V3.2;128in Gemma 4). Dedicating an always-on MLP to common syntactic and formatting operations frees routed experts to specialize without duplicating baseline transformations. - Pure Top-k Routing (
Qwen3-235B-A22B): Activates 8 of 128 experts with no shared expert, simplifying tensor-parallel sharding across standard serving engines. - High-Sparsity Capacity (
Llama 4 Maverick): Activates just 4.3% of total parameters per token (17B of 400B across 128 routed experts plus a shared path), maximizing stored world knowledge per decode FLOP at the cost of tighter router calibration margins.
Across open technical reports, two pre-training stability patterns have also converged into standard practice: running dense transformer blocks at the input stem (DeepSeek V3/V3.2 uses 3 dense layers before MoE routing begins so foundational token representations stabilize prior to discrete top-k selection) and replacing throughput-degrading auxiliary load-balancing losses with auxiliary-loss-free dynamic bias terms on router scores.
DeepSeek’s mHC Paper: Stabilizing Multi-Stream Residuals
On December 31, 2025, DeepSeek published Manifold-Constrained Hyper-Connections (mHC), addressing how deep transformers route signal across layer depth.
Why Unconstrained Hyper-Connections Diverge at Scale
Standard residual connections (x_{l+1} = x_l + f(x_l)) restrict inter-layer communication to a single additive stream. Hyper-Connections (HC), proposed earlier by ByteDance researchers, widen that bottleneck into n parallel residual streams mixed via learned n x n matrices at every block.
At small parameter counts, unconstrained HC looks promising. When DeepSeek scaled unconstrained HC to 27B parameters, repeated multiplication of unconstrained mixing matrices across dozens of blocks caused catastrophic norm amplification: worst-case residual signal gain exceeded 3,000x before training loss diverged.
Projecting Onto the Birkhoff Polytope
mHC constrains each residual stream-mixing matrix onto the Birkhoff polytope, the mathematical manifold of doubly stochastic matrices (non-negative entries whose rows and columns each sum to 1). Because a doubly stochastic operator forms a convex combination of its inputs, multiplying doubly stochastic matrices across arbitrary network depth cannot explode total signal norm.
Using Sinkhorn-Knopp iterative normalization to project learned mixing weights onto the Birkhoff manifold reduced worst-case signal gain across depth from >3,000x down to ~1.6x.
| Controlled 3B Ablation (arXiv:2512.24880) | Standard Residual | Unconstrained HC | mHC (Doubly Stochastic) |
|---|---|---|---|
| BIG-Bench Hard (5-shot) | 43.8% | 48.9% | 51.0% |
| DROP Reading Comprehension (F1) | 47.0 | 51.6 | 53.9 |
| 27B Scaling Run Stability | Stable | Diverged (>3000x gain) | Stable (~1.6x gain, +6.7% compute) |
The headline finding is the 27B survival column: mHC preserves multi-stream representational gains at scale for a 6.7% training compute overhead, providing a mathematically bounded residual substrate for deeper sparse architectures.
Practical Selection Rules for Engineering Teams
Choosing among 2026 architectures comes down to matching HBM math and kernel maturity to your workload:
- High-volume tool dispatch and domain LoRA on commodity GPUs: Reach for
Gemma 4 26B-A4B MoEin FP8. Its 3.8B active footprint and 5:1 sliding/global KV cache run comfortably on 2x L4 GPUs (48 GBtotal VRAM) under standard vLLM with compact 33 MB rank-16 LoRA adapters (runnable tutorial here). - Maximum competition math and cost-efficient API reasoning: Reach for DeepSeek V3.2 via managed endpoints or multi-node H100/H200 clusters with MLA + DSA kernel support (
$0.28/1Minput tokens, tool-use interleaved inside<think>). - Turnkey self-hosting across heterogeneous serving stacks:
Qwen3(Apache 2.0, standard GQA) andLlama 4 Scout(single 80 GB H100 in INT4) avoid specialized latent-attention kernels when portability outweighs minimum KV bytes per token. - Size your own concurrency before provisioning: Plug your prompt lengths, batch concurrency, and daily request volume into the largo.dev GPU VRAM & Serving Economics Lab to see exact weight vs KV-cache splits and managed-API break-even thresholds before spinning up nodes.
Sources & Further Reading:
- DeepSeek V3.2 Architecture & DSA Deep Dive and DeepSeek R1 Architecture & GRPO
- Open-Weight Agent Serving: Gemma 4 26B-A4B MoE on Vertex AI with vLLM
- DeepSeek mHC: Manifold-Constrained Hyper-Connections (arXiv:2512.24880)
- DeepSeek V3.2 Technical Report (arXiv:2512.02556)
- Qwen3 Technical Report (arXiv:2505.09388)
- Meta AI: The Llama 4 Herd
- Sebastian Raschka: The Big LLM Architecture Comparison
Runnable Implementations
Related Tutorials & Guides
DeepSeek V3.2: Frontier Reasoning at 6x Lower Cost
Technical deep dive into DeepSeek V3.2's architecture: DeepSeek Sparse Attention (DSA), integrated reasoning with tool-use, and how it achieves IMO gold-medal performance.
Open-Weight Agent Serving: LoRA-Tuning Gemma 4 26B-A4B MoE on Vertex AI with vLLM
Size Gemma 4 26B-A4B MoE memory and hybrid KV caches, scope LoRA adapters around multimodal vision layers, and connect Google ADK 2.14 directly to vLLM on Vertex AI.
DeepSeek R1 Architecture: How Reinforcement Learning Creates Reasoning
Deep dive into DeepSeek R1's architecture: how pure RL training enables chain-of-thought reasoning, the GRPO algorithm, MoE design, and knowledge distillation.
Discussion
to join the discussion.