{/* Title */}
Pyramid Forcing:
Head-Aware Pyramid KV Cache Policy for
High-Quality Long Video Generation
{/* Authors */}
Jiayu Chen1
{' '}Junbei Tang2,*
{' '}Wenbiao Zhao3,*
{' '}Maoliang Li1
Jiayi Luo4,5
{' '}Zihao Zheng1
{' '}Jiawei Yang1
{' '}Guojie Luo1
{' '}Xiang Chen1,†
1Peking University{' '}
2South China University of Technology{' '}
3Xinjiang University
4Beihang University{' '}
5Zhongguancun Academy
*Work performed during an internship at Peking University.{' '}
†Corresponding author.
{/* Buttons */}
{/* Hero figure — main qualitative comparison (paper Fig. 6) */}
{/* Abstract */}
Abstract
Autoregressive video generation enables streaming and open-ended long video synthesis,
but still suffers from long-term degradation caused by accumulated errors. Existing
KVCache strategies usually apply unified historical-frame retention, implicitly
assuming homogeneous historical dependencies across attention heads. We revisit
historical-frame attention and reveal three distinct head types: Anchor Heads require
broad long-range context, Wave Heads exhibit periodic temporal dependencies, and Veil
Heads focus on initial and adjacent frames. Based on this finding, we propose{' '}
Pyramid Forcing, a head-aware pyramidal KVCache framework that
identifies head types offline, assigns behavior-specific cache policies, and supports
heterogeneous cache lengths via efficient ragged-cache attention. Experiments on Self
Forcing and Causal Forcing show that Pyramid Forcing consistently improves long-horizon
generation quality on VBench-Long, increasing the 60-second Self Forcing score from
77.87 to 81.21 while enhancing motion dynamics, visual fidelity, and semantic
consistency.
{/* Analysis */}
Head-Level Analysis
Figure 2. Head-type attention patterns. Anchor heads spread attention broadly across the rollout, wave heads exhibit periodic structure along the time axis, and veil heads concentrate on the first and most recent frames — the three regimes the pyramid budget is built around.
{/* Method */}
Method Overview
Figure 3. Three-stage pipeline. Offline head profiling labels each head as anchor, wave, or veil; pyramid budget assignment derives a per-head KV size from the role; streaming inference applies the ragged-cache kernel under the bounded budget.
{/* Qualitative grid */}
Qualitative Results
{media.movieGen.map((c, i) => (
))}
{/* Baseline comparison — 30 s rollouts, 2 rows × 4 columns. */}
Comparison with Baselines
30-second video
{media.compare30s.map((row, ri) => {
const isLast = ri === media.compare30s.length - 1;
return (
PROMPT
“{row.prompt}”
{row.cells.map((c, ci) => (
{c.highlight && ★}
{c.label}
))}
);
})}
{/* Baseline comparison — 60 s rollouts, same 2 × 4 layout (no second h3, just a timing label). */}
60-second video
{media.compare60s.map((row, ri) => {
const isLast = ri === media.compare60s.length - 1;
return (
PROMPT
“{row.prompt}”
{row.cells.map((c, ci) => (
{c.highlight && ★}
{c.label}
))}
);
})}
{/* Ablation — same row/column form as the comparison sections above. */}
Ablation
{media.ablation.map((row, ri) => {
const isLast = ri === media.ablation.length - 1;
return (
PROMPT
“{row.prompt}”
{row.cells.map((c, ci) => (
{c.highlight && ★}
{c.label}
))}
);
})}
{/* Footer / cite */}
BibTeX
{`@article{chen2026pyramidforcing,
title={Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation},
author={Chen, Jiayu and Tang, Junbei and Zhao, Wenbiao and Li, Maoliang and Luo, Jiayi and Zheng, Zihao and Yang, Jiawei and Luo, Guojie and Chen, Xiang},
journal={arXiv preprint arXiv:2605.13111},
year={2026}
}`}
Acknowledgements
Page template adapted from the Nerfies project page. Thanks to the open-source
long-video generation community for the models and benchmarks this work builds on.