Ai2’s StateBench shows where long-context state tracking breaks down

What is changing?

Ai2 released StateBench on November 3 to measure how well language models track changing entity states across long documents. The benchmark includes four levels, ranging from immutable assignments to complex updates that depend on another variable’s current value. It also presents the same underlying problem in different formats to separate architectural effects from formatting sensitivity.

Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna remained strong on simpler levels, but accuracy fell to 13% or lower on the longest general-finite examples. In contrast, 275-million-parameter Gated DeltaNet and hybrid models trained directly on StateBench generalized almost perfectly across the first three levels for tens of thousands of assignments.

Training sequences contained at most 256 assignments, while tests extended to 64,000 assignments, or 250 times longer. A flexible Gated DeltaNet stayed strong to roughly 10 times the training length on the hardest level, while transformers never learned that level. The work also notes open limitations, including a fixed number of variables and numerical-precision questions.