Quantizing Recurrent Language Models Is Easier Than You Think

Created on October 16, 2026

2026   ·   Deep-Learning   Quantization

The post-training quantization (PTQ) stack has been developed for softmax attention. For linear attention and hybrids, it remains underexplored. This work aims to change that. We differentiate between extra-state weight/activation (W/A) quantization and state quantization during decode in low precision, investigating how both differ from quantization in classical softmax attention. Storing linear attention state during decode, our final scheme LoReS combines value-axis grouping, stochastic rounding, and decay-based mixed precision.

Joint work with Neehal Tumma, Daniela Rus, and T. Konstantin Rusch.

Paper Code
Abstract

Linear recurrences are becoming an increasingly common building block in large language models, replacing the growing KV cache of softmax attention with a fixed-size recurrent state that enables efficient decoding. However, the quantization stack has largely been developed around softmax Transformers, whose numerical properties and inference state differ from those of recurrent models, motivating quantization schemes tailored to recurrences. We study how quantizing linear recurrences differs from quantizing softmax attention along two complementary axes. First, extra-state quantization concerns weights and activations outside the recurrent state, whose distributions are shaped indirectly by the sequence mixer during training. In controlled 1.3B models, we find that decayed recurrences develop much weaker residual outlier channels than softmax attention and are therefore easier to quantize without incoherence processing. Second, intra-state quantization concerns the recurrent state itself during decode, where repeated updates and recurrent decay create a different numerical problem from KV cache quantization. We use these insights to develop LoReS, a pipeline that combines value-axis scale grouping, decay-based mixed-precision allocation and stochastic rounding for low-bit recurrent state quantization. Across Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, LoReS supports robust 8-bit state quantization and lower effective precision on shorter decode horizons. We implement LoReS in vLLM and demonstrate end-to-end decode speedups.

Architectural Differences: Linear vs. Softmax

Softmax attention and linear attention maintain fundamentally different objects for sequence mixing. While softmax attention appends keys and values to a growing KV cache, linear attention updates a fixed-size matrix state in place. This changes the quantization problem.

Language-model layer

Shared language-model layer Input passes through normalization, quantized query, key, and value projections, the sequence mixer, and the quantized output projection. A residual connection bypasses this path. A second residual path contains normalization and a quantized MLP. The mixer reads the stored state and writes it back through a state quantizer. Norm WQ WK WV WO MLP sequence mixer state Norm + + qt kt vt xt

Recurrent state

Recurrent state: overwrite The recurrence takes the query, key, value, and previous state. It overwrites the entire state through a quantizer after each update. The outline marks the whole state as just quantized. recurrence overwrite St−1 → St qt kt vt

KV cache

KV cache: append Softmax attention reads the stored keys and values. A newly quantized key and value are appended at each step. Only this new entry is outlined as just quantized. Earlier entries are reused. softmax append k1v1 k2v2 ⋯ ktvt qt kt vt
Quantized weights Quantized state Sequence mixer State quantizer Just quantized State update
Figure 1. Extra-state weights and activations. Intra-state inference memory.

We differentiate between two quantization objectives. Extra-state quantization covers weights and activations. Its dependence on the sequence mixer is implicit. Intra-state quantization acts on the state stored during decode. Its dependence on the sequence mixer is explicit.

Extra-State Weight and Activation Quantization

We vary the sequence mixer in 1.3B models while keeping the surrounding backbone and training recipe fixed. All experiments in Table 1 use GPTQ with per-output-channel weight scales and dynamic activation scales. We test rotations of the residual stream and the inputs to the MLP down-projection and mixer output projection.

Decayed recurrences are robust

Without incoherence processing, decayed linear recurrences are more robust than softmax attention to W4A4 quantization. Keeping other recurrent degrees of freedom fixed, decay consistently helps, with diagonal decay working best. Applying all three rotations brings perplexity increases to similar levels across models. Robustness differences are thus mainly due to learned distributions, not error propagation under different algebraic properties of the sequence mixer itself.

Mixer-zoo variant W4A4 W4A4 + rotations
Softmax 98.4% 10.6%
Linear attention 111.4% 12.5%
Mamba-2 56.6% 11.9%
GLA 45.0% 9.3%
Gated DeltaNet 55.4% 11.8%
KDA 47.8% 11.2%
Table 1. Relative WikiText-2 perplexity increase over the BF16 weight/activation baseline.

Residual stream outliers

Softmax attention learns an outlier channel in its residual stream whose magnitude is approximately constant across tokens and layers. This channel coarsens the quantization grid, leading to larger quantization error. Decayed linear recurrences have weaker outliers. Across our mixer zoo, stronger outliers accompany larger W4A4 perplexity increases without rotations.

Softmax attention

KDA

Residual magnitudes at layer 12, across 2,048 channels and every second token of a 128-token window. Both plots use the same height and color scales. Softmax attention has a persistent outlier channel, peaking at 18.875. KDA peaks at 4.0625.

Loading activation data…

Figure 2. Residual stream magnitude.

The nearly constant outlier channel acts as a bias on queries, keys and values. In softmax attention, the query/key bias can sharpen the attention distribution. Linear recurrences lack this softmax mechanism and instead use decay to focus on a few tokens. We hypothesize that this explains their weaker residual outliers.

LoReS: State Quantization

We organize our analysis around properties that distinguish recurrent state quantization from KV cache quantization:

  1. Scale grouping. The state has two channel axes, whereas a KV cache has a channel axis and a time axis.
  2. Mixed precision. Recurrent decay suppresses quantization error, inhibiting temporal error propagation.
  3. Rounding. The state is requantized after every decode step, so rounding errors can accumulate.
  4. Rotations. Rotations can reduce one-step error but worsen error propagation across steps.

The resulting quantization scheme, LoReS, keeps updates in full precision and quantizes the state stored between decode steps.

(1/4) Scale grouping

The recurrent state of linear attention has two feature axes. The KV cache only has one. We compare value- and key-axis grouping. Under round-to-nearest (RTN), value-axis (row) grouping is more robust. It is the LoReS default.

Mathematical details

Proposition (per-channel readout error). For a fixed state and unit-norm query, assume independent, uniform rounding errors within half a grid step. With \(\mathbf e=\mathbf E\mathbf q\),

\[\begin{aligned}\mathbb E[(e_i^{\mathrm{val}})^2]&=\frac{M_i^2}{12r^2},\\[0.5em]\mathbb E[(e_i^{\mathrm{key}})^2]&=\frac{\sum_j q_j^2 C_j^2}{12r^2}.\end{aligned}\]

Here \(M_i=\max_j|S_{ij}|\), \(C_j=\max_i|S_{ij}|\), and \(r=2^{b-1}-1\).

To interpret this result, recall that each value row of the state produces one output channel. Under value-axis grouping, the expected squared error in that output channel is proportional to the square of the row’s largest absolute entry. Under key-axis grouping, every channel has the same expected squared error, determined by the column ranges and the query. Large entries in one row can therefore set coarse quantization steps for other rows.

animation details

For a \(4\times4\) state and unit-norm query, we display the readout and predicted RMS rounding error:

\[o_i\pm\sigma_i=(\mathbf S\mathbf q)_i\pm\sqrt{\mathbb E[(\mathbf E\mathbf q)_i^2]}.\]

Here \(\mathbf E\) follows the independent uniform-error model in the proposition above.

(2/4) Mixed precision

Decay attenuates quantization error over time, so we allocate more bits to the slowest-decaying heads or channels.

Mathematical details

Proposition (decaying error). With identical initial states and fixed keys, values, and gates, assume \(\lVert\boldsymbol\alpha_t\rVert_\infty\le\alpha<1\). Delta correction, if used, has unit-norm keys and \(0<\beta_t<2\). Then accumulated state error satisfies

\[\lVert\mathbf E_t\rVert_F\le\sum_{\tau=1}^t\alpha^{t-\tau}\lVert\boldsymbol\varepsilon_\tau\rVert_F.\]

Each fresh quantization error \(\boldsymbol\varepsilon_\tau\) is suppressed exponentially as it propagates through later updates.

animation details

For one injected error, constant scalar decay, and no further rounding, we plot the surviving fraction:

\[\frac{\lVert\mathbf E_n\rVert_F}{\lVert\mathbf E_0\rVert_F}=\alpha^n.\]

(3/4) Rounding

Unlike RTN, stochastic rounding (SR) is conditionally unbiased. Under SR, interactions between errors from different quantization steps vanish in expectation, while RTN can reinforce them. LoReS uses SR with value grouping.

Mathematical details

Proposition (repeated quantization). Assume scalar decay, no delta correction, and keys, values, and gates fixed independently of rounding. Let \(\boldsymbol\varepsilon_\tau\) be fresh quantization error and \(\gamma_{\tau,t}=\prod_{j=\tau+1}^t\alpha_j\). For RTN and conditionally unbiased SR,

\[\begin{aligned} \lVert\mathbf E_t^{\mathrm{RTN}}\rVert_F^2 &=\sum_{\tau=1}^t\gamma_{\tau,t}^2\lVert\boldsymbol\varepsilon_\tau^{\mathrm{RTN}}\rVert_F^2\\ &\quad+2\!\sum_{1\le\tau<s\le t}\!\gamma_{\tau,t}\gamma_{s,t}\bigl\langle\boldsymbol\varepsilon_\tau^{\mathrm{RTN}},\boldsymbol\varepsilon_s^{\mathrm{RTN}}\bigr\rangle_F,\\[0.8em] \mathbb E\!\left[\lVert\mathbf E_t^{\mathrm{SR}}\rVert_F^2\right] &=\sum_{\tau=1}^t\gamma_{\tau,t}^2\mathbb E\!\left[\lVert\boldsymbol\varepsilon_\tau^{\mathrm{SR}}\rVert_F^2\right]. \end{aligned}\]

RTN can reinforce errors across steps. Under SR, these cross terms vanish in expectation.

animation details

After 100 synthetic scalar updates \(S_t=\alpha S_{t-1}+a v_tk_t\) on a fixed grid, we plot the remainder from the proposition:

\[R_t=E_t^2-\sum_{\tau=1}^t\alpha^{2(t-\tau)}\varepsilon_\tau^2.\]

Here \(E_t=\widehat S_t-S_t\). A fresh error \(\varepsilon_\tau\) survives as \(\gamma_{\tau,t}\varepsilon_\tau=\alpha^{t-\tau}\varepsilon_\tau\). Squaring gives the exponent \(2(t-\tau)\). Each trajectory contributes one RTN value and the mean of SR values.

(4/4) Rotations

Random Hadamard rotations reduce quantization reconstruction error. This benefit need not persist under repeated state quantization.

animation details

For a group \(G\) of 64 scalar recurrences sharing a scale, we compare two scale sources over 200 SR updates:

\[\begin{aligned} s_{G,t}^{\mathrm{noisy}}&=\frac{\lVert\mathbf S_t+\alpha\mathbf E_{t-1}\rVert_\infty}{r},\\[0.5em] s_{G,t}^{\mathrm{copy}}&=\frac{\lVert\mathbf S_t\rVert_\infty}{r}. \end{aligned}\]

Here \(\mathbf S_t\) is the unquantized state, \(\mathbf E_{t-1}\) is accumulated error, and \(r=2^{b-1}-1\) for \(b\) bits. Earlier errors affect the noisy scale. Using an unquantized copy removes this dependence, as in the paper’s experiment. We plot \(s_{G,t}\) and \(\lVert\widehat{\mathbf S}_t-\mathbf S_t\rVert_2\), with identical writes and SR draws.

Query alignment

Queries can suppress state quantization error by placing little weight on the affected dimensions. Rotations can weaken this suppression. This need not increase readout error, but we observe such increases in some of our experiments.

animation details

For synthetic \(8\times8\) states after 128 key–value updates, we plot query alignment:

\[a=d_k\,\frac{\mathbb E\lVert\boldsymbol\varepsilon\mathbf q\rVert_2^2}{\mathbb E\lVert\boldsymbol\varepsilon\rVert_F^2}.\]

Here \(d_k=8\), \(\mathbf q\) has unit norm, \(\boldsymbol\varepsilon\) is fresh rounding error, and expectations are over SR. We compare no rotation with a random Hadamard applied to both state and query. The slider shifts query weight between lower- and higher-error channels of the unrotated state. Rotations stay fixed until resampling. The fixed horizontal range covers the central 99% at every query setting. Sparse tails outside it are omitted.

Results

Accuracy

INT8 states preserve most of the accuracy of FP32 states in Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B. Larger losses generally appear with 4-bit weights.

Qwen-3.5 9B

W/A/S GSM8K MMLU-Pro GPQA AIME RULER
16/16/32 95.9 82.6 80.1 82.3 94.6
16/16/8 96.4 82.2 79.6 79.5 94.7
8/16/32 96.5 82.5 80.1 83.1 94.6
8/16/8 96.0 82.3 79.1 80.1 94.5
8/8/32 96.8 82.4 80.3 82.1 94.6
8/8/8 96.5 82.2 79.8 78.6 94.7
4/8/32 97.1 80.7 75.0 73.5 94.9
4/8/8 96.1 80.1 73.6 70.9 94.8
Table 2. Accuracy (%) · W/A/S: weight/activation/state bits · GPQA/AIME: avg@32 · RULER: 32K context

Nemotron-H 8B

W/A/S GSM8K MMLU-Pro GPQA AIME RULER
16/16/32 92.3 70.6 54.1 51.2 88.0
16/16/8 92.0 70.1 53.5 48.4 88.3
8/16/32 92.9 70.4 54.0 51.7 88.4
8/16/8 92.5 70.0 53.1 49.4 88.2
8/8/32 93.3 70.3 54.8 50.9 88.4
8/8/8 92.6 70.1 53.5 49.5 88.5
4/8/32 90.9 67.8 49.0 43.3 85.4
4/8/8 91.8 67.0 49.7 41.0 85.1
Table 3. Accuracy (%) · W/A/S: weight/activation/state bits · GPQA/AIME: avg@32 · RULER: 32K context

Kimi-Linear 48B-A3B

W/A/S GSM8K MMLU-Pro GPQA AIME RULER
16/16/32 89.1 70.3 63.5 63.6 93.7
16/16/8 89.2 70.2 63.8 62.6 93.8
8/16/32 89.8 70.4 63.9 64.3 93.5
8/16/8 89.6 70.2 61.9 62.4 94.0
8/8/32 88.9 70.1 63.3 62.3 93.7
8/8/8 88.4 69.7 62.6 63.6 93.9
4/8/32 86.4 65.9 53.6 56.9 93.4
4/8/8 88.9 65.5 52.6 55.2 93.3
Table 4. Accuracy (%) · W/A/S: weight/activation/state bits · GPQA/AIME: avg@32 · RULER: 32K context
animation details

The tabs select the paper’s measured benchmark results for three hybrid models. Each table compares weight, activation, and state precisions. Highlighted rows use INT8 states.

Lower state precision

Allocating precision by decay lets us use 5 effective state bits for Qwen-3.5 and 5.5 for Nemotron-H on GSM8K with less than a 5% relative accuracy drop from FP32 states.

GSM8K GPQA AIME

Qwen-3.5

First tested precision with less than 5% relative accuracy loss: GSM8K 5 bits, GPQA 6.5 bits, AIME 7.5 bits. Points are measured accuracies relative to FP32 states. Accuracy / FP32 0.0 0.5 1.0 4 5 6 7 8 Effective state bits GSM8K: 5 effective bits is the first tested precision within 5% of FP32 accuracy. GPQA: 6.5 effective bits is the first tested precision within 5% of FP32 accuracy. AIME: 7.5 effective bits is the first tested precision within 5% of FP32 accuracy. GSM8K · 4 effective bits: 8.0% accuracy (0.083× FP32, baseline 95.9%). GSM8K · 4.5 effective bits: 27.8% accuracy (0.290× FP32, baseline 95.9%). GSM8K · 5 effective bits: 93.1% accuracy (0.971× FP32, baseline 95.9%). GSM8K · 5.5 effective bits: 94.2% accuracy (0.982× FP32, baseline 95.9%). GSM8K · 6 effective bits: 95.3% accuracy (0.994× FP32, baseline 95.9%). GSM8K · 6.5 effective bits: 95.8% accuracy (0.999× FP32, baseline 95.9%). GSM8K · 7 effective bits: 96.1% accuracy (1.002× FP32, baseline 95.9%). GSM8K · 7.5 effective bits: 96.1% accuracy (1.002× FP32, baseline 95.9%). GSM8K · 8 effective bits: 96.4% accuracy (1.005× FP32, baseline 95.9%). GPQA · 4 effective bits: 5.1% accuracy (0.063× FP32, baseline 80.8%). GPQA · 4.5 effective bits: 9.6% accuracy (0.119× FP32, baseline 80.8%). GPQA · 5 effective bits: 55.6% accuracy (0.688× FP32, baseline 80.8%). GPQA · 5.5 effective bits: 62.6% accuracy (0.775× FP32, baseline 80.8%). GPQA · 6 effective bits: 68.2% accuracy (0.844× FP32, baseline 80.8%). GPQA · 6.5 effective bits: 77.8% accuracy (0.963× FP32, baseline 80.8%). GPQA · 7 effective bits: 78.3% accuracy (0.969× FP32, baseline 80.8%). GPQA · 7.5 effective bits: 76.8% accuracy (0.950× FP32, baseline 80.8%). GPQA · 8 effective bits: 80.8% accuracy (1.000× FP32, baseline 80.8%). AIME · 4 effective bits: 0.0% accuracy (0.000× FP32, baseline 83.3%). AIME · 4.5 effective bits: 0.0% accuracy (0.000× FP32, baseline 83.3%). AIME · 5 effective bits: 25.0% accuracy (0.300× FP32, baseline 83.3%). AIME · 5.5 effective bits: 43.3% accuracy (0.520× FP32, baseline 83.3%). AIME · 6 effective bits: 50.0% accuracy (0.600× FP32, baseline 83.3%). AIME · 6.5 effective bits: 76.7% accuracy (0.921× FP32, baseline 83.3%). AIME · 7 effective bits: 78.3% accuracy (0.940× FP32, baseline 83.3%). AIME · 7.5 effective bits: 83.3% accuracy (1.000× FP32, baseline 83.3%). AIME · 8 effective bits: 80.0% accuracy (0.960× FP32, baseline 83.3%).

Nemotron-H

First tested precision with less than 5% relative accuracy loss: GSM8K 5.5 bits, GPQA 7 bits, AIME 8 bits. Points are measured accuracies relative to FP32 states. Accuracy / FP32 0.0 0.5 1.0 4 5 6 7 8 Effective state bits GSM8K: 5.5 effective bits is the first tested precision within 5% of FP32 accuracy. GPQA: 7 effective bits is the first tested precision within 5% of FP32 accuracy. AIME: 8 effective bits is the first tested precision within 5% of FP32 accuracy. GSM8K · 4 effective bits: 3.5% accuracy (0.038× FP32, baseline 92.3%). GSM8K · 4.5 effective bits: 6.2% accuracy (0.067× FP32, baseline 92.3%). GSM8K · 5 effective bits: 77.0% accuracy (0.834× FP32, baseline 92.3%). GSM8K · 5.5 effective bits: 91.4% accuracy (0.990× FP32, baseline 92.3%). GSM8K · 6 effective bits: 91.2% accuracy (0.988× FP32, baseline 92.3%). GSM8K · 6.5 effective bits: 91.6% accuracy (0.992× FP32, baseline 92.3%). GSM8K · 7 effective bits: 92.9% accuracy (1.007× FP32, baseline 92.3%). GSM8K · 7.5 effective bits: 92.3% accuracy (1.000× FP32, baseline 92.3%). GSM8K · 8 effective bits: 92.0% accuracy (0.997× FP32, baseline 92.3%). GPQA · 4 effective bits: 7.6% accuracy (0.148× FP32, baseline 51.5%). GPQA · 4.5 effective bits: 16.2% accuracy (0.315× FP32, baseline 51.5%). GPQA · 5 effective bits: 35.9% accuracy (0.697× FP32, baseline 51.5%). GPQA · 5.5 effective bits: 41.4% accuracy (0.804× FP32, baseline 51.5%). GPQA · 6 effective bits: 48.5% accuracy (0.942× FP32, baseline 51.5%). GPQA · 6.5 effective bits: 47.5% accuracy (0.922× FP32, baseline 51.5%). GPQA · 7 effective bits: 51.5% accuracy (1.000× FP32, baseline 51.5%). GPQA · 7.5 effective bits: 51.5% accuracy (1.000× FP32, baseline 51.5%). GPQA · 8 effective bits: 51.0% accuracy (0.990× FP32, baseline 51.5%). AIME · 4 effective bits: 0.0% accuracy (0.000× FP32, baseline 50.0%). AIME · 4.5 effective bits: 0.0% accuracy (0.000× FP32, baseline 50.0%). AIME · 5 effective bits: 6.7% accuracy (0.134× FP32, baseline 50.0%). AIME · 5.5 effective bits: 26.7% accuracy (0.534× FP32, baseline 50.0%). AIME · 6 effective bits: 26.7% accuracy (0.534× FP32, baseline 50.0%). AIME · 6.5 effective bits: 38.3% accuracy (0.766× FP32, baseline 50.0%). AIME · 7 effective bits: 38.3% accuracy (0.766× FP32, baseline 50.0%). AIME · 7.5 effective bits: 46.7% accuracy (0.934× FP32, baseline 50.0%). AIME · 8 effective bits: 51.7% accuracy (1.034× FP32, baseline 50.0%).
Figure 3. Dotted vertical lines mark the first tested precision within 5% of FP32 accuracy. GPQA and AIME use pass@1.

Speedup

State-quantization speedups grow with batch size. Weight-quantization gains are largest at small batches. The two therefore complement each other.

Citation



Enjoy Reading This Article?

Here are some more articles you might like to read next:

  • Depth Memory
  • Dual Complexity Measures for ReLU Networks
  • The Dual Representation of ReLU Networks