For now, I found what looks like a report of a closely related issue
:
1. A closely related report: LowRankArena
LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression
GitHub repository
LowRankArena compares SVD-based LLM compression methods under standardized evaluation conditions. One relevant finding is that multiple-choice accuracy can conceal substantial perplexity degradation, and method rankings can change across model families and compression ratios.
This is not necessarily the same direction of disagreement as in your examples. However, it provides independent evidence that perplexity and downstream accuracy are not interchangeable measures of compression quality.
It does not establish that attention-sink distortion causes the discrepancy. I would separate the broader evaluation phenomenon from the specific mechanism your sink-preserving intervention investigates.
2. Why perplexity and accuracy can disagree
Two distinctions seem particularly useful.
Average likelihood versus discrete decisions. Perplexity responds continuously to changes in token probabilities. Multiple-choice accuracy depends on which candidate receives the highest score. Candidate scores can improve without changing the answer, while small changes near a decision boundary can flip correctness.
Different evaluation distributions. WikiText-2 perplexity measures average next-token prediction on one text distribution. Zero-shot tasks probe different contexts and candidate distinctions. An improvement in average language-model loss need not produce a corresponding improvement in answer ranking.
The first effect can occur even on the same evaluation examples. The second can amplify it.
So I would retain perplexity as a useful measure of language-model degradation, but not treat it as a substitute for downstream accuracy.
3. A small independent probe
I also tried a limited comparison using Qwen3-0.6B-Base and the standard versus sink-preserving deflation recipes from your implementation.
On an NVIDIA L4 with BF16, I compressed selected K/V projections at matched ranks, then evaluated WikiText-2 perplexity and 64 selected ARC-Easy validation questions using full-answer conditional log-likelihood.
| Method |
WikiText-2 PPL ↓ |
ARC correct / 64 |
Length-normalized correct / 64 |
| Standard whitened SVD |
94.83 |
24 |
23 |
| Sink-preserving deflation |
88.75 |
24 |
23 |
Perplexity improved, but aggregate accuracy did not.
The question-level outcomes were more informative:
- Raw answer-text scoring: 3 incorrect-to-correct and 3 correct-to-incorrect transitions.
- Character-length-normalized scoring: 4 transitions in each direction.
Thus, unchanged aggregate accuracy concealed changes in which questions the model answered correctly.
This was a deliberately aggressive, small-scale experiment: 75% parameter removal within selected K/V matrices in six layers, rather than your 20%/40% model-wide experimental setting. The 64-question evaluation was custom, not an official full ARC-Easy benchmark.
I would treat it as a related observation, not a replication or proof that sink preservation caused the individual decision changes.
4. A low-cost diagnostic path
If per-question candidate scores are already available, I think the most informative next comparison would be paired correctness changes and answer margins, without necessarily running another large benchmark.
For each question:
margin = score(correct answer)
- max(score(incorrect answers))
Compare this margin between standard truncation and sink-preserving compression.
The interpretation can branch naturally:
- Margins improve but accuracy stays unchanged: candidate separation improved without crossing many decision boundaries.
- Correctness switches in both directions: aggregate accuracy may conceal offsetting improvements and regressions.
- Perplexity improves but task margins barely move: the likelihood improvement may occur mainly outside the distinctions tested by that task.
- Effects differ across tasks: task distribution, scoring conventions, or representation sensitivity may matter.
If only task-level aggregates are available, they are still useful. There is no need to reconstruct item-level information from aggregates or repeat expensive experiments solely for this analysis.
My suggested default would be to keep perplexity and downstream accuracy side by side, and inspect paired candidate margins when they disagree.
Your paired-bootstrap analysis already addresses part of the statistical-uncertainty question. The margin analysis would serve a different purpose: identifying where and how the decisions change.
The expandable sections below contain the evaluation details, local probe results, possible mechanisms, and references.
Evaluation metrics, correctness transitions, and a minimal decision tree
Why the metrics are not monotonically related
Perplexity is the exponential of average next-token negative log-likelihood:
L=-\frac{1}{N}\sum_{i=1}^{N}\log p(x_i\mid x_{<i})
\mathrm{PPL}=\exp(L)
This is a continuous measure.
Multiple-choice accuracy instead usually selects the candidate with the highest score and checks whether that candidate is correct.
Consequently, changes in likelihood and changes in accuracy need not move together.
For example, suppose two binary-choice questions have the following probabilities assigned to their correct answers:
| Question |
Before |
After |
| 1 |
0.51 |
0.95 |
| 2 |
0.51 |
0.49 |
Initially, both answers are correct.
Afterward, accuracy decreases from 100% to 50%.
Yet the average negative log-likelihood of the correct choices improves from approximately 0.673 to 0.382.
This is a mathematical illustration, not an observation about your models.
It shows that disagreement does not require different evaluation datasets. It can arise even with matched questions.
Signed answer margins
For each multiple-choice question, define:
gold_score = score(correct candidate)
wrong_score = highest score among incorrect candidates
margin = gold_score - wrong_score
Under an ordinary highest-score-wins rule:
- Positive margin: the correct candidate wins.
- Negative margin: an incorrect candidate wins.
- Zero margin: an exact tie requiring an explicit convention.
Then compare:
delta_margin =
margin(deflation) - margin(standard)
This distinguishes score improvement from correctness improvement.
For example:
Standard margin: -0.80
Deflation margin: -0.20
The margin improved,
but the answer remains incorrect.
Conversely:
Standard margin: +0.01
Deflation margin: -0.01
The score changed only slightly,
but correctness flipped.
Both situations can influence how perplexity and downstream accuracy relate.
Paired correctness transitions
The simplest paired analysis separates four outcomes:
| Standard SVD |
Deflation |
Interpretation |
| Correct |
Correct |
Success preserved |
| Incorrect |
Correct |
Improvement |
| Correct |
Incorrect |
Regression |
| Incorrect |
Incorrect |
Still incorrect |
The final category may also conceal changes between different incorrect predictions.
If candidate scores are available, useful measurements include:
- Correctness transitions in both directions.
- Changes in signed answer margins.
- Selected-answer changes even when both predictions are incorrect.
- Near-zero margins and exact ties.
- Effects grouped by task, model, and compression budget.
- Differences between raw and normalized scoring.
This connects to the correctness agreement perspective in The Illusion of Equivalency.
That study concerns quantization, not SVD compression. The useful connection is methodological: similar aggregate accuracy can conceal different question-level behavior.
Raw and normalized multiple-choice scoring
The ARC-Easy configuration in lm-evaluation-harness uses a context resembling:
Question: <question>
Answer:
The candidate completions are the full answer texts, rather than merely the letters A, B, C, or D.
Raw conditional log-likelihood sums the token log-probabilities assigned to each candidate completion.
The checked multiple-choice scoring implementation also supports acc_norm, which divides candidate log-likelihood by the answer string’s character length in that code path.
Therefore:
acc uses raw candidate log-likelihood.acc_norm uses character-length-normalized scores.- Token-count normalization, when used separately, is another convention.
- Answer-letter scoring differs from answer-text scoring.
These can select different answers for the same question.
For paired comparisons, the scorer, prompt, candidate ordering, tokenizer behavior, and tie-breaking convention should remain fixed.
This is not a claim that your evaluator has a problem. It is a reproducibility consideration when comparing relatively small accuracy differences.
Minimal decision tree
Perplexity and accuracy disagree
|
+-- Are per-question candidate scores available?
| |
| +-- Yes
| | +-- Compare paired correctness transitions
| | +-- Compare signed answer margins
| | +-- Check near-ties and normalization
| | +-- Group changes by task/model/budget
| |
| +-- No
| |
| +-- Per-task aggregates available?
| |
| +-- Yes
| | +-- Compare task-level effects
| | +-- Locate concentrated gains/losses
| |
| +-- No
| +-- Retain aggregate observations
| +-- Avoid item-level causal claims
|
+-- Is attention-sink causality the question?
|
+-- No
| +-- Metric-level analysis may be enough
|
+-- Yes
+-- Consider K/V, activation geometry,
and compression-error controls
The point is not to replace perplexity with another universal metric.
It is to distinguish preservation of average likelihood from preservation of downstream decisions.
Independent Qwen3 probe: full results and limitations
Experimental conditions
The small probe used:
| Component |
Setting |
| Model |
Qwen3-0.6B-Base |
| GPU |
NVIDIA L4 |
| Model arithmetic |
BF16 |
| Factorization |
Higher-precision calculations |
| Calibration |
20 WikiText-2 windows |
| Calibration length |
128 tokens |
| PPL evaluation |
16 separate 128-token WikiText-2 test windows |
| ARC evaluation |
64 selected validation questions |
| Compressed layers |
0, 5, 10, 16, 22, 27 |
| Compression budget |
75% removal within selected K/V matrices |
| Other weights |
Left dense |
| Comparison |
Standard versus deflation at matched ranks |
The purpose was to inspect whether the metric discrepancy could appear in a small publicly available model, not to reproduce the full model-and-budget grid from your paper.
Full comparison
| Variant |
WikiText-2 PPL ↓ |
Raw ARC / 64 |
Normalized ARC / 64 |
| Dense model |
29.19 |
47 |
39 |
| Standard K+V |
94.83 |
24 |
23 |
| Deflation K+V |
88.75 |
24 |
23 |
| Standard K only |
31.79 |
44 |
40 |
| Deflation K only |
31.54 |
45 |
40 |
| Standard V only |
90.93 |
25 |
20 |
| Deflation V only |
85.22 |
26 |
22 |
The results suggest three things within this limited setup.
First, deflation improved measured WikiText-2 perplexity in each matched comparison.
Second, downstream accuracy did not respond consistently. K+V aggregate accuracy was unchanged, while K-only and V-only comparisons showed small differences.
Third, the selected V-only compression caused substantially more degradation than K-only compression.
The third observation should not be generalized into a universal claim that V projections are more sensitive. It depends on the selected model, layers, ranks, calibration sample, and evaluation protocol.
Question-level changes
For K+V compression:
Raw answer-text scoring:
Standard incorrect -> Deflation correct: 3
Standard correct -> Deflation incorrect: 3
Standard: 24/64
Deflation: 24/64
For character-normalized scoring:
Standard incorrect -> Deflation correct: 4
Standard correct -> Deflation incorrect: 4
Standard: 23/64
Deflation: 23/64
The aggregate numbers matched, but the correct-answer subsets differed.
The recorded correctness switches were not all numerical ties. None of the switching items in the full-answer evaluation had a top-candidate score gap at or below 1e-4 in either compared variant.
Thus, the changes cannot all be explained by tie-breaking.
However, they do not establish which internal representation changes caused the decisions to move.
Evaluation-protocol sensitivity
An earlier version of the probe scored answer letters rather than answer texts.
For the same 28-question anchor on the L4:
| Scoring protocol |
Standard K+V |
Deflation K+V |
| Answer-letter scoring |
7/28 |
6/28 |
| Full-answer-text scoring |
9/28 |
10/28 |
| Full-answer-text, normalized |
8/28 |
10/28 |
The apparent direction of the accuracy difference changed with the evaluation protocol.
These protocols differ in their prompts and candidate completions, not merely in a final normalization step. This comparison therefore does not isolate the effect of one scoring operation.
It does show why evaluation details matter when interpreting small accuracy changes.
Limitations
Several boundaries are important:
Compression strength. The test used aggressive compression of selected projections rather than your full-model 20%/40% budgets.
Model size. Qwen3-0.6B may behave differently from larger Qwen3, Llama, or Gemma models.
Evaluation sample. Sixty-four questions are useful for inspecting paired outcomes, but insufficient for strong general benchmark conclusions.
Custom scoring. The full-answer evaluator approximated ARC-Easy’s multiple-choice formulation. It was not an official full lm-evaluation-harness run.
No broad uncertainty estimate. Calibration samples, question subsets, and numerical conditions were not systematically varied.
No causal attribution. An answer-ranking change after deflation does not prove that a particular attention-sink signal caused it.
The narrow conclusion is:
In this configuration, deflation improved measured perplexity without changing aggregate ARC accuracy, while altering individual question outcomes.
That is consistent with the broader discrepancy you described, without establishing equivalence to your experimental conditions.
Possible mechanisms and conditional controls
The metric-level explanation establishes why discrepancies are possible.
The deeper question is why a particular sink-preserving intervention changes some downstream decisions.
Several mechanisms seem worth keeping separate.
1. Exactness on a direction versus full-model behavior
Your published factorization code extracts a dominant direction from the first-position activation Gram matrix.
Deflation then constructs a rank-constrained approximation that reproduces the original weight matrix’s action on that direction.
This provides a specific fidelity guarantee without increasing the factorization rank.
For interpretation, three properties should be distinguished:
- Exact reconstruction on the protected direction.
- Reconstruction of all observed first-position activations.
- Preservation of the complete compressed model’s downstream behavior.
The construction directly targets the first property, subject to numerical precision.
The second depends on how well the protected direction represents the activation distribution.
The third additionally depends on interactions with other layers and matrices.
This distinction is useful because a local exactness guarantee can be valid even when downstream accuracy changes.
2. Fixed BOS versus variable first-position tokens
The beginning of each calibration window can affect the geometry of first-position activations.
In a deterministic, strictly causal transformer, the first position cannot attend to later tokens.
If every window begins with the same BOS token under the same position and execution conventions, first-position activations can be identical across otherwise different windows.
The activation distribution may then be effectively one-dimensional.
If the first token varies between windows, first-position activations may occupy multiple directions.
Protecting one dominant direction can still help, but may not cover the full distribution.
Your data preparation implementation already distinguishes ordinary windows from windows with an inserted BOS token.
A conditional diagnostic could inspect:
- The leading eigenvalue’s share of first-position activation energy.
- Reconstruction error on actual first-position vectors.
- Variation across layers and model families.
- Whether protecting additional directions improves coverage.
If one direction already captures almost all first-position energy, protecting more directions may offer little benefit.
If energy is distributed across several directions, multiple-direction protection becomes a more meaningful comparison.
The total rank should remain fixed when comparing these alternatives.
3. Attention mass, gradient energy, and task relevance
High attention mass, high output-gradient energy, and downstream answer relevance are related but distinct properties.
High attention mass indicates that many queries attend strongly to a position.
Gradient energy describes sensitivity of an objective to changes in particular representations.
Downstream relevance concerns whether those representations help distinguish task-specific candidate answers.
The paper When Attention Sink Emerges in Language Models: An Empirical View reports evidence that attention sinks can behave partly like key biases rather than ordinary semantic information carried through values.
This provides a useful background hypothesis for K/V asymmetry.
It does not contradict your gradient-energy observations or prove the mechanism operating in your models.
If the question becomes which projection contributes to downstream changes, a matched comparison could be:
| Condition |
K projection |
V projection |
| Standard baseline |
Standard |
Standard |
| K-only intervention |
Deflation |
Standard |
| V-only intervention |
Standard |
Deflation |
| Joint intervention |
Deflation |
Deflation |
Keeping ranks and calibration inputs matched would help distinguish the roles of the interventions.
The local reconstruction results and downstream candidate margins could then be evaluated separately.
4. Dense calibration versus compressed-model activations
The published statistics collector collects layer-input statistics from calibration forwards.
After earlier matrices are compressed, later layers may receive slightly different inputs.
Therefore, reproducing an original protected direction exactly at a particular matrix does not guarantee that every new input reaching that matrix lies on the same direction.
This connects to the broader error-propagation problem discussed in:
These are related mechanisms, not explanations already established for your results.
A compact control would be:
1. Collect dense-model first-position statistics.
2. Compress the selected matrices.
3. Run the compressed model on the same inputs.
4. Collect compressed-model first-position activations.
5. Compare:
- Protected-direction reconstruction error
- Actual first-position reconstruction error
- Activation drift by layer
- Downstream candidate-margin changes
This separates local factorization fidelity from the behavior of the complete compressed network.
5. Choosing experiments by the question
The useful controls depend on what remains unexplained.
| Question |
Possible comparison |
| Is the sink position specifically important? |
Sink-direction versus matched non-sink-direction protection |
| Is one protected direction sufficient? |
One versus multiple directions at fixed total rank |
| Does K or V drive the effect? |
K-only, V-only, and joint deflation |
| Does compression change the relevant inputs? |
Dense versus compressed activation statistics |
| Does window construction matter? |
Fixed-BOS versus variable-first-token conditions |
| Does the effect survive milder compression? |
Same evaluation protocol across several budgets |
| Why does accuracy not track PPL? |
Paired correctness transitions and answer margins |
These are alternatives, not a checklist of additional requirements.
For the immediate perplexity/accuracy question, the final row offers the most direct and usually cheapest information.
For causal analysis of sink preservation, the other comparisons become more useful.
6. Nominal cost versus measured deployment cost
One additional distinction may matter to deployment-oriented readers.
A same-rank deflation does not add another low-rank component relative to a rank-matched standard factorization.
This supports the same nominal parameter budget and operation count.
However, theoretical storage and measured inference performance are different questions.
The published compression implementation supports both dense materialization and factorized LowRankLinear modules.
Dense materialization is useful for examining approximation quality, while factorized modules realize the corresponding parameter-storage reduction.
Actual speed and memory savings depend on matrix dimensions, kernels, batch size, and deployment backend.
This does not affect the central metric-discrepancy observation, but keeps rank-matched theoretical cost separate from measured serving efficiency.
Related research and reproducibility resources
These references address different parts of the problem. Similar observations should not automatically be attributed to identical mechanisms.
LowRankArena: standardized SVD evaluation
LowRankArena paper
GitHub implementation
The closest independent reference for the explicit metric-discrepancy question.
It shows that SVD compression comparisons depend on architecture, retention ratio, evaluation protocol, and measurement conditions.
Its results support evaluating perplexity and multiple-choice accuracy separately.
They do not establish attention-sink-specific causality.
LEMS and KFAC-SVD: allocation versus decomposition
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Project and code
This work separates two optimization decisions:
- Constructing accurate low-rank approximations.
- Distributing a global parameter budget across layers and matrices.
Your combination with LEMS is relevant because an improvement in the factorization statistic may interact with the rank-allocation policy.
That interaction is distinct from the metric-level reason why perplexity and accuracy can disagree.
Per-matrix fidelity versus model-level behavior
Per-Matrix Optimality Is Not Enough
This examines how independently optimized matrix approximations interact through the nonlinear transformer forward pass.
Its broader message is that improving local reconstruction error does not automatically restore every aspect of model-level performance.
The connection is complementary to your selected-direction fidelity guarantee.
Protecting important initial tokens
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
GitHub implementation
IntactKV preserves important initial-token KV-cache representations during quantization.
The conceptual connection is protecting disproportionately influential positions.
However, it preserves cached representations rather than constraining the low-rank approximation of projection weights.
It is neighboring work, not an equivalent intervention.
Attention-sink mechanisms
When Attention Sink Emerges in Language Models
This investigates the emergence and possible roles of attention sinks, including their relationship to key-side behavior.
It provides useful context for interpreting K/V asymmetry, without establishing that all sinks have identical functions.
Behavioral agreement under quantization
The Illusion of Equivalency
This study reports that similar aggregate accuracy and perplexity can coexist with different question-level behavior in quantized models.
The connection is methodological: comparing which questions are answered correctly, rather than only how many.
The compression mechanism is different from activation-whitened SVD.
Practical evaluation references
The following lm-evaluation-harness resources are useful for reproducing established multiple-choice evaluation conventions:
For a reproducible comparison, it is helpful to retain:
Model:
Checkpoint and revision
Tokenizer and special-token behavior
Dtype and attention backend
Compression:
Code revision
Affected matrices and layers
Rank-allocation rule
Parameter-removal definition
Calibration data and window construction
Covariance damping
Protected directions
Factor precision and materialization
Evaluation:
Dataset, split, and item IDs
Task and evaluator revisions
Prompt and candidate texts
Raw/normalized scoring definitions
Tie-breaking rule
Per-item candidate scores, if available
Results:
Dense baseline
Standard compression
Sink-preserving compression
Perplexity and downstream accuracy
Paired correctness transitions
Candidate-margin changes
Not every forum post needs this complete record.
However, retaining per-question candidate scores makes later analysis much easier, especially when aggregate accuracy stays unchanged or the direction of a small difference changes with the scoring protocol.
It can also avoid repeating expensive compression experiments merely to investigate evaluation-level discrepancies.
Overall takeaway
The evidence seems to support two separate conclusions.
The weak relationship between perplexity and downstream accuracy has close independent precedents in SVD-compressed LLMs. LowRankArena is a useful starting reference, and the small Qwen3 probe provides another limited example.
The specific contribution of attention-sink preservation to those differences remains a separate mechanism question. Your method provides a concrete intervention on a particular activation direction, making that question more experimentally accessible.
For the immediate discrepancy, I would start with paired correctness transitions and candidate margins, reusing existing evaluation outputs wherever possible.
For the deeper mechanism, the next controls depend on whether the variation follows sink-direction geometry, K/V asymmetry, compression-induced activation drift, or model-specific calibration conditions.