# \[Research\] From Functional Geometry to Dynamic Grammar: New LIMEN Audits (V23–V24) Across 7 Architectures

**URL:** <https://discuss.huggingface.co/t/research-from-functional-geometry-to-dynamic-grammar-new-limen-audits-v23-v24-across-7-architectures/177260>\
**Category:** 🤗Transformers\
**Created:** [June 30, 2026, 12:26am UTC](https://discuss.huggingface.co/t/research-from-functional-geometry-to-dynamic-grammar-new-limen-audits-v23-v24-across-7-architectures/177260 "2026-06-30T00:26:32Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 2, 2026, 1:16pm UTC](https://discuss.huggingface.co/t/research-from-functional-geometry-to-dynamic-grammar-new-limen-audits-v23-v24-across-7-architectures/177260/2 "2026-07-02T13:16:03Z")

</div>

I broadened the search terms and found a few leads that may be useful:

* * *

I would search less for **“symbolic dynamics of hidden states”** alone and more for terms like **activation-derived state abstraction** , **coarse state-transition structure** , **Markov/HMM-style hidden-state dynamics** , **internal state-transition dynamics during generation** , **cross-layer hidden-state dynamics** , and **belief-state geometry**.

My impression is that your V24 framing is adjacent to **state-transition abstractions over activation trajectories** , rather than ordinary probing alone. I would not treat the papers below as identical to LIMEN, but they seem like useful comparison points and search anchors.

## 1. Nearby literature / comparison points

| Area | Why it seems relevant |
| --- | --- |
| **Markov/HMM-style activation-state abstraction** | Closest to the idea of recovering coarse internal states and transition motifs from activations. |
| **Hallucination detection via state transitions** | Relevant to the B→D / grammar-violation idea. |
| **Cross-layer hidden-state dynamics** | Relevant because the signal is not just a static hidden vector, but how representations evolve across layers. |
| **Belief-state geometry** | A useful theoretical bridge from continuous representation geometry to coarse state dynamics. |
| **Latent prediction trajectories** | Useful for checking whether a “D” state is actually decision-like / commitment-like. |

### Closest matches I found

1. **[Markovian Circuit Tracing for Transformer State Dynamic](https://arxiv.org/abs/2605.20824)**  
This seems like the closest comparison point I found. It frames transformer computation as movement through internal states and tests whether activations contain **coarse state-transition structure**. The setup is synthetic HMM tasks where latent states, transition matrices, Bayesian belief vectors, Bayes-optimal predictions, and counterfactual targets are known.

2. **[PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics](https://aclanthology.org/2024.findings-naacl.294/)**  
This seems especially relevant to your question about whether a B→D-like jump, or a violation of a B→A→D grammar, could be related to hallucination or reasoning failure. PoLLMgraph treats hallucination detection/forecasting as a problem of **internal state-transition dynamics during generation**.

3. **[ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs](https://aclanthology.org/2025.acl-long.880/)** and its [code repository](https://github.com/XavierZhang2002/ICR_Probe)  
This is not a symbolic-transition-grammar paper, but it is relevant because it shifts from static hidden-state probes to the **cross-layer evolution of hidden states**. That seems adjacent to your layer-wise dynamic-regime framing.

4. **[Transformers Represent Belief State Geometry in Their Residual Stream](https://arxiv.org/abs/2405.15943)**  
This seems like a useful theoretical bridge. It argues that, in HMM-like generative settings, transformer residual streams can linearly represent belief-state geometry. That does not directly prove anything about natural-language hallucination or B/A/D regimes, but it gives a vocabulary for connecting **functional geometry** to **coarse state dynamics**.

5. **[Eliciting Latent Predictions from Transformers with the Tuned Lens](https://arxiv.org/abs/2303.08112)**  
This is not a state-transition grammar method, but it is useful for checking whether a state called “D” is actually decision-like. Tuned Lens decodes hidden states into vocabulary distributions layer by layer, so it can help compare state occupancy with prediction commitment.

6. **[LogitLens4LLMs](https://arxiv.org/abs/2503.11667)** / [repo](https://github.com/zhenyu-02/LogitLens4LLMs)  
Potentially useful if you want a more implementation-oriented logit-lens style tool for modern HF models, including Qwen-family models.

> **More detailed relationship to your B/A/D framing**
>
> The way I would map these to your terminology is roughly:
> 
> | Your framing | Nearby existing vocabulary |
> | --- | --- |
> | Functional geometry | residual-stream geometry, belief-state geometry, representation geometry |
> | Dynamic grammar | coarse state-transition model, activation-derived transition structure |
> | B/A/D states | abstract activation states, latent regimes, clustered/probed state labels |
> | B→A→D motif | transition motif, state-transition path, Markov/HMM-style transition pattern |
> | B→D jump | transition violation, skipped intermediate regime, possible error/hallucination signal |
> | D state | possible commitment/decision-like state, to be checked against entropy/margin/logit-lens trajectories |
> 
> The strongest direct analogy is probably not “probing” in the usual static sense. It is closer to:
> 
> > continuous activation trajectory → coarse state abstraction → transition matrix / transition motif → optional predictive or causal validation
> 
> That is why MCT and PoLLMgraph seem like better anchors than generic probing papers.

## 2. Search terms that may help future readers

A useful phrase might be:

> **activation-derived coarse-state transition model**

That wording avoids over-committing to “symbolic grammar,” while still capturing the useful part of the idea: you are turning continuous activation trajectories into discrete-ish regimes and then studying transitions among those regimes.

Other search terms I would try:

- activation-derived state abstraction
- coarse state-transition structure
- Markovian hidden-state dynamics
- HMM-style activation dynamics
- internal state-transition dynamics during generation
- cross-layer hidden-state dynamics
- belief-state geometry in residual streams
- latent prediction trajectories
- layer-wise prediction refinement
- hidden-state hallucination detection
- activation trajectory clustering
- state-transition interpretability
- fixed-layer time-transition analysis
- hidden-state dynamics hallucination detection

## 3. One thing I would separate: clustering vs naming

If B/A/D are meant as functional names rather than just cluster labels, I would separate two steps:

1. **State construction**  
How are the states obtained?  
Clustering? Thresholding? A linear probe? A hand-built rule? A model-specific classifier?

2. **State naming**  
Why is a state called exploration, stabilization, decision, etc.?  
What observable behavior supports that name?

For example, if **D = decision-like** , I would expect some relation to prediction commitment, such as:

- lower next-token entropy,
- larger top-1 / top-2 margin,
- higher agreement with final-layer predictions,
- earlier/later commitment timing,
- possibly a Tuned Lens / Logit Lens trajectory becoming more stable.

But if D can be predicted almost entirely from entropy or top-token margin, then “D” may still be useful, but the interpretation shifts toward **confidence proxy** rather than an independent functional state.

> **Possible naming checks for B/A/D**
>
> For a D-like state:
> 
> | Check | Why it helps |
> | --- | --- |
> | next-token entropy | Tests whether D corresponds to low uncertainty. |
> | top-1 / top-2 margin | Tests whether D corresponds to strong token preference. |
> | logit-lens or tuned-lens agreement with final prediction | Tests whether D corresponds to prediction commitment across layers. |
> | D occupancy timing | Tests whether newer/larger models delay or accelerate commitment. |
> | entropy-only / margin-only baseline | Tests whether D is just a confidence proxy. |
> 
> For B/A-like states:
> 
> | Check | Why it helps |
> | --- | --- |
> | layer-depth baseline | Tests whether B/A/D are just shallow/mid/deep layer bands. |
> | prompt-family baseline | Tests whether states track prompt category or template. |
> | token-position / step baseline | Tests whether states track position in the generation. |
> | shuffled-layer control | Tests whether the transition motif depends on actual layer order. |
> | random Gaussian control | Tests whether clustering always creates a plausible-looking transition matrix. |
> 
> I would treat the names as provisional until these are reported.

## 4. A small routing diagnostic I tried

I also tried a small Colab-T4-style routing check. I would not treat this as validation or refutation of the dynamic-grammar idea. The point was only to see what simple state abstractions tend to pick up.

The setup was intentionally lightweight:

- small public models,
- manual greedy decoding,
- hidden states collected with `output_hidden_states=True`,
- raw vs layer-centered vs layer-z-scored hidden states,
- PCA/KMeans-style state abstraction,
- checks against depth-only, prompt-family, entropy/margin, generation-step, shuffled-layer, and fixed-layer time-transition baselines.

The most useful observation was:

> Raw layer-wise transition motifs can look clean while being strongly aligned with layer depth.

Layer centering or z-scoring can reduce that signal in some models, but it can also expose other proxy alignments such as prompt family, entropy/margin bins, or generation step. So I would frame these checks as **routing diagnostics** , not as an objection to the dynamic-grammar idea.

> **What the routing diagnostic suggested**
>
> The main lesson was not “the motif is false.” It was more practical:
> 
> > before assigning functional names to B/A/D-like states, it helps to show what simple proxy each state abstraction is most aligned with.
> 
> A useful reporting table might include:
> 
> | Variant | Question it answers |
> | --- | --- |
> | raw hidden states | What motif appears in the original geometry? |
> | layer-centered states | Does anything remain after subtracting layer-specific means? |
> | layer-z-scored states | Does anything remain after reducing layer-specific scale/mean effects? |
> | fixed-layer time transitions | What happens if layer depth is removed from the transition axis? |
> | shuffled-layer transitions | Does the motif require true layer order? |
> | random Gaussian baseline | Does the same pipeline invent motifs from noise? |
> | depth-only baseline | Are states basically shallow/mid/deep layer labels? |
> | entropy/margin-only baseline | Are states basically confidence labels? |
> | prompt-family baseline | Are states basically prompt category or template labels? |
> | generation-step baseline | Are states basically time-position labels? |
> 
> This was useful because it turned the question from:
> 
> > “Is the dynamic grammar real or not?”
> 
> into the more testable question:
> 
> > “Which part of the observed motif survives depth, confidence, prompt-family, and step controls?”
> 
> That seems like a more productive framing.

## 5. Suggested reporting table

If you continue this line of work, I think a compact table like this would make the claim easier to compare across models and future replications.

| View | What it tests | Useful controls |
| --- | --- | --- |
| raw layer-wise states | Original observed motif | shuffled-layer, random Gaussian |
| layer-centered states | Non-depth residual structure | depth-only, prompt-family |
| layer-z-scored states | Mean/scale controlled structure | entropy/margin, step |
| fixed-layer time transitions | Generation-time dynamics without layer-depth transitions | shuffled-time, prompt-family |
| D-state checks | Whether “D” is decision-like | entropy, margin, tuned/logit lens |
| B→D violation checks | Whether transition skips predict failure | hallucination / reasoning-error cases |
| held-out prompt families | Generality beyond prompt template | prompt-family split |
| held-out model families | Generality beyond one architecture | model-family split |

A minimal version could be:

```plaintext
raw / layer-centered / layer-z-scored
×
depth / prompt-family / entropy / margin / step / shuffled-order / fixed-layer time-transition

```

That would make the B/A/D interpretation much easier to evaluate without requiring a huge model run.

## 6. Reproducibility note: hidden-state extraction contract

Not as an objection to your result, but for this kind of geometry/transition analysis, I think the exact hidden-state extraction contract matters a lot.

The HF docs note that `outputs.hidden_states[-1]` may not always match `outputs.last_hidden_state`, because some models apply normalization or other processing to the returned last hidden state: [Transformers model output docs](https://huggingface.co/docs/transformers/en/main_classes/output).

Generation-time hidden states also need care. There is an HF Forum thread on retrieving hidden states from `model.generate()` where the answer separates generated-token states from input-sequence states and mentions `return_dict_in_generate=True` / `output_hidden_states=True`: [The hidden\_states when I use model.generate](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169).

There is also a Transformers issue where `model()` and `model.generate()` hidden states did not match as expected in a Qwen2.5 example: [Hidden states are different for model() and model.generate()](https://github.com/huggingface/transformers/issues/38538). I would not read that issue as applying directly to your setup, but it is a good reminder that extraction details should be explicit.

> **Extraction details I would report**
>
> For geometry/transition work, I would report something like:
> 
> | Detail | Why it matters |
> | --- | --- |
> | `forward()` vs `generate()` | Generation output structures can differ from plain forward outputs. |
> | prompt-token vs generated-token states | Input-side and generation-side states are not always collected the same way. |
> | `hidden_states[-1]` vs `last_hidden_state` | Some models apply normalization or post-processing. |
> | hook point | Residual stream, block output, attention output, MLP output, and post-norm are different objects. |
> | `use_cache` | Cached generation can change what is recomputed and returned. |
> | embedding output counted as layer 0 or not | Important for cross-paper layer indexing. |
> | chat template / special tokens | Role tokens and template boundaries can become hidden confounds. |
> | token position sampled | Last prompt token, generated token, EOS-adjacent token, etc. may behave differently. |
> | per-model vs shared state labeling | Important for interpreting “same state” across models. |
> 
> This is mostly a reproducibility issue, not a criticism. It lets another reader know exactly what object is being treated as the trajectory.

## 7. Probe / format confound caution

One related caution is that high hidden-state separability does not automatically imply a functional internal variable. A recent example is [Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States](https://arxiv.org/abs/2606.02907), where high probe accuracy on Qwen3-14B was explained by format confounds such as source identity, option count, and response length.

I would not treat that as a refutation of your result. I would treat it as a useful reminder to include format/depth/confidence controls before assigning strong functional interpretations to state labels.

In this context, the main question becomes:

> Are B/A/D labels still informative after controlling for layer depth, prompt family, response format, token position, entropy, top-token margin, and generation step?

If yes, the dynamic-grammar interpretation becomes easier to compare.  
If no, the result may still be useful, but the interpretation shifts toward depth/confidence/prompt-format dynamics.

## 8. Possible next tracks

I would split the next steps into three tracks.

### Track A — terminology / literature mapping

Use these as comparison anchors:

- [Markovian Circuit Tracing](https://arxiv.org/abs/2605.20824) for activation-state abstraction and transition matrices.
- [PoLLMgraph](https://aclanthology.org/2024.findings-naacl.294/) for hallucination as internal state-transition dynamics.
- [ICR Probe](https://aclanthology.org/2025.acl-long.880/) for cross-layer hidden-state evolution.
- [Belief State Geometry](https://arxiv.org/abs/2405.15943) for the continuous-geometry side.
- [Tuned Lens](https://arxiv.org/abs/2303.08112) / [LogitLens4LLMs](https://arxiv.org/abs/2503.11667) for prediction-trajectory checks.

### Track B — observational controls

Before moving to large models or causal claims, I would add a small diagnostic table:

- raw vs layer-centered vs layer-z-scored states,
- depth-only baseline,
- entropy/margin-only baseline,
- prompt-family baseline,
- generation-step baseline,
- shuffled-layer transition matrix,
- random Gaussian baseline,
- fixed-layer time-transition matrix.

This is relatively cheap and makes the interpretation much clearer.

### Track C — later predictive / causal validation

Once the observational state definition is stable:

- test whether B→D-like jumps predict factual hallucination or reasoning errors,
- compare correct vs incorrect generations,
- try held-out prompt families,
- try model-family-held-out mapping,
- eventually test centroid patching / activation patching.

For tooling, [TransformerLens](https://github.com/TransformerLensOrg/TransformerLens) is useful for caching and editing activations in supported models, and [nnsight](https://nnsight.net/tutorials/tutorials/causal_mediation_analysis/activation_patching/) has activation-patching style workflows. I would treat that as a later stage, after the observational labels are stable.

## 9. How I would frame the claim

I would avoid framing the current result as either:

- “a universal transformer grammar,” or
- “just a probing artifact.”

A safer and more useful middle formulation might be:

> Under this extraction and labeling setup, there appear to be recurrent activation-derived transition motifs. The next step is to report which motifs survive depth, confidence, prompt-family, step, shuffled-order, and fixed-layer controls.

That keeps the core idea alive while making it easier for other people to reproduce, compare, and extend.

In short: the B/A/D grammar idea seems worth pursuing, but I would make the next iteration less about proving the names and more about making the state abstraction auditable.

---

_[View the full topic](https://discuss.huggingface.co/t/research-from-functional-geometry-to-dynamic-grammar-new-limen-audits-v23-v24-across-7-architectures/177260)._