# Cross-architectural runtime probability dynamics in transformer LLMs — two clusters not explained by parameter count **URL:** \ **Category:** Research\ **Created:** [June 9, 2026, 3:41am UTC](https://discuss.huggingface.co/t/cross-architectural-runtime-probability-dynamics-in-transformer-llms-two-clusters-not-explained-by-parameter-count/176630 "2026-06-09T03:41:43Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jeanbatuli](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jeanbatuli](https://discuss.huggingface.co/u/jeanbatuli)\ **Post date:** [June 9, 2026, 3:41am UTC](https://discuss.huggingface.co/t/cross-architectural-runtime-probability-dynamics-in-transformer-llms-two-clusters-not-explained-by-parameter-count/176630/1 "2026-06-09T03:41:43Z")
I want to share a finding from a measurement framework I’ve been working on, because the result is counterintuitive enough that I think it might interest people thinking about architectural differences between transformer LLMs. ## The setup I measured the runtime geometry of probability distributions across eight open-source attention-based transformers ranging from 70M to 1.3B parameters: Pythia-70M, DistilGPT-2, GPT-2, OPT-125M, Pythia-160M, Qwen2.5-0.5B, TinyLlama-1.1B, and Phi-1.5. For each (token, layer) point during inference, the framework computes geometric properties of the probability distribution over the vocabulary: entropy, concentration on the top candidates, competition between the leading and runner-up tokens, dispersion above a 1% threshold. From these metrics, a bicephalic operator separates two distinct geometric tensions that probability distributions can carry, which I label G (concentration pole) and D (competition pole). The ratio between mean G and mean D, what I call the GD\_ratio, becomes a per-model signature. ## What I found The eight models do not vary continuously on the GD\_ratio. They partition into two clusters with no overlap and roughly an order of magnitude of gap between them: ## GPT-2 GD\_ratio 2.458 Phi-1.5 GD\_ratio 1.764 DistilGPT-2 GD\_ratio 1.577 Qwen-0.5B GD\_ratio 0.079 OPT-125M GD\_ratio 0.074 Pythia-70M GD\_ratio 0.059 Pythia-160M GD\_ratio 0.039 TinyLlama-1.1B GD\_ratio 0.021 The cluster split appears on three independent components of the operator: the GD\_ratio itself, the mean G alone, and the mean D alone. The separation is not an artifact of one metric. The interesting part is what does not explain the clustering. Parameter count does not. GPT-2 has 124M parameters and is in the upper cluster. OPT-125M has 125M parameters and is in the lower cluster. Phi-1.5 has 1.3B parameters and sits with GPT-2. TinyLlama-1.1B has roughly the same size as Phi-1.5 and sits with OPT. ## What might explain it (hypothesis only) The most parsimonious pattern I can see is that the upper cluster shares characteristics of training corpus curation. Phi-1.5 was trained on heavily curated synthetic data. GPT-2 and DistilGPT-2 share the original GPT-2 WebText distribution and tokenizer, which had its own filtering protocol. The lower cluster spans more heterogeneous training corpora, including older (OPT, Pythia) and newer (Qwen, TinyLlama) architectures trained on relatively unfiltered web text. I want to be careful here: this is a hypothesis, not a finding. I do not have an experimental setup that isolates training corpus from architecture choices. The hypothesis is consistent with the data but cannot be established by it. ## Why this might matter If the two-cluster structure generalizes, any tool or analysis that implicitly assumes a single dynamic profile across transformer models will produce inconsistent results depending on which cluster the target model falls into. This includes calibration techniques, uncertainty estimation methods, and probably some interpretability approaches that were tuned on one architectural family and may not transfer cleanly to the other. ## Other observations in the same study A few things worth noting briefly: - The framework also defines a five-state taxonomy of dynamic regimes (stable, hidden turbulence, surface branching, committed, full bifurcation). The full bifurcation state turns out to be consistently transient across architectures: on three primary models tested in depth, its self-transition probability is 0.023 (GPT-2) or exactly 0.000 (OPT-125M, Qwen-0.5B). Models pass through this regime, they do not settle into it. - Three models tested under controlled hidden-state perturbation respond in qualitatively different ways. GPT-2 absorbs the perturbation with state percentages shifting by less than 1.5 points. OPT-125M converts the perturbation into surface dispersion (branching state rises +12.5 points). Qwen-0.5B destabilizes its dominant state (stable state drops -18.8 points). Three architectural perturbation signatures, same input noise. - One model (Phi-1.5) produces an anomalous taxonomy distribution under the standard threshold rule. I report it openly as needing dedicated investigation rather than smoothing it over. ## What I’m not claiming The panel is eight models, all under 1.3B parameters. The two-cluster structure could collapse, stretch, or restructure when extended to 7B+ models. I have not validated on non-transformer architectures within this study. The work is single-author and has not been independently replicated. The training-corpus hypothesis is offered, not established. I included explicit “limited findings” and “rejected claims” sections in the paper, listing five things in each category that initial intuitions suggested but that the data either partially support or actively reject. I treat this as central to the framework’s credibility. ## Why I’m posting I would be interested in hearing whether anyone working with larger or more architecturally diverse models has observed similar partitioning phenomena in their own measurements, whether on attention, hidden states, gradients, or any other intermediate quantity. The two-cluster structure felt unexpected enough that I want to understand whether it is a transformer-wide phenomenon, an artifact of the parameter range I tested, or something specific to the particular operator I defined. I would also be interested in alternative interpretations of the cluster split beyond the training-corpus hypothesis. Possible candidates I have considered but cannot test from this panel alone: pre-norm vs post-norm architecture, tokenizer differences, attention head configurations, intermediate layer dimensionality, positional encoding choices. ## Where the details are Full methodology, all tables, the explicit limitations section, and the list of rejected claims are in the preprint on Zenodo: > **[A Runtime Trajectory Dynamics Framework for Large Language Models](https://zenodo.org/records/20602685)** > > Most existing observability tools for large language models analyze attention patterns or final outputs in isolation, leaving the runtime dynamics of probability distributions under-characterized. We introduce V20, a framework that measures the... Happy to discuss the operator definition, the threshold methodology, the cluster finding, or any concern about the panel size and statistical robustness. ---
**Author:** ![jeanbatuli](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jeanbatuli](https://discuss.huggingface.co/u/jeanbatuli)\ **Post date:** [June 9, 2026, 12:27pm UTC](https://discuss.huggingface.co/t/cross-architectural-runtime-probability-dynamics-in-transformer-llms-two-clusters-not-explained-by-parameter-count/176630/3 "2026-06-09T12:27:20Z")
This is exceptional feedback. Thank you for taking the time to dissect the operator and the methodology with such rigor. You’ve pinpointed the exact confounds that keep me up at night, particularly regarding calibration and activation norms. I want to address your points directly, as they align perfectly with my own internal audits (some of which are in the “Rejected Claims” section of the preprint, but deserve more prominence): **1. The Temperature/Calibration Confound (Priority #1)** You are absolutely right. `GD_ratio` could simply be a proxy for per-model “confidence temperature” driven by training objectives (e.g., label smoothing, final-LN gain). - **Action:** I will run the scalar temperature fitting test you suggested. If the clusters collapse after entropy-matching, then `GD_ratio` is indeed a calibration metric, not a dynamic one. If they hold, it proves there is a structural geometric difference beyond simple sharpness. This is the cheapest and most decisive test. I’ll update the repo with these results. **2. Vocabulary Invariance** Good catch on the vocab size disparity (32k vs 152k). - **Current Mitigation:** My operator uses relative thresholds (top-k competition, dispersion above 1% _of the mass_), which helps, but raw entropy is definitely biased by V_V_. - **Action:** I will recompute using entropy normalized by log⁡(V)log(_V_) and also try the “top-p mass geometry” approach you mentioned. This should isolate the shape of the distribution from its granularity. **3. Perturbation Magnitude & Nulls** This is a critical methodological flaw in many interpretability studies. - **Action:** I currently inject fixed Gaussian noise. I will switch to **RMS-normalized noise** (scaled to each layer’s activation RMS) to ensure equitable perturbation strength. I will also add the **zero-perturbation null** to baseline the measurement pipeline’s stability. This will strengthen the claim that the “three signatures” are architectural, not artifacts of scale. **4. Error Bars & Bootstrapping** Agreed. Visual separation on n=8_n_=8 is suggestive, not definitive. - **Action:** I will bootstrap the `GD_ratio` over prompts and seeds to generate Confidence Intervals (95% CI) for each model. This will quantify the “no overlap” claim statistically. **5. The Large Model Test (Gemma & Softcapping)** This is a brilliant concrete test. Gemma’s logit softcapping is a known architectural constraint that directly bounds concentration metrics. - **Hypothesis:** If Gemma clusters with the “Low GD” group solely due to softcapping, it confirms that architecture can override training corpus signals. - **Action:** I am currently limited by compute for full 12B runs, but I can run inference on smaller softcapped variants or use existing hidden-state logs if available. Alternatively, I can simulate softcapping on my current panel to see if it forces them into the lower cluster. **Next Steps:** I’m prioritizing the **Temperature Normalization** and **RMS-scaled Perturbation** audits this week. These are high-leverage controls that will either validate the core hypothesis or refine it into a more precise statement about calibration vs. dynamics. I’ll post an update here once the bootstrapped CIs and temperature-calibrated ratios are ready. Thanks again for pushing the rigor bar higher this is exactly how robust science gets done. Best, Jean-Denis ---
**Author:** ![jeanbatuli](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jeanbatuli](https://discuss.huggingface.co/u/jeanbatuli)\ **Post date:** [June 9, 2026, 3:37pm UTC](https://discuss.huggingface.co/t/cross-architectural-runtime-probability-dynamics-in-transformer-llms-two-clusters-not-explained-by-parameter-count/176630/4 "2026-06-09T15:37:57Z")
Thanks for this — and you were right. Ran the temperature normalization test you suggested as priority 1. The result is a partial falsification of the strongest interpretation of the finding. Setup: fitted a per-model scalar temperature T to match a common target entropy across the panel (target = 3.01), recomputed geometry on the calibrated logits, recomputed clustering. What happened: two models migrate between clusters after calibration. GPT-2 and Phi-1.5 both move. The raw clustering structure is therefore substantially driven by effective logit temperature, not purely by runtime dynamics. What this falsifies: “Raw GD\_ratio directly measures model dynamics independently of calibration.” This interpretation is rejected by the test you proposed. What may still hold: residual structure after calibration. Two clusters still appear post-calibration, but their composition changes — so the question becomes “is there a calibration-independent component to the clustering, and if so what does it measure?” That requires the additional controls you listed: vocab normalization (log V), top-p truncation, perturbation magnitude-matched to per-layer activation RMS, bootstrap CIs on the gap. None of those are done yet. The published version of the V20 preprint (deposited yesterday) overstates the dynamical interpretation of the raw GD\_ratio. I am preparing a revision that incorporates the temperature audit explicitly. The falsification is going into the rejected-claims section. The remaining structure question is moving into limited-findings with the controls you specified as the protocol for elevation. The attention-based findings from companion work (different observation level than logits) are not affected by this confound and remain the direction I will continue to develop. Genuinely useful review. The “temperature first because it kills the result or makes it much stronger” framing was the right priority. ---
**Author:** ![jeanbatuli](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jeanbatuli](https://discuss.huggingface.co/u/jeanbatuli)\ **Post date:** [June 28, 2026, 4:29pm UTC](https://discuss.huggingface.co/t/cross-architectural-runtime-probability-dynamics-in-transformer-llms-two-clusters-not-explained-by-parameter-count/176630/7 "2026-06-28T16:29:32Z")
Thanks for checking back and yes, I definitely kept going down the rabbit hole. Your previous comments ended up changing the direction of the project in a meaningful way. Following your suggestion, I treated logit calibration as a potential confound rather than assuming GD\_ratio was measuring intrinsic dynamics. I then ran a series of follow-up audits: - **Temperature normalization audit (V22.5):** after matching entropy across models, a substantial part of the original structure changed. Temperature is indeed a major driver. - **Vocabulary-invariant audit (V22.6):** normalizing for vocabulary size and tokenizer granularity only partially altered the structure. The result was **PARTIAL** , suggesting vocabulary explains some, but not all, of the signal. - **Bootstrap confidence audit (V22.7):** only part of the pairwise differences remained statistically robust, so I downgraded the strength of the original claim rather than strengthening it. - **Residual structure audit (V22.8b):** with the pipeline corrected, the remaining signal is still strongly associated with effective temperature, so I’ve revised the interpretation again. At this point I no longer describe GD\_ratio as a direct measure of “internal dynamics.” A more accurate interpretation is that it captures aspects of **output distribution geometry** , with temperature acting as an important confounding factor. The interesting part now is the residual after controlling for those obvious mechanisms. Rather than trying to defend the original interpretation, I’m using the negative results to refine the framework. Your suggestion about **per-token / entropy-bin calibration** is still on my list. I think that’s probably the next meaningful control because it asks whether any residual survives after local calibration instead of only matching the global mean. I’m also moving beyond logits toward hidden-state and attention analyses, but with the same philosophy you recommended: identify the boring explanations first (calibration, tokenizer effects, attention sinks, architectural constraints) before making stronger claims. So yes the rabbit hole got deeper, but also narrower. The framework is becoming less ambitious in its claims and, I hope, more trustworthy because of that. Thanks again for pushing on the weak points rather than just the positive results.