# Exploring Functional Regimes Inside Small Language Models (Independent Research)

**URL:** <https://discuss.huggingface.co/t/exploring-functional-regimes-inside-small-language-models-independent-research/177219>\
**Category:** Models\
**Created:** [June 29, 2026, 2:23am UTC](https://discuss.huggingface.co/t/exploring-functional-regimes-inside-small-language-models-independent-research/177219 "2026-06-29T02:23:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeanbatuli](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@jeanbatuli](https://discuss.huggingface.co/u/jeanbatuli)\
**Post date:** [June 29, 2026, 2:23am UTC](https://discuss.huggingface.co/t/exploring-functional-regimes-inside-small-language-models-independent-research/177219/1 "2026-06-29T02:23:53Z")

</div>

Hi everyone,

Over the last few months I’ve been working on an independent research project exploring the **internal dynamics** of small and medium-sized language models.

Rather than evaluating models only by their outputs (benchmarks, perplexity, etc.), I’m trying to characterize **how their hidden representations evolve during inference**.

The project currently covers 7 open models:

- GPT-2
- DistilGPT2
- OPT-125M
- Qwen2.5-0.5B-Instruct
- TinyLlama-1.1B-Chat
- Phi-1.5
- Llama-3.2-1B

* * *

## Phase 1 — Dynamic behavior

The first part of the framework studies internal trajectories through hidden-state dynamics.

Instead of asking _“Which model is more accurate?”_, I ask:

- How stable are hidden-state trajectories?
- When do bifurcations appear?
- How does commitment evolve?
- Can different architectures be grouped according to their internal dynamics?

This produced several reproducible dynamical fingerprints and architecture clusters.

* * *

## Phase 2 — Functional organization

The second phase moves away from pure dynamics and investigates whether different functional properties become linearly decodable inside hidden representations.

Across multiple probe experiments I observed evidence that:

- functional information is linearly decodable,
- several functional properties can coexist within the same layer,
- some layers have substantially higher functional capacity than others.

One interesting observation is that the position of these high-capacity regions varies across architectures rather than appearing at identical absolute depths.

* * *

## Phase 3 — Geometry

The result that surprised me the most came from a series of control experiments.

After training linear probes I compared:

- Gaussian perturbations
- Feature permutation
- Orthogonal rotations

Gaussian noise and feature permutation substantially reduced decodability.

Orthogonal rotations, however, preserved it almost entirely.

That suggests (at least empirically) that the functional signal depends more on the **geometry of the representation space** than on specific embedding dimensions.

This seems broadly consistent with ideas discussed in mechanistic interpretability about distributed feature directions.

* * *

## Architecture Clusters

Across several independent audits, the models repeatedly separate into two broad behavioral groups.

**Cluster A**

- GPT-2
- DistilGPT2

These models consistently exhibit similar dynamic and functional profiles.

**Cluster B**

- OPT-125M
- Qwen2.5-0.5B-Instruct
- TinyLlama-1.1B-Chat
- Phi-1.5
- Llama-3.2-1B

Despite architectural differences, these models repeatedly cluster together across multiple analyses.

Seeing the same grouping emerge from different metrics was one of the motivations for continuing the project.

* * *

## Current Direction

I’m now moving from observation toward causal testing.

The next experiments aim to answer questions such as:

- Can perturbing specific functional regions causally alter downstream behavior?
- How much functional information can a single hidden representation simultaneously support?
- Are these organizational principles preserved as model size increases?

* * *

This is entirely independent research, so I’d genuinely appreciate feedback.

I’m especially interested in hearing from people working on:

- mechanistic interpretability,
- representation geometry,
- linear probes,
- sparse autoencoders,
- causal interventions,
- hidden-state analysis.

I’d love to know whether these observations resonate with existing work—or whether there are obvious control experiments I should run next.