# Dev-4B: quick calibrated decisions on a document, reason only when the router flags risk

**URL:** <https://discuss.huggingface.co/t/dev-4b-quick-calibrated-decisions-on-a-document-reason-only-when-the-router-flags-risk/182924>\
**Category:** Show and Tell\
**Created:** [October 5, 2026, 6:10pm UTC](https://discuss.huggingface.co/t/dev-4b-quick-calibrated-decisions-on-a-document-reason-only-when-the-router-flags-risk/182924 "2026-10-05T18:10:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![suhaas-teja](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@suhaas-teja](https://discuss.huggingface.co/u/suhaas-teja)\
**Post date:** [October 5, 2026, 6:10pm UTC](https://discuss.huggingface.co/t/dev-4b-quick-calibrated-decisions-on-a-document-reason-only-when-the-router-flags-risk/182924/1 "2026-10-05T18:10:29Z")

</div>

I released Dev-4B — a small local model that answers typed questions about a document (choice / yes-no / score) with calibrated confidence, and only thinks step by step when a built-in router predicts the quick answer is likely wrong.

Space: [Dev-4B - a Hugging Face Space by suhaas-teja](https://huggingface.co/spaces/suhaas-teja/Dev-4B-demo)  
Model: [suhaas-teja/Dev-4B · Hugging Face](https://huggingface.co/suhaas-teja/Dev-4B)

MLX 8-bit build is on the same profile as Dev-4B-MLX-8bit (search that name on Hugging Face).

Stack: Qwen3-4B-Instruct-2507 + ~133 MB of add-ons (LoRA adapter switched off while reading the document and on for the question, decision head, temperatures for calibration, tiny router). Base generation is unchanged when add-ons are off.

Results on 7,100 frozen test questions:

- Always quick: 76.3% accuracy, 0.16 s/q
- Always reason: 78.0%, 10.2 s/q
- Dev-4B (route): 81.8%, 1.37 s/q  
Arithmetic/counting is where escalation helps most. On a 16 GB Mac (MLX 8-bit), five questions ≈ 3 s, ~6 GB peak.

Limits to be clear about: weaker on unseen task types than on trained ones; reasoning is still a 4B model; English; docs ~4k chars in training; NC weights (CC BY-NC-SA); no Ollama/LM Studio (LoRA + decision head + router). Question format follows System One; not affiliated with TypeSafe.

Most open System One clones are decision-only. Dev-4B’s wedge is local calibrated decisions plus a router that escalates to CoT when the quick pass looks wrong — for Mac indie / offline routing experimenters.

Would love feedback from people running local decision models — especially Mac numbers and cases where the router should/shouldn’t escalate.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 10, 2026, 10:22am UTC](https://discuss.huggingface.co/t/dev-4b-quick-calibrated-decisions-on-a-document-reason-only-when-the-router-flags-risk/182924/2 "2026-10-10T10:22:42Z")

</div>

I gave it a quick try:

* * *

## Scope of the test

I ran the released [Dev-4B model and code](https://huggingface.co/suhaas-teja/Dev-4B) on a Linux/NVIDIA L4 using BF16, with **26 short fictional document questions in 13 paired groups**. For each question I recorded the quick typed answer and its top probability, the released router’s score and escalation choice, and the result of a separately forced step-by-step generation. I designed these examples to look at routing behavior, **not** to repeat the refund/stamp demos.

This was **not** a Mac/MLX test, an independent reproduction of the published 7,100-question evaluation, or a representative real-world benchmark. I can offer concrete **should/shouldn’t-escalate candidates** ; I cannot offer Mac performance numbers.

| Observation | In this probe |
| --- | --- |
| Quick correct | **21/26** |
| Router escalations, published threshold ~0.516 | **6/26** |
| Forced outputs with parseable final label | **24/26** |
| Forced correct among the 24 parseable outputs | **24/24** |

Two forced generations reached the 384-token ceiling before emitting a parseable final `Answer:` line. I kept those as **unscored** , rather than assuming they were right or wrong. Source and base revisions were pinned; the model/LoRA/head load checks passed.

## Cases directly relevant to the request

The most useful contrast is between these **two different routing decisions** :

| Question | Gold | Quick answer (top probability) | Router score | Escalated? | Forced final |
| --- | --- | --- | --- | --- | --- |
| Sum of blue rows (7 + 9 + 5) | **21** | 19 (0.699), wrong | **0.328** | No | **21** , correct |
| Number of distinct colors (amber/cyan/teal/violet) | **4** | 4 (0.960), correct | **0.752** | Yes | **4** , correct |

For the **blue-row sum** , the quick decision was wrong but the forced reasoning pass repaired it. The router left the case on the quick path. This is a concrete **missed repair opportunity** on this input.

For the **distinct-color count** , the quick decision was already correct and the router still escalated. The forced result did not improve correctness. This is an **escalation with no observed accuracy gain in hindsight** ; it is not proof that the router should have known in advance that reasoning was unnecessary.

The other direction matters just as much. The router also caught **two fairly confident quick errors** :

| Question | Gold | Quick answer (top probability) | Router score | Forced final |
| --- | --- | --- | --- | --- |
| Default Dock A; X9 alone is an exception to Dock B; carton C13 has Y2 | **Dock A** | Dock B (0.874) | 0.540, escalated | **Dock A** |
| Review only for a total **strictly greater than 18** ; the three loads total 18 | **No** | Yes (0.804) | 0.779, escalated | **No** |

There was also a successful repair of the **non-blue row sum** : quick 21 versus gold 27; router 0.598 (escalated); forced answer 27. A neighboring case with total load 19 was quick-correct at Yes but also escalated, with no observed change after reasoning.

These examples are why I would **not** interpret the blue-row miss as an argument against the learned router. In this same small test, it detected high-confidence errors that a naive low-confidence-only cutoff might fail to select. The fixed inputs and full results are expanded below.

## A small comparison at the same escalation count

On the **24 items for which a final forced answer was actually parseable** , I reconstructed each policy’s selected final label from the saved quick and forced outputs:

| Selection rule | Correct / 24 | Additional generations selected |
| --- | --- | --- |
| Always quick | **20/24** | 0 |
| Published router threshold | **23/24** | 5 |
| Five lowest quick top probabilities | **24/24** | 5 |
| Always use forced generation | **24/24** | 24 |

The two five-generation selections differ in **only one item each**. Both selected `except_y2`, `load_at_18`, `load_at_19` and `sum_non_blue_values`. The router additionally selected **correct `count_distinct_colors`** ; confidence ranking instead selected **wrong-but-repairable `sum_blue_values`**.

This is a useful diagnostic disagreement, **not a claim that confidence-only routing beats Dev-4B in general**. The comparator was selected after seeing how many items the released router escalated; the 26 questions are hand-built, some share documents, and equal numbers of generations do not imply equal token cost or wall time. The 23/24 and 24/24 values are **conditional on the common scorable 24** , not estimates of performance on the author’s benchmark. I did not rerun the `/v1/auto` HTTP endpoint for final-policy throughput; those policy outputs were reconstructed from the paired traces.

## A possible way to interpret the contrast

The two questions **“Is the quick answer likely wrong?”** and **“Will another generation pass actually improve the final answer?”** are related, but not identical. The [released router](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/router.py) learns an error-risk signal from the decision-token representation and three confidence features. That is well aligned with detecting suspicious decisions. When the action is specifically extra generation, the _chance of successful repair_ and the _cost of that attempt_ may also matter.

I would treat this as an analysis axis, not a request to retrain anything. The existing architecture of fast typed decisions plus selective generation can stay unchanged. Also, **`router_score` is not automatically a calibrated probability of an error** : e.g., 0.752 is the routing score here, not independently validated “75.2% wrong.” Temperature-scaled option probabilities and the router’s scoring function have different roles.

## Low-cost next checks, depending on the goal

**If per-item evaluation traces already exist:** compare quick label, gold, router score, generated final label, and generated tokens for the _same items_. Separate wrong→right, right→wrong, right→right, wrong→wrong, and **unscored/truncated**. At a fixed number of escalations, inspect cases where router ranking and simple confidence ranking disagree. Only if cost is a central deployment concern would I move on to matching actual generated tokens or wall time.

**If item-level traces are not available:** the fixed blue/non-blue sum, X9/Y2 exception, and 18/19 strict-threshold pairs below are low-cost starting controls. Vary one dimension at a time: target row/predicate, candidate order, threshold boundary, or irrelevant document content. Exact prompt and option order matter to the quick classification head.

**If the goal is Mac deployment:** this L4 run cannot substitute for a test of the [released MLX 8-bit model](https://huggingface.co/suhaas-teja/Dev-4B-MLX-8bit). Useful measurements would identify Mac hardware, model/MLX version, document length, number of questions sharing the prefill, warm/cold state, generated tokens and definition of memory.

**If the goal is reliably typed output:** distinguish generated reasoning from a completed and parsed final choice. The two capped outputs here show why this matters to evaluation. And if a document does not contain the required fact, more reasoning is not automatically a substitute for an explicit unknown/review path.

Below are exact example inputs, the 26-case outcome list, paired fixture descriptions, and the pinned run conditions. They are meant as reproducibility aids, **not a grading of the project or a new leaderboard**.

> **Exact case inputs for the four most informative routing behaviors**
>
> ### 1. A repairable quick error that was not escalated: `sum_blue_values`
> 
> Document:
> 
> ```plaintext
> Audit lines: row A = blue, 7 points; row B = yellow, 12 points; row C = blue, 9 points; row D = red, 15 points; row E = blue, 5 points. A row contributes its points only when the question selects its color.
> 
> ```
> 
> Question and options (in actual order):
> 
> ```json
> {"type":"choice","instructions":"Total points on blue rows?","criteria":{"19":null,"21":null,"33":null,"48":null}}
> 
> ```
> 
> The three blue rows are A, C, and E: `7 + 9 + 5 = 21`. The quick decision was `19` (`p_top = 0.69911873`). Router score was `0.32785931`, below `0.51559174`, so no escalation. Forced generation produced the final label `21` in 197 generated tokens. This establishes a repairable missed quick error **for this particular input and execution path**. It does not establish the prevalence of that failure, nor does it prove why the router scored it low.
> 
> A useful feature of this example is that the available answers include plausible distractors. In particular, `19` could arise from a mistaken selection or arithmetic path, but I cannot infer the exact hidden computation performed by the quick head. I would not describe the cause as “it forgot the last blue row” unless an additional probe isolates that interpretation. The reliable observation is narrower: wrong selected label; router below threshold; deterministic generated label correct.
> 
> ### 2. Escalation with no observed correctness gain: `count_distinct_colors`
> 
> Document:
> 
> ```plaintext
> Shelf labels in order: amber, cyan, amber, teal, cyan, cyan, violet, teal. Each occurrence counts as one label.
> 
> ```
> 
> Question and options:
> 
> ```json
> {"type":"choice","instructions":"How many different label colors appear?","criteria":{"3":null,"4":null,"5":null,"8":null}}
> 
> ```
> 
> Distinct labels: `amber`, `cyan`, `teal`, `violet`, so the answer is `4`. Quick label `4` (`p_top = 0.96007693`); router score `0.75211662` (above the threshold); forced final label `4` in 181 generated tokens. The router deliberately chose a longer path even though the quick answer happened to be correct. This is a reasonable **cost/benefit inspection candidate** , not enough to label the router miscalibrated or broken.
> 
> The wording matters: the document includes repeated occurrences while the question asks for distinct _types_. Replacing the question with “How many cyan labels?” would test a different counting operation. In fact the paired occurrence-count case is quick-correct and not escalated in this run. That pair might help distinguish task-specific representation effects from a simple low-confidence rule, although one pair is far too little to establish the mechanism.
> 
> ### 3. A high-confidence quick error detected by the router: `except_y2`
> 
> Document:
> 
> ```plaintext
> Default storage plan: all cartons go to Dock A. Explicit exception: cartons with code X9 go to Dock B. List: C12 code X9, C13 code Y2. The specific exception takes precedence over default.
> 
> ```
> 
> Question and options:
> 
> ```json
> {"type":"choice","instructions":"Where should carton C13 go?","criteria":{"Dock A":null,"Dock B":null,"Undecided":null}}
> 
> ```
> 
> Carton C13 has Y2, not X9, so the exception does not apply; Dock A is the gold answer. The quick head selected Dock B (`p_top = 0.87404186`). The router scored `0.54035252`, just above its published threshold, and the generated answer was Dock A after 211 tokens. This is a useful positive control for the router’s **nontrivial signal** , since a threshold based only on very low quick confidence could easily miss a confident wrong label. It does not prove which hidden feature caused the router’s higher score.
> 
> The paired `except_x9` item instead asks about C12. That input legitimately activates the exception and was quick-correct at Dock B, with a low router score. Together they give an interpretable change in the correct conditional rule while keeping the fictional storage policy fixed.
> 
> ### 4. A strict numeric boundary caught by the router: `load_at_18`
> 
> Document:
> 
> ```plaintext
> A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 5 units.
> 
> ```
> 
> Question:
> 
> ```json
> {"type":"noul","instructions":"Does this shipment require extra review?"}
> 
> ```
> 
> Here `6+7+5=18`, and `18 > 18` is false. Gold No. Quick was Yes (`p_top = 0.80355495`), router score `0.77924889`, and forced generation concluded No after 158 tokens. The small router flagged a fairly confident error, and the reasoning pass fixed it.
> 
> The paired input changes the last crate from 5 to 6; the total becomes 19 and the gold answer becomes Yes. On that paired input, quick Yes was already correct, but the router still escalated (`0.83207101`), and generation remained Yes. The pair is useful for comparing the _direction of a strict inequality at the boundary_ with the _routing cost_ spent on either side; it does not establish a systematic over-triggering pattern from two samples.

> **Complete 26-case outcome inventory (observed, not population estimates)**
>
> Every row below was read from the returned result ZIP for this probe. `top-p` is the largest quick label probability, not a measure of calibration on this tiny panel. `risk` is the router score. `Y/N` indicates whether the published threshold triggered an escalation. `—` in the forced column means that no final answer was parseable before the configured cap; it does **not** count as an incorrect answer. Each selected label is displayed as a human-readable typed label, not its internal choice letter.
> 
> | ID | Gold | Quick | top-p | risk | Escalate | Forced | Generated tokens |
> | --- | --- | --- | --- | --- | --- | --- | --- |
> | `rev_target_p82` | Amber Pier | Amber Pier | 0.977 | 0.027 | N | Amber Pier | 286 |
> | `rev_neighbor_p83` | Willow Gate | Willow Gate | 0.973 | 0.030 | N | Willow Gate | 258 |
> | `hyphen_kdash07` | Room East | Room East | 0.977 | 0.004 | N | — (capped) | 384 |
> | `hyphen_k07` | Room West | Room West | 0.981 | 0.037 | N | Room West | 219 |
> | `count_cyan_occurrences` | 3 | 3 | 0.971 | 0.474 | N | 3 | 169 |
> | `count_distinct_colors` | 4 | 4 | 0.960 | 0.752 | Y | 4 | 181 |
> | `sum_blue_values` | 21 | 19 | 0.699 | 0.328 | N | 21 | 197 |
> | `sum_non_blue_values` | 27 | 21 | 0.470 | 0.598 | Y | 27 | 250 |
> | `and_r2` | no | no | 0.978 | 0.002 | N | no | 132 |
> | `and_r3` | yes | yes | 0.969 | 0.006 | N | yes | 161 |
> | `except_x9` | Dock B | Dock B | 0.990 | 0.016 | N | Dock B | 203 |
> | `except_y2` | Dock A | Dock B | 0.874 | 0.540 | Y | Dock A | 211 |
> | `missing_end_time` | Not recorded | Not recorded | 0.991 | 0.051 | N | Not recorded | 134 |
> | `stated_start` | 14:00 | 14:00 | 0.987 | 0.004 | N | 14:00 | 227 |
> | `join_manifest31` | Hall Copper | Hall Copper | 0.989 | 0.034 | N | Hall Copper | 123 |
> | `join_manifest32` | Hall Silver | Hall Silver | 0.983 | 0.007 | N | Hall Silver | 159 |
> | `load_at_18` | no | yes | 0.804 | 0.779 | Y | no | 158 |
> | `load_at_19` | yes | yes | 0.764 | 0.832 | Y | yes | 136 |
> | `permute_ab` | Sol | Sol | 0.992 | 0.029 | N | Sol | 223 |
> | `permute_ba` | Sol | Sol | 0.985 | 0.026 | N | Sol | 169 |
> | `not_inspected_b` | Not inspected | Not inspected | 0.992 | 0.013 | N | Not inspected | 220 |
> | `unknown_inspected_c` | Unrecorded | Unrecorded | 0.988 | 0.070 | N | Unrecorded | 275 |
> | `supervisor_of_vale` | Ibis | Ibis | 0.986 | 0.014 | N | Ibis | 187 |
> | `supervisee_of_ibis` | Vale | Vale | 0.984 | 0.007 | N | Vale | 204 |
> | `urgent_first` | Q1 | Q1 | 0.986 | 0.077 | N | Q1 | 289 |
> | `normal_first` | Q4 | Q1 | 0.455 | 0.705 | Y | — (capped) | 384 |
> 
> These paired rows are a transparent outcome log, not 26 independent deployment samples.
> 
> The four quick/forced outcome types (wrong→right, right→wrong, right→right, wrong→wrong) should be kept separate from **unscored** generated answers.
> 
> For this actual run, the four **parsed** quick-wrong/forced-correct cases were `sum_blue_values`, `sum_non_blue_values`, `except_y2`, and `load_at_18`. Three were escalated; `sum_blue_values` was not. No right-to-wrong outcome was observed among the 24 parsed forced generations, but that absence should not be generalized to an unseen dataset.

> **All 26 input fixtures, grouped by the 13 controlled contrasts**
>
> These are the **exact fictional document strings, question strings, choice order, and gold labels** used in the L4 probe, rearranged by pair to save space. The fixed `state` for a pair is shown once unless the two states differ. The `gold` value is the test author’s annotation, **not a field accepted by Dev-4B’s API**. `noul` is the yes/no question type in the published API. Options below retain their insertion order. This section plus the outcome inventory makes the reported selection disagreement inspectable; it is not a claim that these fixtures are publicly hosted as a benchmark dataset.
> 
> **Pair `revision` — 2 questions**
> 
> > Facility memo 8:10: send parcel P-82 to Cedar Bay. At 09:15 the dispatcher changes P-82 to Amber Pier. At 10:20 the dispatcher changes parcel P-83 (not P-82) to Willow Gate. Latest valid instruction for each parcel supersedes its earlier instruction.
> 
> - `rev_target_p82` (`choice`): **Final destination for P-82?** Choices, in order: Cedar Bay; Amber Pier; Willow Gate; Not recorded. Gold: **Amber Pier**.
> - `rev_neighbor_p83` (`choice`): **Final destination for P-83?** Choices, in order: Cedar Bay; Amber Pier; Willow Gate; Not recorded. Gold: **Willow Gate**.
> 
> **Pair `identifier` — 2 questions**
> 
> > Registry: sample K-07 belongs to crate D4. Sample K07 belongs to crate A9. Crate D4 is stored in Room East; crate A9 is stored in Room West. The dash is significant in sample identifiers.
> 
> - `hyphen_kdash07` (`choice`): **Which room holds sample K-07?** Choices, in order: Room East; Room West; Not recorded. Gold: **Room East**.
> - `hyphen_k07` (`choice`): **Which room holds sample K07?** Choices, in order: Room East; Room West; Not recorded. Gold: **Room West**.
> 
> **Pair `count_scope` — 2 questions**
> 
> > Shelf labels in order: amber, cyan, amber, teal, cyan, cyan, violet, teal. Each occurrence counts as one label.
> 
> - `count_cyan_occurrences` (`choice`): **How many entries are labelled cyan?** Choices, in order: 2; 3; 4; 5. Gold: **3**.
> - `count_distinct_colors` (`choice`): **How many different label colors appear?** Choices, in order: 3; 4; 5; 8. Gold: **4**.
> 
> **Pair `filtered_sum` — 2 questions**
> 
> > Audit lines: row A = blue, 7 points; row B = yellow, 12 points; row C = blue, 9 points; row D = red, 15 points; row E = blue, 5 points. A row contributes its points only when the question selects its color.
> 
> - `sum_blue_values` (`choice`): **Total points on blue rows?** Choices, in order: 19; 21; 33; 48. Gold: **21**.
> - `sum_non_blue_values` (`choice`): **Total points on non-blue rows?** Choices, in order: 21; 27; 36; 48. Gold: **27**.
> 
> **Pair `logic_and` — 2 questions**
> 
> > A package is ready to release only if both tests pass: seal intact AND label verified. Package R2: seal intact, label not verified. Package R3: seal intact, label verified. No other overrides exist.
> 
> - `and_r2` (`noul`): **Is package R2 ready to release?** Choices, in order: No; Yes (implicit). Gold: **no**.
> - `and_r3` (`noul`): **Is package R3 ready to release?** Choices, in order: No; Yes (implicit). Gold: **yes**.
> 
> **Pair `policy_priority` — 2 questions**
> 
> > Default storage plan: all cartons go to Dock A. Explicit exception: cartons with code X9 go to Dock B. List: C12 code X9, C13 code Y2. The specific exception takes precedence over default.
> 
> - `except_x9` (`choice`): **Where should carton C12 go?** Choices, in order: Dock A; Dock B; Undecided. Gold: **Dock B**.
> - `except_y2` (`choice`): **Where should carton C13 go?** Choices, in order: Dock A; Dock B; Undecided. Gold: **Dock A**.
> 
> **Pair `missingness` — 2 questions**
> 
> > Meeting card for Session M4: conference room Atlas, moderator Wren, begins at 14:00. No duration or end time is listed on this card.
> 
> - `missing_end_time` (`choice`): **When does Session M4 end?** Choices, in order: 14:30; 15:00; 16:00; Not recorded. Gold: **Not recorded**.
> - `stated_start` (`choice`): **When does Session M4 begin?** Choices, in order: 13:30; 14:00; 15:00; Not recorded. Gold: **14:00**.
> 
> **Pair `three_hop` — 2 questions**
> 
> > Index: manifest N31 lists coil T8; manifest N32 lists coil T9. Coil T8 was assigned to rack E6; coil T9 was assigned to rack E8. Rack E6 is in Hall Copper; rack E8 is in Hall Silver. Always follow the manifest-to-coil-to-rack link.
> 
> - `join_manifest31` (`choice`): **Which hall contains the coil on manifest N31?** Choices, in order: Hall Silver; Hall Copper; Not recorded. Gold: **Hall Copper**.
> - `join_manifest32` (`choice`): **Which hall contains the coil on manifest N32?** Choices, in order: Hall Silver; Hall Copper; Not recorded. Gold: **Hall Silver**.
> 
> **Pair `strict_gt` — 2 questions**
> 
> - **State for `load_at_18`:** A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 5 units.
> - `load_at_18` (`noul`): **Does this shipment require extra review?** Choices, in order: No; Yes (implicit). Gold: **no**.
> - **State for `load_at_19`:** A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 6 units.
> - `load_at_19` (`noul`): **Does this shipment require extra review?** Choices, in order: No; Yes (implicit). Gold: **yes**.
> 
> **Pair `option_position` — 2 questions**
> 
> > Responsibility sheet: feature Cedar belongs to owner Uma; feature Willow belongs to owner Sol. Work order WO-19 concerns feature Willow. Do not use the creator of the work order to assign an owner: the creator is Uma.
> 
> - `permute_ab` (`choice`): **Who owns the feature in WO-19?** Choices, in order: Uma; Sol; Not recorded. Gold: **Sol**.
> - `permute_ba` (`choice`): **Who owns the feature in WO-19?** Choices, in order: Not recorded; Sol; Uma. Gold: **Sol**.
> 
> **Pair `missing_vs_no` — 2 questions**
> 
> > Inventory sheet: panel A has an inspection stamp; panel B is marked “not inspected”. Panel C has a blank inspection field. Blank means unrecorded, not an explicit negative.
> 
> - `not_inspected_b` (`choice`): **What is the inspection status of panel B?** Choices, in order: Inspected; Not inspected; Unrecorded. Gold: **Not inspected**.
> - `unknown_inspected_c` (`choice`): **What is the inspection status of panel C?** Choices, in order: Inspected; Not inspected; Unrecorded. Gold: **Unrecorded**.
> 
> **Pair `direction` — 2 questions**
> 
> > Workshop handoff record: technician Nova supervises technician Ibis. Technician Ibis supervises technician Vale. Nobody else supervises anyone in this record. The relation “supervises” is directional.
> 
> - `supervisor_of_vale` (`choice`): **Who directly supervises Vale?** Choices, in order: Nova; Ibis; Vale; Not recorded. Gold: **Ibis**.
> - `supervisee_of_ibis` (`choice`): **Who is directly supervised by Ibis?** Choices, in order: Nova; Ibis; Vale; Not recorded. Gold: **Vale**.
> 
> **Pair `queue_sort` — 2 questions**
> 
> > Queue order after scanning: Q4, Q1, Q3, Q2. Only Q1 and Q3 are tagged urgent. Process urgent items in their order of appearance, then process the rest in their order of appearance.
> 
> - `urgent_first` (`choice`): **Which item is processed first?** Choices, in order: Q4; Q1; Q3; Q2. Gold: **Q1**.
> - `normal_first` (`choice`): **Which item is processed third overall?** Choices, in order: Q4; Q1; Q3; Q2. Gold: **Q4**.
> 
> The paired design changes a target, predicate, option order, or boundary value while retaining as much context as possible. It should not be treated as 26 statistically independent samples. The exact prompt construction uses the [Dev-4B template](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/template.py); a reimplementation that changes the template or candidate order would be testing a different input.

> **How the comparison was calculated and what it does not establish**
>
> ### Pinned environment and result integrity
> 
> ```plaintext
> Dev-4B revision: e4c9005aeffd50df1fcc2e89e498ffa0cf142dab
> Qwen3-4B-Instruct-2507 revision: cdbee75f17c01a7cc42f958dc650907174af0554
> NVIDIA L4; Linux Colab; BF16; SDPA
> Python 3.12.13; PyTorch 2.11.0+cu128; Transformers 5.18.0
> huggingface_hub 1.33.0; NumPy 2.0.2
> router threshold: 0.5155917406082153
> forced generation: greedy, max_new_tokens=384
> 
> ```
> 
> Source hashes were checked, and the 504 LoRA tensors plus decision head were successfully loaded. Those checks qualify this particular run; they do not establish numerical identity with the Mac MLX implementation. The implementation consulted here is the [pinned engine](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/engine.py) and [router](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/router.py).
> 
> ### Denominators and policy construction
> 
> All **26** examples have an observed quick answer and router score; the released threshold flagged **6**. Only **24** forced generations emitted a valid final label within 384 tokens. In those common 24 cases, quick accuracy was **20/24** ; the released threshold flagged **5** , and replacing the five selected quick labels with the forced labels gives **23/24**. The simple confidence comparator ranked those same 24 by ascending quick top probability and selected the five lowest, giving **24/24**. The selections differ only by the `count_distinct_colors` / `sum_blue_values` swap. This is a retrospective matched-_count_ comparison, not matched latency or a separately measured API run.
> 
> The two capped cases are `hyphen_kdash07` (quick correct and not escalated) and `normal_first` (quick wrong and escalated). In the latter, the generated text reasoned toward the right ordering but never emitted a final scorable answer. It cannot be counted as a routed success or failure without a completion/parsing policy. Changing the generation cap would constitute another test.
> 
> ### Interpretation boundaries
> 
> Dev-4B’s quick path uses the question-side adapter and typed decision head; the forced path generates with the base model and a step-by-step prompt. This is a comparison of **answering modes** , not a controlled intervention that changes only “thinking-token count.” The base [Qwen model card](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) calls this variant _non-thinking_, so the generated path is prompted reasoning, not an enabled special `<think>` mode.
> 
> The [API description](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/docs/api.md) specifies `noul`, `choice` and `score`, and separates the quick decision from generated text. This probe kept exact option order and original prompt construction. It tested short, hand-built English fictional documents; it did not use real production records, perform full train/test de-duplication, run multiple seeds, reproduce the 7,100-item benchmark or measure an end-to-end Mac deployment. Reported quick/forced correctness pertains only to these fixtures. No right-to-wrong generated result appeared among the 24 scorable cases; the remaining two are **unknown** , not evidence of correctness or failure.

> **Practical follow-up options and reference links**
>
> The lightest useful next step depends on what the existing test artifacts already contain:
> 
> ```plaintext
> Have saved per-question quick/router/generated traces?
> Yes -> compare disagreements at the same escalation count.
> Then inspect outcome pairs: wrong->right, right->wrong,
> right->right, wrong->wrong, and incomplete/unparseable.
> No -> rerun a few fixed contrast pairs below at one pinned revision.
> Change one factor at a time; avoid broad new training first.
> 
> Need accurate local cost numbers?
> Measure wall-time and token count per escalated item, with warm/cold
> state and shared-prefill conditions stated; equal calls != equal cost.
> 
> Generated output incomplete or lacks a parseable final label?
> Qualify output budgeting / parsing separately from the router.
> Avoid inferring correctness from unfinished reasoning text.
> 
> ```
> 
> If a task repeatedly shows a genuine missed _repairable_ error, the options range from a cheap task-specific validation guard, to threshold analysis on a suitable validation population, to more targeted router training data. Which to use depends on the deployment cost, not on whether one of my individual counterexamples looks striking. A document with insufficient evidence may warrant an explicit `Unknown` candidate or human review rather than more generation. These are alternatives, not homework or a demand to modify the published system.
> 
> For context, [Guo et al., _On Calibration of Modern Neural Networks_](https://proceedings.mlr.press/v70/guo17a.html) helps distinguish confidence calibration from router ranking; [SelectiveNet](https://proceedings.mlr.press/v97/geifman19a.html) provides a risk/coverage framing if a system later adds abstention; and [adaptive test-time compute allocation](https://arxiv.org/abs/2604.14853) is related to cost-aware selection, not a Dev-4B replication. [DROP](https://aclanthology.org/N19-1246/), [MuSiQue](https://aclanthology.org/2022.tacl-1.31/) and [SQuAD 2.0](https://aclanthology.org/P18-2124/) illustrate arithmetic, multi-hop and unanswerable-document tests, respectively. I did **not** benchmark Dev-4B on them.
> 
> Primary project references: [original forum post](https://discuss.huggingface.co/t/dev-4b-quick-calibrated-decisions-on-a-document-reason-only-when-the-router-flags-risk/182924), [Dev-4B card and sources](https://huggingface.co/suhaas-teja/Dev-4B), [demo Space](https://huggingface.co/spaces/suhaas-teja/Dev-4B-demo), [router](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/router.py), [engine](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/engine.py), [prompt template](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/dmm/template.py), and [API docs](https://huggingface.co/suhaas-teja/Dev-4B/blob/e4c9005aeffd50df1fcc2e89e498ffa0cf142dab/docs/api.md).
> 
> **Takeaway:** the released router caught some surprisingly confident quick errors, while a different repairable error stayed below its threshold. One correct answer also received an escalation without a correctness gain. That is the practical feedback from this small test; it does not overturn the project’s broader evaluation or the value of selective generation.