Dev-4B: quick calibrated decisions on a document, reason only when the router flags risk

Show and Tell
1

I released Dev-4B — a small local model that answers typed questions about a document (choice / yes-no / score) with calibrated confidence, and only thinks step by step when a built-in router predicts the quick answer is likely wrong.

Space: Dev-4B - a Hugging Face Space by suhaas-teja
Model: suhaas-teja/Dev-4B · Hugging Face

MLX 8-bit build is on the same profile as Dev-4B-MLX-8bit (search that name on Hugging Face).

Stack: Qwen3-4B-Instruct-2507 + ~133 MB of add-ons (LoRA adapter switched off while reading the document and on for the question, decision head, temperatures for calibration, tiny router). Base generation is unchanged when add-ons are off.

Results on 7,100 frozen test questions:

  • Always quick: 76.3% accuracy, 0.16 s/q
  • Always reason: 78.0%, 10.2 s/q
  • Dev-4B (route): 81.8%, 1.37 s/q
    Arithmetic/counting is where escalation helps most. On a 16 GB Mac (MLX 8-bit), five questions ≈ 3 s, ~6 GB peak.

Limits to be clear about: weaker on unseen task types than on trained ones; reasoning is still a 4B model; English; docs ~4k chars in training; NC weights (CC BY-NC-SA); no Ollama/LM Studio (LoRA + decision head + router). Question format follows System One; not affiliated with TypeSafe.

Most open System One clones are decision-only. Dev-4B’s wedge is local calibrated decisions plus a router that escalates to CoT when the quick pass looks wrong — for Mac indie / offline routing experimenters.

Would love feedback from people running local decision models — especially Mac numbers and cases where the router should/shouldn’t escalate.

2

I gave it a quick try:

Scope of the test

I ran the released Dev-4B model and code on a Linux/NVIDIA L4 using BF16, with 26 short fictional document questions in 13 paired groups. For each question I recorded the quick typed answer and its top probability, the released router’s score and escalation choice, and the result of a separately forced step-by-step generation. I designed these examples to look at routing behavior, not to repeat the refund/stamp demos.

This was not a Mac/MLX test, an independent reproduction of the published 7,100-question evaluation, or a representative real-world benchmark. I can offer concrete should/shouldn’t-escalate candidates; I cannot offer Mac performance numbers.

Observation In this probe
Quick correct 21/26
Router escalations, published threshold ~0.516 6/26
Forced outputs with parseable final label 24/26
Forced correct among the 24 parseable outputs 24/24

Two forced generations reached the 384-token ceiling before emitting a parseable final Answer: line. I kept those as unscored, rather than assuming they were right or wrong. Source and base revisions were pinned; the model/LoRA/head load checks passed.

Cases directly relevant to the request

The most useful contrast is between these two different routing decisions:

Question Gold Quick answer (top probability) Router score Escalated? Forced final
Sum of blue rows (7 + 9 + 5) 21 19 (0.699), wrong 0.328 No 21, correct
Number of distinct colors (amber/cyan/teal/violet) 4 4 (0.960), correct 0.752 Yes 4, correct

For the blue-row sum, the quick decision was wrong but the forced reasoning pass repaired it. The router left the case on the quick path. This is a concrete missed repair opportunity on this input.

For the distinct-color count, the quick decision was already correct and the router still escalated. The forced result did not improve correctness. This is an escalation with no observed accuracy gain in hindsight; it is not proof that the router should have known in advance that reasoning was unnecessary.

The other direction matters just as much. The router also caught two fairly confident quick errors:

Question Gold Quick answer (top probability) Router score Forced final
Default Dock A; X9 alone is an exception to Dock B; carton C13 has Y2 Dock A Dock B (0.874) 0.540, escalated Dock A
Review only for a total strictly greater than 18; the three loads total 18 No Yes (0.804) 0.779, escalated No

There was also a successful repair of the non-blue row sum: quick 21 versus gold 27; router 0.598 (escalated); forced answer 27. A neighboring case with total load 19 was quick-correct at Yes but also escalated, with no observed change after reasoning.

These examples are why I would not interpret the blue-row miss as an argument against the learned router. In this same small test, it detected high-confidence errors that a naive low-confidence-only cutoff might fail to select. The fixed inputs and full results are expanded below.

A small comparison at the same escalation count

On the 24 items for which a final forced answer was actually parseable, I reconstructed each policy’s selected final label from the saved quick and forced outputs:

Selection rule Correct / 24 Additional generations selected
Always quick 20/24 0
Published router threshold 23/24 5
Five lowest quick top probabilities 24/24 5
Always use forced generation 24/24 24

The two five-generation selections differ in only one item each. Both selected except_y2, load_at_18, load_at_19 and sum_non_blue_values. The router additionally selected correct count_distinct_colors; confidence ranking instead selected wrong-but-repairable sum_blue_values.

This is a useful diagnostic disagreement, not a claim that confidence-only routing beats Dev-4B in general. The comparator was selected after seeing how many items the released router escalated; the 26 questions are hand-built, some share documents, and equal numbers of generations do not imply equal token cost or wall time. The 23/24 and 24/24 values are conditional on the common scorable 24, not estimates of performance on the author’s benchmark. I did not rerun the /v1/auto HTTP endpoint for final-policy throughput; those policy outputs were reconstructed from the paired traces.

A possible way to interpret the contrast

The two questions “Is the quick answer likely wrong?” and “Will another generation pass actually improve the final answer?” are related, but not identical. The released router learns an error-risk signal from the decision-token representation and three confidence features. That is well aligned with detecting suspicious decisions. When the action is specifically extra generation, the chance of successful repair and the cost of that attempt may also matter.

I would treat this as an analysis axis, not a request to retrain anything. The existing architecture of fast typed decisions plus selective generation can stay unchanged. Also, router_score is not automatically a calibrated probability of an error: e.g., 0.752 is the routing score here, not independently validated “75.2% wrong.” Temperature-scaled option probabilities and the router’s scoring function have different roles.

Low-cost next checks, depending on the goal

If per-item evaluation traces already exist: compare quick label, gold, router score, generated final label, and generated tokens for the same items. Separate wrong→right, right→wrong, right→right, wrong→wrong, and unscored/truncated. At a fixed number of escalations, inspect cases where router ranking and simple confidence ranking disagree. Only if cost is a central deployment concern would I move on to matching actual generated tokens or wall time.

If item-level traces are not available: the fixed blue/non-blue sum, X9/Y2 exception, and 18/19 strict-threshold pairs below are low-cost starting controls. Vary one dimension at a time: target row/predicate, candidate order, threshold boundary, or irrelevant document content. Exact prompt and option order matter to the quick classification head.

If the goal is Mac deployment: this L4 run cannot substitute for a test of the released MLX 8-bit model. Useful measurements would identify Mac hardware, model/MLX version, document length, number of questions sharing the prefill, warm/cold state, generated tokens and definition of memory.

If the goal is reliably typed output: distinguish generated reasoning from a completed and parsed final choice. The two capped outputs here show why this matters to evaluation. And if a document does not contain the required fact, more reasoning is not automatically a substitute for an explicit unknown/review path.

Below are exact example inputs, the 26-case outcome list, paired fixture descriptions, and the pinned run conditions. They are meant as reproducibility aids, not a grading of the project or a new leaderboard.

Exact case inputs for the four most informative routing behaviors

1. A repairable quick error that was not escalated: sum_blue_values

Document:

Audit lines: row A = blue, 7 points; row B = yellow, 12 points; row C = blue, 9 points; row D = red, 15 points; row E = blue, 5 points. A row contributes its points only when the question selects its color.

Question and options (in actual order):

{"type":"choice","instructions":"Total points on blue rows?","criteria":{"19":null,"21":null,"33":null,"48":null}}

The three blue rows are A, C, and E: 7 + 9 + 5 = 21. The quick decision was 19 (p_top = 0.69911873). Router score was 0.32785931, below 0.51559174, so no escalation. Forced generation produced the final label 21 in 197 generated tokens. This establishes a repairable missed quick error for this particular input and execution path. It does not establish the prevalence of that failure, nor does it prove why the router scored it low.

A useful feature of this example is that the available answers include plausible distractors. In particular, 19 could arise from a mistaken selection or arithmetic path, but I cannot infer the exact hidden computation performed by the quick head. I would not describe the cause as “it forgot the last blue row” unless an additional probe isolates that interpretation. The reliable observation is narrower: wrong selected label; router below threshold; deterministic generated label correct.

2. Escalation with no observed correctness gain: count_distinct_colors

Document:

Shelf labels in order: amber, cyan, amber, teal, cyan, cyan, violet, teal. Each occurrence counts as one label.

Question and options:

{"type":"choice","instructions":"How many different label colors appear?","criteria":{"3":null,"4":null,"5":null,"8":null}}

Distinct labels: amber, cyan, teal, violet, so the answer is 4. Quick label 4 (p_top = 0.96007693); router score 0.75211662 (above the threshold); forced final label 4 in 181 generated tokens. The router deliberately chose a longer path even though the quick answer happened to be correct. This is a reasonable cost/benefit inspection candidate, not enough to label the router miscalibrated or broken.

The wording matters: the document includes repeated occurrences while the question asks for distinct types. Replacing the question with “How many cyan labels?” would test a different counting operation. In fact the paired occurrence-count case is quick-correct and not escalated in this run. That pair might help distinguish task-specific representation effects from a simple low-confidence rule, although one pair is far too little to establish the mechanism.

3. A high-confidence quick error detected by the router: except_y2

Document:

Default storage plan: all cartons go to Dock A. Explicit exception: cartons with code X9 go to Dock B. List: C12 code X9, C13 code Y2. The specific exception takes precedence over default.

Question and options:

{"type":"choice","instructions":"Where should carton C13 go?","criteria":{"Dock A":null,"Dock B":null,"Undecided":null}}

Carton C13 has Y2, not X9, so the exception does not apply; Dock A is the gold answer. The quick head selected Dock B (p_top = 0.87404186). The router scored 0.54035252, just above its published threshold, and the generated answer was Dock A after 211 tokens. This is a useful positive control for the router’s nontrivial signal, since a threshold based only on very low quick confidence could easily miss a confident wrong label. It does not prove which hidden feature caused the router’s higher score.

The paired except_x9 item instead asks about C12. That input legitimately activates the exception and was quick-correct at Dock B, with a low router score. Together they give an interpretable change in the correct conditional rule while keeping the fictional storage policy fixed.

4. A strict numeric boundary caught by the router: load_at_18

Document:

A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 5 units.

Question:

{"type":"noul","instructions":"Does this shipment require extra review?"}

Here 6+7+5=18, and 18 > 18 is false. Gold No. Quick was Yes (p_top = 0.80355495), router score 0.77924889, and forced generation concluded No after 158 tokens. The small router flagged a fairly confident error, and the reasoning pass fixed it.

The paired input changes the last crate from 5 to 6; the total becomes 19 and the gold answer becomes Yes. On that paired input, quick Yes was already correct, but the router still escalated (0.83207101), and generation remained Yes. The pair is useful for comparing the direction of a strict inequality at the boundary with the routing cost spent on either side; it does not establish a systematic over-triggering pattern from two samples.

Complete 26-case outcome inventory (observed, not population estimates)

Every row below was read from the returned result ZIP for this probe. top-p is the largest quick label probability, not a measure of calibration on this tiny panel. risk is the router score. Y/N indicates whether the published threshold triggered an escalation. — in the forced column means that no final answer was parseable before the configured cap; it does not count as an incorrect answer. Each selected label is displayed as a human-readable typed label, not its internal choice letter.

ID Gold Quick top-p risk Escalate Forced Generated tokens
rev_target_p82 Amber Pier Amber Pier 0.977 0.027 N Amber Pier 286
rev_neighbor_p83 Willow Gate Willow Gate 0.973 0.030 N Willow Gate 258
hyphen_kdash07 Room East Room East 0.977 0.004 N — (capped) 384
hyphen_k07 Room West Room West 0.981 0.037 N Room West 219
count_cyan_occurrences 3 3 0.971 0.474 N 3 169
count_distinct_colors 4 4 0.960 0.752 Y 4 181
sum_blue_values 21 19 0.699 0.328 N 21 197
sum_non_blue_values 27 21 0.470 0.598 Y 27 250
and_r2 no no 0.978 0.002 N no 132
and_r3 yes yes 0.969 0.006 N yes 161
except_x9 Dock B Dock B 0.990 0.016 N Dock B 203
except_y2 Dock A Dock B 0.874 0.540 Y Dock A 211
missing_end_time Not recorded Not recorded 0.991 0.051 N Not recorded 134
stated_start 14:00 14:00 0.987 0.004 N 14:00 227
join_manifest31 Hall Copper Hall Copper 0.989 0.034 N Hall Copper 123
join_manifest32 Hall Silver Hall Silver 0.983 0.007 N Hall Silver 159
load_at_18 no yes 0.804 0.779 Y no 158
load_at_19 yes yes 0.764 0.832 Y yes 136
permute_ab Sol Sol 0.992 0.029 N Sol 223
permute_ba Sol Sol 0.985 0.026 N Sol 169
not_inspected_b Not inspected Not inspected 0.992 0.013 N Not inspected 220
unknown_inspected_c Unrecorded Unrecorded 0.988 0.070 N Unrecorded 275
supervisor_of_vale Ibis Ibis 0.986 0.014 N Ibis 187
supervisee_of_ibis Vale Vale 0.984 0.007 N Vale 204
urgent_first Q1 Q1 0.986 0.077 N Q1 289
normal_first Q4 Q1 0.455 0.705 Y — (capped) 384

These paired rows are a transparent outcome log, not 26 independent deployment samples.

The four quick/forced outcome types (wrong→right, right→wrong, right→right, wrong→wrong) should be kept separate from unscored generated answers.

For this actual run, the four parsed quick-wrong/forced-correct cases were sum_blue_values, sum_non_blue_values, except_y2, and load_at_18. Three were escalated; sum_blue_values was not. No right-to-wrong outcome was observed among the 24 parsed forced generations, but that absence should not be generalized to an unseen dataset.

All 26 input fixtures, grouped by the 13 controlled contrasts

These are the exact fictional document strings, question strings, choice order, and gold labels used in the L4 probe, rearranged by pair to save space. The fixed state for a pair is shown once unless the two states differ. The gold value is the test author’s annotation, not a field accepted by Dev-4B’s API. noul is the yes/no question type in the published API. Options below retain their insertion order. This section plus the outcome inventory makes the reported selection disagreement inspectable; it is not a claim that these fixtures are publicly hosted as a benchmark dataset.

Pair revision — 2 questions

Facility memo 8:10: send parcel P-82 to Cedar Bay. At 09:15 the dispatcher changes P-82 to Amber Pier. At 10:20 the dispatcher changes parcel P-83 (not P-82) to Willow Gate. Latest valid instruction for each parcel supersedes its earlier instruction.

  • rev_target_p82 (choice): Final destination for P-82? Choices, in order: Cedar Bay; Amber Pier; Willow Gate; Not recorded. Gold: Amber Pier.
  • rev_neighbor_p83 (choice): Final destination for P-83? Choices, in order: Cedar Bay; Amber Pier; Willow Gate; Not recorded. Gold: Willow Gate.

Pair identifier — 2 questions

Registry: sample K-07 belongs to crate D4. Sample K07 belongs to crate A9. Crate D4 is stored in Room East; crate A9 is stored in Room West. The dash is significant in sample identifiers.

  • hyphen_kdash07 (choice): Which room holds sample K-07? Choices, in order: Room East; Room West; Not recorded. Gold: Room East.
  • hyphen_k07 (choice): Which room holds sample K07? Choices, in order: Room East; Room West; Not recorded. Gold: Room West.

Pair count_scope — 2 questions

Shelf labels in order: amber, cyan, amber, teal, cyan, cyan, violet, teal. Each occurrence counts as one label.

  • count_cyan_occurrences (choice): How many entries are labelled cyan? Choices, in order: 2; 3; 4; 5. Gold: 3.
  • count_distinct_colors (choice): How many different label colors appear? Choices, in order: 3; 4; 5; 8. Gold: 4.

Pair filtered_sum — 2 questions

Audit lines: row A = blue, 7 points; row B = yellow, 12 points; row C = blue, 9 points; row D = red, 15 points; row E = blue, 5 points. A row contributes its points only when the question selects its color.

  • sum_blue_values (choice): Total points on blue rows? Choices, in order: 19; 21; 33; 48. Gold: 21.
  • sum_non_blue_values (choice): Total points on non-blue rows? Choices, in order: 21; 27; 36; 48. Gold: 27.

Pair logic_and — 2 questions

A package is ready to release only if both tests pass: seal intact AND label verified. Package R2: seal intact, label not verified. Package R3: seal intact, label verified. No other overrides exist.

  • and_r2 (noul): Is package R2 ready to release? Choices, in order: No; Yes (implicit). Gold: no.
  • and_r3 (noul): Is package R3 ready to release? Choices, in order: No; Yes (implicit). Gold: yes.

Pair policy_priority — 2 questions

Default storage plan: all cartons go to Dock A. Explicit exception: cartons with code X9 go to Dock B. List: C12 code X9, C13 code Y2. The specific exception takes precedence over default.

  • except_x9 (choice): Where should carton C12 go? Choices, in order: Dock A; Dock B; Undecided. Gold: Dock B.
  • except_y2 (choice): Where should carton C13 go? Choices, in order: Dock A; Dock B; Undecided. Gold: Dock A.

Pair missingness — 2 questions

Meeting card for Session M4: conference room Atlas, moderator Wren, begins at 14:00. No duration or end time is listed on this card.

  • missing_end_time (choice): When does Session M4 end? Choices, in order: 14:30; 15:00; 16:00; Not recorded. Gold: Not recorded.
  • stated_start (choice): When does Session M4 begin? Choices, in order: 13:30; 14:00; 15:00; Not recorded. Gold: 14:00.

Pair three_hop — 2 questions

Index: manifest N31 lists coil T8; manifest N32 lists coil T9. Coil T8 was assigned to rack E6; coil T9 was assigned to rack E8. Rack E6 is in Hall Copper; rack E8 is in Hall Silver. Always follow the manifest-to-coil-to-rack link.

  • join_manifest31 (choice): Which hall contains the coil on manifest N31? Choices, in order: Hall Silver; Hall Copper; Not recorded. Gold: Hall Copper.
  • join_manifest32 (choice): Which hall contains the coil on manifest N32? Choices, in order: Hall Silver; Hall Copper; Not recorded. Gold: Hall Silver.

Pair strict_gt — 2 questions

  • State for load_at_18: A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 5 units.
  • load_at_18 (noul): Does this shipment require extra review? Choices, in order: No; Yes (implicit). Gold: no.
  • State for load_at_19: A device must get an extra review when the combined load is strictly greater than 18 units. One crate contains 6 units, one contains 7 units, and the third contains 6 units.
  • load_at_19 (noul): Does this shipment require extra review? Choices, in order: No; Yes (implicit). Gold: yes.

Pair option_position — 2 questions

Responsibility sheet: feature Cedar belongs to owner Uma; feature Willow belongs to owner Sol. Work order WO-19 concerns feature Willow. Do not use the creator of the work order to assign an owner: the creator is Uma.

  • permute_ab (choice): Who owns the feature in WO-19? Choices, in order: Uma; Sol; Not recorded. Gold: Sol.
  • permute_ba (choice): Who owns the feature in WO-19? Choices, in order: Not recorded; Sol; Uma. Gold: Sol.

Pair missing_vs_no — 2 questions

Inventory sheet: panel A has an inspection stamp; panel B is marked “not inspected”. Panel C has a blank inspection field. Blank means unrecorded, not an explicit negative.

  • not_inspected_b (choice): What is the inspection status of panel B? Choices, in order: Inspected; Not inspected; Unrecorded. Gold: Not inspected.
  • unknown_inspected_c (choice): What is the inspection status of panel C? Choices, in order: Inspected; Not inspected; Unrecorded. Gold: Unrecorded.

Pair direction — 2 questions

Workshop handoff record: technician Nova supervises technician Ibis. Technician Ibis supervises technician Vale. Nobody else supervises anyone in this record. The relation “supervises” is directional.

  • supervisor_of_vale (choice): Who directly supervises Vale? Choices, in order: Nova; Ibis; Vale; Not recorded. Gold: Ibis.
  • supervisee_of_ibis (choice): Who is directly supervised by Ibis? Choices, in order: Nova; Ibis; Vale; Not recorded. Gold: Vale.

Pair queue_sort — 2 questions

Queue order after scanning: Q4, Q1, Q3, Q2. Only Q1 and Q3 are tagged urgent. Process urgent items in their order of appearance, then process the rest in their order of appearance.

  • urgent_first (choice): Which item is processed first? Choices, in order: Q4; Q1; Q3; Q2. Gold: Q1.
  • normal_first (choice): Which item is processed third overall? Choices, in order: Q4; Q1; Q3; Q2. Gold: Q4.

The paired design changes a target, predicate, option order, or boundary value while retaining as much context as possible. It should not be treated as 26 statistically independent samples. The exact prompt construction uses the Dev-4B template; a reimplementation that changes the template or candidate order would be testing a different input.

How the comparison was calculated and what it does not establish

Pinned environment and result integrity

Dev-4B revision: e4c9005aeffd50df1fcc2e89e498ffa0cf142dab
Qwen3-4B-Instruct-2507 revision: cdbee75f17c01a7cc42f958dc650907174af0554
NVIDIA L4; Linux Colab; BF16; SDPA
Python 3.12.13; PyTorch 2.11.0+cu128; Transformers 5.18.0
huggingface_hub 1.33.0; NumPy 2.0.2
router threshold: 0.5155917406082153
forced generation: greedy, max_new_tokens=384

Source hashes were checked, and the 504 LoRA tensors plus decision head were successfully loaded. Those checks qualify this particular run; they do not establish numerical identity with the Mac MLX implementation. The implementation consulted here is the pinned engine and router.

Denominators and policy construction

All 26 examples have an observed quick answer and router score; the released threshold flagged 6. Only 24 forced generations emitted a valid final label within 384 tokens. In those common 24 cases, quick accuracy was 20/24; the released threshold flagged 5, and replacing the five selected quick labels with the forced labels gives 23/24. The simple confidence comparator ranked those same 24 by ascending quick top probability and selected the five lowest, giving 24/24. The selections differ only by the count_distinct_colors / sum_blue_values swap. This is a retrospective matched-count comparison, not matched latency or a separately measured API run.

The two capped cases are hyphen_kdash07 (quick correct and not escalated) and normal_first (quick wrong and escalated). In the latter, the generated text reasoned toward the right ordering but never emitted a final scorable answer. It cannot be counted as a routed success or failure without a completion/parsing policy. Changing the generation cap would constitute another test.

Interpretation boundaries

Dev-4B’s quick path uses the question-side adapter and typed decision head; the forced path generates with the base model and a step-by-step prompt. This is a comparison of answering modes, not a controlled intervention that changes only “thinking-token count.” The base Qwen model card calls this variant non-thinking, so the generated path is prompted reasoning, not an enabled special <think> mode.

The API description specifies noul, choice and score, and separates the quick decision from generated text. This probe kept exact option order and original prompt construction. It tested short, hand-built English fictional documents; it did not use real production records, perform full train/test de-duplication, run multiple seeds, reproduce the 7,100-item benchmark or measure an end-to-end Mac deployment. Reported quick/forced correctness pertains only to these fixtures. No right-to-wrong generated result appeared among the 24 scorable cases; the remaining two are unknown, not evidence of correctness or failure.

Practical follow-up options and reference links

The lightest useful next step depends on what the existing test artifacts already contain:

Have saved per-question quick/router/generated traces?
  Yes -> compare disagreements at the same escalation count.
         Then inspect outcome pairs: wrong->right, right->wrong,
         right->right, wrong->wrong, and incomplete/unparseable.
  No  -> rerun a few fixed contrast pairs below at one pinned revision.
         Change one factor at a time; avoid broad new training first.

Need accurate local cost numbers?
  Measure wall-time and token count per escalated item, with warm/cold
  state and shared-prefill conditions stated; equal calls != equal cost.

Generated output incomplete or lacks a parseable final label?
  Qualify output budgeting / parsing separately from the router.
  Avoid inferring correctness from unfinished reasoning text.

If a task repeatedly shows a genuine missed repairable error, the options range from a cheap task-specific validation guard, to threshold analysis on a suitable validation population, to more targeted router training data. Which to use depends on the deployment cost, not on whether one of my individual counterexamples looks striking. A document with insufficient evidence may warrant an explicit Unknown candidate or human review rather than more generation. These are alternatives, not homework or a demand to modify the published system.

For context, Guo et al., On Calibration of Modern Neural Networks helps distinguish confidence calibration from router ranking; SelectiveNet provides a risk/coverage framing if a system later adds abstention; and adaptive test-time compute allocation is related to cost-aware selection, not a Dev-4B replication. DROP, MuSiQue and SQuAD 2.0 illustrate arithmetic, multi-hop and unanswerable-document tests, respectively. I did not benchmark Dev-4B on them.

Primary project references: original forum post, Dev-4B card and sources, demo Space, router, engine, prompt template, and API docs.

Takeaway: the released router caught some surprisingly confident quick errors, while a different repairable error stayed below its threshold. One correct answer also received an escalation without a correctness gain. That is the practical feedback from this small test; it does not overturn the project’s broader evaluation or the value of selective generation.