RLCR Gemma 4 E2B-it — Math — seed 44

Gemma 4 E2B-it trained on Big-Math-Digits (mehuldamani/big-math-digits) to verbalize the confidence in its own answer with RLCR (our Gemma training of the RLCR baseline, not a checkpoint released by its authors).

Paper: Cross-Question Reliable Reinforcement Learning — Hector G. Rodriguez and Marcus Rohrbach.

Abstract

AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.

Results of this checkpoint

Selective prediction, all values in %; C@r is coverage at risk r; AURC lower is better. ID: big-math-digits, math-500, gsm8k. OOD: commonsenseqa, gpqa, simpleqa, trivia, hotpotqa. The paper's Gemma table reports the mean of seeds 43 and 44.

ID C@0 ID C@1 ID C@5 ID C@10 ID AURC ↓ ID Acc OOD C@0 OOD C@1 OOD C@5 OOD C@10 OOD AURC ↓ OOD Acc
0.0 0.0 0.0 0.0 17.2 74.2 0.0 0.0 9.8 31.8 37.4 53.8

Use

Use the exact system prompt in system_prompt.txt, the saved chat template and all EOS IDs from generation_config.json. The paper evaluates with one completion, temperature 0 and a 4096-token generation limit. Outputs contain <answer>...</answer> and <confidence>...</confidence>. Rank answers by confidence and choose a threshold on suitable held-out data; a confidence score is not a guarantee of correctness.

These checkpoints require a Gemma 4-capable runtime. They are text-only post-training; the vision and audio encoders were frozen and multimodal use has not been evaluated. The training configuration is in training_config.yaml (per-device batch 4, gradient accumulation 96, maximum completion length 4096).

License and attribution

Base model: Google DeepMind Gemma 4 E2B-it, revision 70af34e20bd4b7a91f0de6b22675850c43922a03, Apache 2.0 (license and modification notice included). Post-training: Cross-Question Reliable Reinforcement Learning, Hector G. Rodriguez and Marcus Rohrbach. RLCR: Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (Damani et al., 2025).

Downloads last month
36
Safetensors
Model size
6B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for hector-gr/RLCR-Gemma-4-E2B-it-Math-seed44

Finetuned
(490)
this model

Dataset used to train hector-gr/RLCR-Gemma-4-E2B-it-Math-seed44

Collection including hector-gr/RLCR-Gemma-4-E2B-it-Math-seed44