Show complete LFM checkpoint progression, coverage, and tool/token diagnostics
Browse files- .gitattributes +2 -0
- LFM_DATA.md +14 -0
- README.md +24 -17
- app.py +4 -1
- lfm_checkpoint_coverage.csv +23 -0
- lfm_final_efficiency.pdf +0 -0
- lfm_final_efficiency.png +3 -0
- lfm_final_overview.pdf +0 -0
- lfm_final_overview.png +3 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
comparison.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
comparison.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
lfm_final_overview.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
lfm_final_efficiency.png filter=lfs diff=lfs merge=lfs -text
|
LFM_DATA.md
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# LFM data verification
|
| 2 |
+
|
| 3 |
+
Published from the final 23 September 2026 snapshot. Exact API readback verified 2,023 additive events on 2026-09-24T06:46:59.240281+00:00.
|
| 4 |
+
|
| 5 |
+
- OpenCode: baseline plus checkpoints 100 through 1,000.
|
| 6 |
+
- Multi-harness: baseline plus checkpoints 100 through 1,000, and 850.
|
| 7 |
+
- Both training series: exactly 1,000 committed optimizer steps.
|
| 8 |
+
- Observed scores were independently recounted from unique successful grading records keyed by harness, task index and repetition.
|
| 9 |
+
- Difficulties were read from the fixed 250-task manifest: 33 easy, 118 medium, 99 hard.
|
| 10 |
+
- Native tool-call counts were verified from ATIF trajectories for every included graded rollout.
|
| 11 |
+
- Token figures use captured prompt/completion IDs across all calls, including auxiliary calls. Means include unsuccessful graded rollouts. Savings use matched successful rollouts.
|
| 12 |
+
- Original logs and incomplete checkpoints were preserved; no missing score was filled or inferred.
|
| 13 |
+
|
| 14 |
+
The snapshot scripts and frozen data are retained with the experiment. The public coverage CSV and figures contain aggregate measurements only.
|
README.md
CHANGED
|
@@ -10,29 +10,36 @@ pinned: false
|
|
| 10 |
|
| 11 |
# Data Agent RL comparison
|
| 12 |
|
| 13 |
-
|
| 14 |
|
| 15 |
-
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|---|---|
|
| 19 |
-
|
|
| 20 |
-
|
|
| 21 |
-
|
|
| 22 |
-
|
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
-
|
| 26 |
|
| 27 |
-
|
| 28 |
|
| 29 |
-
|
| 30 |
-
- **eval/provisional_pass_at_1:** successes divided by currently graded cells. Missing results are excluded; final coverage and auditing remain pending.
|
| 31 |
-
- **eval/graded_cells / eval/missing_cells:** coverage alongside the score.
|
| 32 |
-
- **train/reward_rolling20:** trailing mean over 20 recorded training updates; raw reward is also available. Evaluation smoothing should remain zero.
|
| 33 |
|
| 34 |
-
The
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
-
|
|
|
|
| 10 |
|
| 11 |
# Data Agent RL comparison
|
| 12 |
|
| 13 |
+
Training and evaluation results in one public Trackio project: **data-agent-rl-comparison**.
|
| 14 |
|
| 15 |
+
## LFM2.5-2.6B: OpenCode-only vs multi-harness
|
| 16 |
|
| 17 |
+
Both runs reached 1,000 optimizer steps. Both final evaluations have 1,000/1,000 graded cases: **52.3% OpenCode-only**, **54.2% multi-harness**. The evaluation set is 250 fixed tasks × four harnesses, pass@1.
|
| 18 |
+
|
| 19 |
+
[Overview from step 0](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5E%28eval_observed%2F%28pass_at_1%7Ccombined_reward_normalized%7Ctool_call_savings_pct%29%7Ceval_coverage%2F%28graded_cells%7Cmissing_cells%29%7Ctrain_verified%2Freward_mean50%29%24) · [All LFM metrics](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5E%28eval_observed%7Ceval_efficiency%7Ceval_usage%7Ceval_coverage%7Ctrain_verified%29%2F) · [Harness scores](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_observed%2Fharness%2F) · [Difficulty](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_observed%2F%28difficulty%7Charness_difficulty%29%2F) · [Tool and token savings](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_efficiency%2F) · [Actual token/tool usage](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_usage%2F%28overall%7Charness%2F%5B%5E%2F%5D%2B%29%2F%28generated_tokens_mean%7Cinput_tokens_mean%7Cnative_tool_calls_mean%29%24) · [Training diagnostics](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Etrain_verified%2F) · [Coverage](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_coverage%2F)
|
| 20 |
+
|
| 21 |
+
The overview contains the baseline and every available checkpoint, including incomplete early evaluations. Scores use fractions: 0.542 means 54.2%. Keep dashboard smoothing at **0**; the training `reward_mean50` series is already smoothed over 50 updates.
|
| 22 |
+
|
| 23 |
+
| Metric group | Meaning |
|
| 24 |
|---|---|
|
| 25 |
+
| `eval_observed` | All available graded results, including provisional checkpoints; missing cases are excluded, not counted as failures |
|
| 26 |
+
| `eval_coverage` | Graded, missing and correct counts, completeness, native tool-count and token coverage |
|
| 27 |
+
| `eval_efficiency` | Positive tool/generated-token/input-token savings versus baseline on tasks both models solved; matched counts included |
|
| 28 |
+
| `eval_usage` | Actual average native tool calls, input/generated tokens, turns and duration across graded rollouts |
|
| 29 |
+
| `train_verified` | Committed training lineage: reward, loss, gradients, tokens, calls, staleness and performance diagnostics |
|
| 30 |
+
|
| 31 |
+
The shared pretrained baseline is repeated at step 0 in both training curves for comparison; it is one evaluation, not two independent measurements. Baseline coverage is **998/1,000**. OpenCode checkpoints 100–400 and multi-harness checkpoints 100–200 are also incomplete. See coverage charts or [checkpoint coverage CSV](lfm_checkpoint_coverage.csv). Later checkpoints, including both final results, are complete.
|
| 32 |
+
|
| 33 |
+
Normalized combined reward is `mean(correctness × (1 + tool bonus)) / 1.1`, with bonus `1.5 / (15 + native tool calls)` only when a positive native count is verified. Efficiency cohorts vary across checkpoints and runs; comparisons are descriptive, not causal. Input tokens include repeated history and are not cache-adjusted.
|
| 34 |
|
| 35 |
+
The raw multi-harness logs preserve 17 superseded pre-resume updates (851–867). `train_verified` excludes them and follows the saved checkpoint lineage used in the final figures. Original logs remain available.
|
| 36 |
|
| 37 |
+
[Overview figure](lfm_final_overview.png) · [Tool and token figure](lfm_final_efficiency.png) · [Overview PDF](lfm_final_overview.pdf) · [Tool and token PDF](lfm_final_efficiency.pdf) · [Verification and source notes](LFM_DATA.md)
|
| 38 |
|
| 39 |
+
## Historical runs
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
+
The same project retains Qwen 2B multi-harness, native OpenCode, Harbor OpenCode-only and hard-task continuation runs, plus the LFM pretrained baseline. Clear the run selection to include them. Historical backends, retry policies and reasoning settings differ; compare with their protocol notes.
|
| 42 |
|
| 43 |
+
`eval/pass_at_1` retains completed audited scores; `eval/provisional_pass_at_1` retains partial results. The new `eval_observed` series unifies the available LFM checkpoints into continuous curves while preserving explicit coverage metrics. Original databases and scalar records were retained.
|
| 44 |
|
| 45 |
+
[Historical report](REPORT.md) · [Historical checkpoint CSV](checkpoint_scores.csv) · [Historical provisional scores](PROVISIONAL_EVALS.md)
|
app.py
CHANGED
|
@@ -1,4 +1,7 @@
|
|
| 1 |
import os
|
| 2 |
-
|
|
|
|
|
|
|
| 3 |
import trackio
|
|
|
|
| 4 |
trackio.show(project='data-agent-rl-comparison', open_browser=False)
|
|
|
|
| 1 |
import os
|
| 2 |
+
|
| 3 |
+
os.environ["TRACKIO_PLOT_ORDER"] = 'eval_observed/pass_at_1,eval_observed/combined_reward_normalized,eval_observed/tool_call_savings_pct,train_verified/reward_mean50,eval_coverage/graded_cells,eval_coverage/missing_cells,eval_observed/harness/*,eval_observed/difficulty/*,eval_efficiency/*,eval_usage/overall/*,eval_usage/harness/*,train_verified/reward,train_verified/tools/*,train_verified/completions/*,train_verified/batch/trained_tokens_per_step,train_verified/batch/forwarded_tokens_per_step,eval_observed/harness_difficulty/*,train_verified/*,eval_coverage/*,eval/pass_at_1,eval/provisional_pass_at_1,train/reward_rolling20,eval/harness/*,eval/difficulty/*'
|
| 4 |
+
|
| 5 |
import trackio
|
| 6 |
+
|
| 7 |
trackio.show(project='data-agent-rl-comparison', open_browser=False)
|
lfm_checkpoint_coverage.csv
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
arm,step,graded,correct,pass_at_1,complete,native_count_coverage
|
| 2 |
+
baseline,0,998,421,0.4218436873747495,False,998
|
| 3 |
+
opencode,100,986,473,0.47971602434077076,False,986
|
| 4 |
+
opencode,200,979,501,0.5117466802860061,False,979
|
| 5 |
+
opencode,300,989,496,0.5015166835187057,False,989
|
| 6 |
+
opencode,400,999,506,0.5065065065065065,False,999
|
| 7 |
+
opencode,500,1000,501,0.501,True,1000
|
| 8 |
+
opencode,600,1000,518,0.518,True,1000
|
| 9 |
+
opencode,700,1000,508,0.508,True,1000
|
| 10 |
+
opencode,800,1000,512,0.512,True,1000
|
| 11 |
+
opencode,900,1000,524,0.524,True,1000
|
| 12 |
+
opencode,1000,1000,523,0.523,True,1000
|
| 13 |
+
multi-harness,100,988,465,0.47064777327935226,False,988
|
| 14 |
+
multi-harness,200,988,520,0.5263157894736842,False,988
|
| 15 |
+
multi-harness,300,1000,507,0.507,True,1000
|
| 16 |
+
multi-harness,400,1000,510,0.51,True,1000
|
| 17 |
+
multi-harness,500,1000,521,0.521,True,1000
|
| 18 |
+
multi-harness,600,1000,507,0.507,True,1000
|
| 19 |
+
multi-harness,700,1000,546,0.546,True,1000
|
| 20 |
+
multi-harness,800,1000,540,0.54,True,1000
|
| 21 |
+
multi-harness,850,1000,545,0.545,True,1000
|
| 22 |
+
multi-harness,900,1000,520,0.52,True,1000
|
| 23 |
+
multi-harness,1000,1000,542,0.542,True,1000
|
lfm_final_efficiency.pdf
ADDED
|
Binary file (82.7 kB). View file
|
|
|
lfm_final_efficiency.png
ADDED
|
Git LFS Details
|
lfm_final_overview.pdf
ADDED
|
Binary file (74.3 kB). View file
|
|
|
lfm_final_overview.png
ADDED
|
Git LFS Details
|