AdithyaSK HF Staff commited on
Commit
205de3e
·
verified ·
1 Parent(s): 96cc0ae

Show complete LFM checkpoint progression, coverage, and tool/token diagnostics

Browse files
.gitattributes CHANGED
@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  comparison.png filter=lfs diff=lfs merge=lfs -text
 
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  comparison.png filter=lfs diff=lfs merge=lfs -text
37
+ lfm_final_overview.png filter=lfs diff=lfs merge=lfs -text
38
+ lfm_final_efficiency.png filter=lfs diff=lfs merge=lfs -text
LFM_DATA.md ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # LFM data verification
2
+
3
+ Published from the final 23 September 2026 snapshot. Exact API readback verified 2,023 additive events on 2026-09-24T06:46:59.240281+00:00.
4
+
5
+ - OpenCode: baseline plus checkpoints 100 through 1,000.
6
+ - Multi-harness: baseline plus checkpoints 100 through 1,000, and 850.
7
+ - Both training series: exactly 1,000 committed optimizer steps.
8
+ - Observed scores were independently recounted from unique successful grading records keyed by harness, task index and repetition.
9
+ - Difficulties were read from the fixed 250-task manifest: 33 easy, 118 medium, 99 hard.
10
+ - Native tool-call counts were verified from ATIF trajectories for every included graded rollout.
11
+ - Token figures use captured prompt/completion IDs across all calls, including auxiliary calls. Means include unsuccessful graded rollouts. Savings use matched successful rollouts.
12
+ - Original logs and incomplete checkpoints were preserved; no missing score was filled or inferred.
13
+
14
+ The snapshot scripts and frozen data are retained with the experiment. The public coverage CSV and figures contain aggregate measurements only.
README.md CHANGED
@@ -10,29 +10,36 @@ pinned: false
10
 
11
  # Data Agent RL comparison
12
 
13
- Five main runs in one project: **data-agent-rl-comparison**. Scores and rewards use fractions (0.37 = 37%). The x-axis is optimizer step; pretrained baselines are at step 0.
14
 
15
- [Open dashboard](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison) · [Checkpoint evaluations](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&smoothing=0&metric_filter=%5Eeval%2F%28pass_at_1%7Cprovisional_pass_at_1%29%24) · [Training reward](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&smoothing=0&metric_filter=%5Etrain%2Freward%28_rolling20%29%3F%24) · [Harness comparison](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&smoothing=0&metric_filter=%5Eeval%2F%28harness%7Cprovisional_harness%29%2F) · [Difficulty](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&smoothing=0&metric_filter=%5Eeval%2Fdifficulty%2F) · [Coverage](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&smoothing=0&metric_filter=%5Eeval%2F%28graded_cells%7Cmissing_cells%29%24)
16
 
17
- | Run | What it represents |
 
 
 
 
 
 
18
  |---|---|
19
- | Qwen 2B · Multi-harness | Original four-harness training; completed 1,000 steps |
20
- | Qwen 2B · Native OpenCode | Native OpenCode training; evaluated across four harnesses |
21
- | Qwen 2B · Harbor OpenCode | Harbor training with OpenCode only; evaluated across four harnesses |
22
- | Qwen 2B · Hard-task continuation | Resumes multi-harness checkpoint 500 on hard tasks; training and checkpoint evals together |
23
- | LFM 2.6B · Pretrained baseline | LFM2.5-2.6B, four-harness baseline only; no production training yet |
 
 
 
 
24
 
25
- Models: Qwen3.5-2B and LFM2.5-2.6B. The current eval set has 250 fixed tasks × four harnesses, pass@1. Historical backend, retry policy and reasoning settings differ; this is an observational comparison.
26
 
27
- ## Read the evaluation charts
28
 
29
- - **eval/pass_at_1:** completed, audited checkpoint scores.
30
- - **eval/provisional_pass_at_1:** successes divided by currently graded cells. Missing results are excluded; final coverage and auditing remain pending.
31
- - **eval/graded_cells / eval/missing_cells:** coverage alongside the score.
32
- - **train/reward_rolling20:** trailing mean over 20 recorded training updates; raw reward is also available. Evaluation smoothing should remain zero.
33
 
34
- The hard-task continuation is a separate branch from checkpoint 500. Its later points do not replace the original multi-harness run. LFM is 998/1,000 graded; its reasoning is enabled, while the Qwen 2B baseline used thinking disabled. Historical 4B measurements use another cohort and are excluded from this matched-cohort project.
35
 
36
- [Latest provisional scores](PROVISIONAL_EVALS.md) · [Original completed-run report](REPORT.md) · [Original checkpoint CSV](checkpoint_scores.csv)
37
 
38
- Smoke projects and original project databases are retained as archives, outside this main project. No historical source data was deleted. Metrics were copied with provenance and verified by remote readback.
 
10
 
11
  # Data Agent RL comparison
12
 
13
+ Training and evaluation results in one public Trackio project: **data-agent-rl-comparison**.
14
 
15
+ ## LFM2.5-2.6B: OpenCode-only vs multi-harness
16
 
17
+ Both runs reached 1,000 optimizer steps. Both final evaluations have 1,000/1,000 graded cases: **52.3% OpenCode-only**, **54.2% multi-harness**. The evaluation set is 250 fixed tasks × four harnesses, pass@1.
18
+
19
+ [Overview from step 0](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5E%28eval_observed%2F%28pass_at_1%7Ccombined_reward_normalized%7Ctool_call_savings_pct%29%7Ceval_coverage%2F%28graded_cells%7Cmissing_cells%29%7Ctrain_verified%2Freward_mean50%29%24) · [All LFM metrics](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5E%28eval_observed%7Ceval_efficiency%7Ceval_usage%7Ceval_coverage%7Ctrain_verified%29%2F) · [Harness scores](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_observed%2Fharness%2F) · [Difficulty](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_observed%2F%28difficulty%7Charness_difficulty%29%2F) · [Tool and token savings](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_efficiency%2F) · [Actual token/tool usage](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_usage%2F%28overall%7Charness%2F%5B%5E%2F%5D%2B%29%2F%28generated_tokens_mean%7Cinput_tokens_mean%7Cnative_tool_calls_mean%29%24) · [Training diagnostics](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Etrain_verified%2F) · [Coverage](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=2710d85285055cff752a1ab35e0ac343%2C7a4d3a85b9b0718a035f6242627ebfc7&smoothing=0&metric_filter=%5Eeval_coverage%2F)
20
+
21
+ The overview contains the baseline and every available checkpoint, including incomplete early evaluations. Scores use fractions: 0.542 means 54.2%. Keep dashboard smoothing at **0**; the training `reward_mean50` series is already smoothed over 50 updates.
22
+
23
+ | Metric group | Meaning |
24
  |---|---|
25
+ | `eval_observed` | All available graded results, including provisional checkpoints; missing cases are excluded, not counted as failures |
26
+ | `eval_coverage` | Graded, missing and correct counts, completeness, native tool-count and token coverage |
27
+ | `eval_efficiency` | Positive tool/generated-token/input-token savings versus baseline on tasks both models solved; matched counts included |
28
+ | `eval_usage` | Actual average native tool calls, input/generated tokens, turns and duration across graded rollouts |
29
+ | `train_verified` | Committed training lineage: reward, loss, gradients, tokens, calls, staleness and performance diagnostics |
30
+
31
+ The shared pretrained baseline is repeated at step 0 in both training curves for comparison; it is one evaluation, not two independent measurements. Baseline coverage is **998/1,000**. OpenCode checkpoints 100–400 and multi-harness checkpoints 100–200 are also incomplete. See coverage charts or [checkpoint coverage CSV](lfm_checkpoint_coverage.csv). Later checkpoints, including both final results, are complete.
32
+
33
+ Normalized combined reward is `mean(correctness × (1 + tool bonus)) / 1.1`, with bonus `1.5 / (15 + native tool calls)` only when a positive native count is verified. Efficiency cohorts vary across checkpoints and runs; comparisons are descriptive, not causal. Input tokens include repeated history and are not cache-adjusted.
34
 
35
+ The raw multi-harness logs preserve 17 superseded pre-resume updates (851–867). `train_verified` excludes them and follows the saved checkpoint lineage used in the final figures. Original logs remain available.
36
 
37
+ [Overview figure](lfm_final_overview.png) · [Tool and token figure](lfm_final_efficiency.png) · [Overview PDF](lfm_final_overview.pdf) · [Tool and token PDF](lfm_final_efficiency.pdf) · [Verification and source notes](LFM_DATA.md)
38
 
39
+ ## Historical runs
 
 
 
40
 
41
+ The same project retains Qwen 2B multi-harness, native OpenCode, Harbor OpenCode-only and hard-task continuation runs, plus the LFM pretrained baseline. Clear the run selection to include them. Historical backends, retry policies and reasoning settings differ; compare with their protocol notes.
42
 
43
+ `eval/pass_at_1` retains completed audited scores; `eval/provisional_pass_at_1` retains partial results. The new `eval_observed` series unifies the available LFM checkpoints into continuous curves while preserving explicit coverage metrics. Original databases and scalar records were retained.
44
 
45
+ [Historical report](REPORT.md) · [Historical checkpoint CSV](checkpoint_scores.csv) · [Historical provisional scores](PROVISIONAL_EVALS.md)
app.py CHANGED
@@ -1,4 +1,7 @@
1
  import os
2
- os.environ["TRACKIO_PLOT_ORDER"] = 'eval/pass_at_1,eval/provisional_pass_at_1,train/reward_rolling20,train/reward,eval/graded_cells,eval/missing_cells,eval/difficulty/*,eval/harness/*,eval/provisional_harness/*,eval/harness_difficulty/*,train/grad_norm,train/nonzero_gradient_rolling20,train/loss,train/entropy,train/kl,train/learning_rate'
 
 
3
  import trackio
 
4
  trackio.show(project='data-agent-rl-comparison', open_browser=False)
 
1
  import os
2
+
3
+ os.environ["TRACKIO_PLOT_ORDER"] = 'eval_observed/pass_at_1,eval_observed/combined_reward_normalized,eval_observed/tool_call_savings_pct,train_verified/reward_mean50,eval_coverage/graded_cells,eval_coverage/missing_cells,eval_observed/harness/*,eval_observed/difficulty/*,eval_efficiency/*,eval_usage/overall/*,eval_usage/harness/*,train_verified/reward,train_verified/tools/*,train_verified/completions/*,train_verified/batch/trained_tokens_per_step,train_verified/batch/forwarded_tokens_per_step,eval_observed/harness_difficulty/*,train_verified/*,eval_coverage/*,eval/pass_at_1,eval/provisional_pass_at_1,train/reward_rolling20,eval/harness/*,eval/difficulty/*'
4
+
5
  import trackio
6
+
7
  trackio.show(project='data-agent-rl-comparison', open_browser=False)
lfm_checkpoint_coverage.csv ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ arm,step,graded,correct,pass_at_1,complete,native_count_coverage
2
+ baseline,0,998,421,0.4218436873747495,False,998
3
+ opencode,100,986,473,0.47971602434077076,False,986
4
+ opencode,200,979,501,0.5117466802860061,False,979
5
+ opencode,300,989,496,0.5015166835187057,False,989
6
+ opencode,400,999,506,0.5065065065065065,False,999
7
+ opencode,500,1000,501,0.501,True,1000
8
+ opencode,600,1000,518,0.518,True,1000
9
+ opencode,700,1000,508,0.508,True,1000
10
+ opencode,800,1000,512,0.512,True,1000
11
+ opencode,900,1000,524,0.524,True,1000
12
+ opencode,1000,1000,523,0.523,True,1000
13
+ multi-harness,100,988,465,0.47064777327935226,False,988
14
+ multi-harness,200,988,520,0.5263157894736842,False,988
15
+ multi-harness,300,1000,507,0.507,True,1000
16
+ multi-harness,400,1000,510,0.51,True,1000
17
+ multi-harness,500,1000,521,0.521,True,1000
18
+ multi-harness,600,1000,507,0.507,True,1000
19
+ multi-harness,700,1000,546,0.546,True,1000
20
+ multi-harness,800,1000,540,0.54,True,1000
21
+ multi-harness,850,1000,545,0.545,True,1000
22
+ multi-harness,900,1000,520,0.52,True,1000
23
+ multi-harness,1000,1000,542,0.542,True,1000
lfm_final_efficiency.pdf ADDED
Binary file (82.7 kB). View file
 
lfm_final_efficiency.png ADDED

Git LFS Details

  • SHA256: cead6101f2ee4a7fce0e69cdfd0f8d23fb7fde94096f07a1761c0459768190e4
  • Pointer size: 132 Bytes
  • Size of remote file: 1.52 MB
lfm_final_overview.pdf ADDED
Binary file (74.3 kB). View file
 
lfm_final_overview.png ADDED

Git LFS Details

  • SHA256: 2b4657ec24c13b32265511ff7a5c6c131be1d96f0d9bcad309909dc8e29eb39d
  • Pointer size: 131 Bytes
  • Size of remote file: 540 kB