AdithyaSK HF Staff commited on
Commit
8ee5d8a
·
verified ·
1 Parent(s): ee45a15

Complete Harbor multi-harness 900/1000 with audited verifier-budget outcomes

Browse files
Files changed (8) hide show
  1. README.md +48 -19
  2. RECONCILIATION.md +11 -0
  3. REPORT.md +42 -8
  4. checkpoint_scores.csv +48 -0
  5. comparison.pdf +0 -0
  6. comparison.png +2 -2
  7. comparison.svg +0 -0
  8. snapshot.json +0 -0
README.md CHANGED
@@ -10,22 +10,51 @@ pinned: false
10
 
11
  # Qwen3.5-2B training comparison
12
 
13
- Compare **Harbor multi-harness**, **Native OpenCode**, and **Harbor OpenCode-only** in one Trackio project.
14
-
15
- [Open the pass@1 comparison — all three runs selected](https://huggingenvs-data-agent-training-comparison-trackio.hf.space/?project=qwen35-2b-harbor-vs-opencode-20260916&run_ids=3ae29a23763093285702b71a1f76805a%2Cbeb2604c8f737da4262b1ab19b8b0cbd%2Cc5b445fa337b56b139c1e6b34ae35409&smoothing=0&metric_filter=%5Eeval%2Fpass_at_1%24)
16
-
17
- | Harbor OpenCode-only | Pass@1 |
18
- | --- | ---: |
19
- | Baseline (step 0) | 14.6% |
20
- | Checkpoint 100 | 24.5% |
21
- | Checkpoint 200 | 27.1% |
22
-
23
- Each point contains 250 fixed test tasks × four evaluation harnesses. All three points passed full coverage, TiTO and harness-version checks. The overall metric is `eval/pass_at_1`; the dashboard stores fractions (0.146, 0.245, 0.271). The report also compares difficulty and harness scores across all runs.
24
-
25
- - [Checkpoint comparison and difficulty breakdown](REPORT.md)
26
- - [Comparison figure](comparison.png) · [SVG](comparison.svg) · [PDF](comparison.pdf)
27
- - [Checkpoint scores CSV](checkpoint_scores.csv) · [Source snapshot](snapshot.json)
28
-
29
- The report and image are a snapshot from 2026-09-16T22:11:36.405238+00:00. The live dashboard receives newly accepted checkpoint results automatically. Baselines use separate measured E2B and Daytona cohorts; see the report for provenance and training history.
30
-
31
- Public CPU Basic Space, Trackio 0.33.0, persistent Hugging Face bucket. Use zero smoothing for exact checkpoint values.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
 
11
  # Qwen3.5-2B training comparison
12
 
13
+ [Open all three runs in Trackio](https://huggingenvs-data-agent-training-comparison-trackio.hf.space/?project=qwen35-2b-harbor-vs-opencode-20260916&run_ids=3ae29a23763093285702b71a1f76805a%2Cbeb2604c8f737da4262b1ab19b8b0cbd%2Cc5b445fa337b56b139c1e6b34ae35409&smoothing=0&metric_filter=%5Eeval%2Fpass_at_1%24)
14
+
15
+ | Run | Checkpoint | Pass@1 |
16
+ | --- | ---: | ---: |
17
+ | Harbor multi-harness | 0 | 14.6% |
18
+ | Harbor multi-harness | 100 | 24.8% |
19
+ | Harbor multi-harness | 200 | 26.3% |
20
+ | Harbor multi-harness | 300 | 28.6% |
21
+ | Harbor multi-harness | 400 | 33.3% |
22
+ | Harbor multi-harness | 500 | 37.0% |
23
+ | Harbor multi-harness | 600 | 31.8% |
24
+ | Harbor multi-harness | 684 | 32.1% |
25
+ | Harbor multi-harness | 700 | 28.8% |
26
+ | Harbor multi-harness | 800 | 27.0% |
27
+ | Harbor multi-harness | 900 | 22.7% |
28
+ | Harbor multi-harness | 1000 | 26.3% |
29
+ | Native OpenCode | 0 | 15.9% |
30
+ | Native OpenCode | 100 | 19.7% |
31
+ | Native OpenCode | 200 | 22.1% |
32
+ | Native OpenCode | 300 | 21.6% |
33
+ | Native OpenCode | 400 | 26.4% |
34
+ | Native OpenCode | 500 | 23.1% |
35
+ | Native OpenCode | 600 | 25.1% |
36
+ | Native OpenCode | 700 | 23.2% |
37
+ | Native OpenCode | 800 | 25.6% |
38
+ | Native OpenCode | 900 | 25.3% |
39
+ | Native OpenCode | 1000 | 29.8% |
40
+ | Harbor OpenCode-only | 0 | 14.6% |
41
+ | Harbor OpenCode-only | 100 | 24.5% |
42
+ | Harbor OpenCode-only | 200 | 27.1% |
43
+ | Harbor OpenCode-only | 300 | 28.5% |
44
+ | Harbor OpenCode-only | 400 | 32.8% |
45
+
46
+ ## Budget-outcome reconciliation
47
+
48
+ Harbor multi-harness checkpoints 900 and 1,000 now include the original verifier scores for their remaining agent-budget failures. The old driver discarded these scores whenever an agent error was present. Recovery accepts only recorded deadline/output-budget terminations with matching native results, verifier files and exact-token captures; transport and capture failures remain excluded. No missing value was guessed or assigned a default zero. Every previously accepted grade is unchanged.
49
+
50
+ This accounting correction is limited to the missing cells at these checkpoints. Earlier accepted cells retain the historical retry policy. A uniform retrospective audit is required before interpreting the whole curve as strict first-attempt pass@1. Scores here retain the experiment's existing pass@1 metric name and provenance.
51
+
52
+ All three trainers finished 1,000 optimizer steps. Harbor OpenCode-only reached the step cap with 523 distinct training tasks represented in optimizer input; it did not cover the entire 1,000-task set.
53
+
54
+ - Checkpoint 900: **22.7%**, 1,000/1,000 cells; 993 prior grades preserved, 7 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
55
+
56
+ - Checkpoint 1000: **26.3%**, 1,000/1,000 cells; 984 prior grades preserved, 16 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
57
+
58
+ [Full report and difficulty breakdown](REPORT.md) · [Figure](comparison.png) · [CSV](checkpoint_scores.csv)
59
+
60
+ Static artifacts updated 2026-09-17T07:54:52.334591+00:00; the live dashboard receives newly accepted results automatically. Public CPU Basic Space.
RECONCILIATION.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## Budget-outcome reconciliation
2
+
3
+ Harbor multi-harness checkpoints 900 and 1,000 now include the original verifier scores for their remaining agent-budget failures. The old driver discarded these scores whenever an agent error was present. Recovery accepts only recorded deadline/output-budget terminations with matching native results, verifier files and exact-token captures; transport and capture failures remain excluded. No missing value was guessed or assigned a default zero. Every previously accepted grade is unchanged.
4
+
5
+ This accounting correction is limited to the missing cells at these checkpoints. Earlier accepted cells retain the historical retry policy. A uniform retrospective audit is required before interpreting the whole curve as strict first-attempt pass@1. Scores here retain the experiment's existing pass@1 metric name and provenance.
6
+
7
+ All three trainers finished 1,000 optimizer steps. Harbor OpenCode-only reached the step cap with 523 distinct training tasks represented in optimizer input; it did not cover the entire 1,000-task set.
8
+
9
+ - Checkpoint 900: **22.7%**, 1,000/1,000 cells; 993 prior grades preserved, 7 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
10
+
11
+ - Checkpoint 1000: **26.3%**, 1,000/1,000 cells; 984 prior grades preserved, 16 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
REPORT.md CHANGED
@@ -1,14 +1,24 @@
1
  # Harbor and OpenCode — consolidated training and pass@1
2
 
3
- Updated: 2026-09-16T22:11:36.405238+00:00
4
 
5
  [Live Trackio dashboard](https://huggingface.co/spaces/HuggingEnvs/data-agent-training-comparison-trackio) · [Overview image](comparison.png) · [Snapshot](snapshot.json)
6
 
7
- Qwen3.5-2B; 1,000 optimizer-step target per run. Recorded training: Harbor multi-harness: 1000 steps; Native OpenCode: 1000 steps; Harbor OpenCode-only: 518 steps. Every accepted checkpoint has 250 fixed tasks × four harnesses = 1,000 grades. Task difficulty: 33 easy, 118 medium, 99 hard (13.2% / 47.2% / 39.6%). Scores retain first graded attempts; incomplete and failed-audit evaluations are excluded. Missing scores are not estimated.
8
 
9
  Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9% for the native OpenCode checkpoint evaluator. The standalone native OpenCode 8.4% baseline uses a different harness protocol and is excluded here. Infrastructure and training recipe histories differ; this is an observational comparison, not a controlled causal experiment.
10
 
11
- [Open the pass@1 comparison with all three runs selected](https://huggingenvs-data-agent-training-comparison-trackio.hf.space/?project=qwen35-2b-harbor-vs-opencode-20260916&run_ids=3ae29a23763093285702b71a1f76805a%2Cbeb2604c8f737da4262b1ab19b8b0cbd%2Cc5b445fa337b56b139c1e6b34ae35409&smoothing=0&metric_filter=%5Eeval%2Fpass_at_1%24)
 
 
 
 
 
 
 
 
 
 
12
 
13
  ## Overall checkpoint curve
14
 
@@ -17,15 +27,15 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
17
  | 0 (baseline) | 14.6% | 15.9% | 14.6% |
18
  | 100 | 24.8% | 19.7% | 24.5% |
19
  | 200 | 26.3% | 22.1% | 27.1% |
20
- | 300 | 28.6% | 21.6% | Pending |
21
- | 400 | 33.3% | 26.4% | Pending |
22
  | 500 | 37.0% | 23.1% | Pending |
23
  | 600 | 31.8% | 25.1% | Pending |
24
  | 684 (recovery) | 32.1% | Not scheduled | Not scheduled |
25
  | 700 | 28.8% | 23.2% | Pending |
26
  | 800 | 27.0% | 25.6% | Pending |
27
- | 900 | Pending | 25.3% | Pending |
28
- | 1000 | Pending | 29.8% | Pending |
29
 
30
  ## Harbor multi-harness
31
 
@@ -43,6 +53,8 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
43
  | 684 | 32.1% | 75.8% | 35.8% | 13.1% |
44
  | 700 | 28.8% | 60.6% | 33.9% | 12.1% |
45
  | 800 | 27.0% | 59.1% | 32.6% | 9.6% |
 
 
46
 
47
  ### Harness × difficulty at every checkpoint
48
 
@@ -88,6 +100,14 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
88
  | 800 | claude-code | 32.0% | 66.7% (22/33) | 37.3% (44/118) | 14.1% (14/99) |
89
  | 800 | codex | 26.8% | 66.7% (22/33) | 31.4% (37/118) | 8.1% (8/99) |
90
  | 800 | mini-swe-agent | 26.0% | 66.7% (22/33) | 28.8% (34/118) | 9.1% (9/99) |
 
 
 
 
 
 
 
 
91
 
92
  ### Training history
93
 
@@ -113,6 +133,8 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
113
  - Step 684: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-20260915/checkpoint-evals/step-000684/scores.json`; SHA256 `fb54bd52f352463e405bee8067ef24b27147bfd02d41429561f440a916f5d776`.
114
  - Step 700: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-000700/scores.json`; SHA256 `39c2ead8681847c6c398eb6b22ae8919aabaff3ad6b668d81b5961564139f8be`.
115
  - Step 800: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-000800/scores.json`; SHA256 `65130786707d07f68cfa145fcd5ad7308890cb89a305b65382dfbec36ef7a150`.
 
 
116
 
117
  ## Native OpenCode
118
 
@@ -210,6 +232,8 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
210
  | 0 | 14.6% | 40.2% | 14.4% | 6.3% |
211
  | 100 | 24.5% | 50.0% | 29.2% | 10.4% |
212
  | 200 | 27.1% | 62.1% | 32.2% | 9.3% |
 
 
213
 
214
  ### Harness × difficulty at every checkpoint
215
 
@@ -227,18 +251,28 @@ Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9%
227
  | 200 | claude-code | 28.4% | 57.6% (19/33) | 37.3% (44/118) | 8.1% (8/99) |
228
  | 200 | codex | 30.8% | 66.7% (22/33) | 37.3% (44/118) | 11.1% (11/99) |
229
  | 200 | mini-swe-agent | 20.0% | 66.7% (22/33) | 18.6% (22/118) | 6.1% (6/99) |
 
 
 
 
 
 
 
 
230
 
231
  ### Training history
232
 
233
  | Allocation | First optimizer step | Last optimizer step |
234
  | --- | ---: | ---: |
235
- | 81075 | 1 | 518 |
236
 
237
  ### Score provenance
238
 
239
  - Step 0: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-baseline-20260914/job-78215/canonical_results.json`; SHA256 `8c4f5bced4eff04b0c2e5f41806da9ae1b8a4c0fe356ddf926e1f781c6bb9ac6`.
240
  - Step 100: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000100/scores.json`; SHA256 `08278ea43f152961c9fd3b0b8cc598a0cf5d891b658ef7f5314f95c62ade44d8`.
241
  - Step 200: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000200/scores.json`; SHA256 `718d706a3b61a8a171e567e99084c30f0b4e532d55d9cbe1f38b0a0f88e9741b`.
 
 
242
 
243
  ## Dashboard metric guide
244
 
 
1
  # Harbor and OpenCode — consolidated training and pass@1
2
 
3
+ Updated: 2026-09-17T07:54:52.334591+00:00
4
 
5
  [Live Trackio dashboard](https://huggingface.co/spaces/HuggingEnvs/data-agent-training-comparison-trackio) · [Overview image](comparison.png) · [Snapshot](snapshot.json)
6
 
7
+ Qwen3.5-2B; 1,000 optimizer-step target per run. Recorded training: Harbor multi-harness: 1000 steps; Native OpenCode: 1000 steps; Harbor OpenCode-only: 1000 steps. Every accepted checkpoint has 250 fixed tasks × four harnesses = 1,000 grades. Task difficulty: 33 easy, 118 medium, 99 hard (13.2% / 47.2% / 39.6%). Scores retain first graded attempts; incomplete and failed-audit evaluations are excluded. Missing scores are not estimated.
8
 
9
  Baselines are separate measured cohorts: Harbor/E2B 14.6%; Harbor/Daytona 15.9% for the native OpenCode checkpoint evaluator. The standalone native OpenCode 8.4% baseline uses a different harness protocol and is excluded here. Infrastructure and training recipe histories differ; this is an observational comparison, not a controlled causal experiment.
10
 
11
+ ## Budget-outcome reconciliation
12
+
13
+ Harbor multi-harness checkpoints 900 and 1,000 now include the original verifier scores for their remaining agent-budget failures. The old driver discarded these scores whenever an agent error was present. Recovery accepts only recorded deadline/output-budget terminations with matching native results, verifier files and exact-token captures; transport and capture failures remain excluded. No missing value was guessed or assigned a default zero. Every previously accepted grade is unchanged.
14
+
15
+ This accounting correction is limited to the missing cells at these checkpoints. Earlier accepted cells retain the historical retry policy. A uniform retrospective audit is required before interpreting the whole curve as strict first-attempt pass@1. Scores here retain the experiment's existing pass@1 metric name and provenance.
16
+
17
+ All three trainers finished 1,000 optimizer steps. Harbor OpenCode-only reached the step cap with 523 distinct training tasks represented in optimizer input; it did not cover the entire 1,000-task set.
18
+
19
+ - Checkpoint 900: **22.7%**, 1,000/1,000 cells; 993 prior grades preserved, 7 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
20
+
21
+ - Checkpoint 1000: **26.3%**, 1,000/1,000 cells; 984 prior grades preserved, 16 original budget outcomes recovered. Full TiTO and pinned harness-version checks passed.
22
 
23
  ## Overall checkpoint curve
24
 
 
27
  | 0 (baseline) | 14.6% | 15.9% | 14.6% |
28
  | 100 | 24.8% | 19.7% | 24.5% |
29
  | 200 | 26.3% | 22.1% | 27.1% |
30
+ | 300 | 28.6% | 21.6% | 28.5% |
31
+ | 400 | 33.3% | 26.4% | 32.8% |
32
  | 500 | 37.0% | 23.1% | Pending |
33
  | 600 | 31.8% | 25.1% | Pending |
34
  | 684 (recovery) | 32.1% | Not scheduled | Not scheduled |
35
  | 700 | 28.8% | 23.2% | Pending |
36
  | 800 | 27.0% | 25.6% | Pending |
37
+ | 900 | 22.7% | 25.3% | Pending |
38
+ | 1000 | 26.3% | 29.8% | Pending |
39
 
40
  ## Harbor multi-harness
41
 
 
53
  | 684 | 32.1% | 75.8% | 35.8% | 13.1% |
54
  | 700 | 28.8% | 60.6% | 33.9% | 12.1% |
55
  | 800 | 27.0% | 59.1% | 32.6% | 9.6% |
56
+ | 900 | 22.7% | 46.2% | 26.1% | 10.9% |
57
+ | 1000 | 26.3% | 42.4% | 32.6% | 13.4% |
58
 
59
  ### Harness × difficulty at every checkpoint
60
 
 
100
  | 800 | claude-code | 32.0% | 66.7% (22/33) | 37.3% (44/118) | 14.1% (14/99) |
101
  | 800 | codex | 26.8% | 66.7% (22/33) | 31.4% (37/118) | 8.1% (8/99) |
102
  | 800 | mini-swe-agent | 26.0% | 66.7% (22/33) | 28.8% (34/118) | 9.1% (9/99) |
103
+ | 900 | opencode | 2.4% | 0.0% (0/33) | 3.4% (4/118) | 2.0% (2/99) |
104
+ | 900 | claude-code | 36.4% | 72.7% (24/33) | 43.2% (51/118) | 16.2% (16/99) |
105
+ | 900 | codex | 19.2% | 48.5% (16/33) | 20.3% (24/118) | 8.1% (8/99) |
106
+ | 900 | mini-swe-agent | 32.8% | 63.6% (21/33) | 37.3% (44/118) | 17.2% (17/99) |
107
+ | 1000 | opencode | 5.6% | 6.1% (2/33) | 8.5% (10/118) | 2.0% (2/99) |
108
+ | 1000 | claude-code | 38.8% | 72.7% (24/33) | 44.9% (53/118) | 20.2% (20/99) |
109
+ | 1000 | codex | 24.8% | 45.5% (15/33) | 31.4% (37/118) | 10.1% (10/99) |
110
+ | 1000 | mini-swe-agent | 36.0% | 45.5% (15/33) | 45.8% (54/118) | 21.2% (21/99) |
111
 
112
  ### Training history
113
 
 
133
  - Step 684: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-20260915/checkpoint-evals/step-000684/scores.json`; SHA256 `fb54bd52f352463e405bee8067ef24b27147bfd02d41429561f440a916f5d776`.
134
  - Step 700: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-000700/scores.json`; SHA256 `39c2ead8681847c6c398eb6b22ae8919aabaff3ad6b668d81b5961564139f8be`.
135
  - Step 800: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-000800/scores.json`; SHA256 `65130786707d07f68cfa145fcd5ad7308890cb89a305b65382dfbec36ef7a150`.
136
+ - Step 900: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-000900/scores.json`; SHA256 `2276bc4d8ad85c2913b0bed921013951ec2ca968bfbbece737681ce13df9b250`.
137
+ - Step 1000: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-long-prod-cont-20260915/checkpoint-evals/step-001000/scores.json`; SHA256 `3172c293f6386c7732267fa46c9718403b3fa3462f88e9299a01139ad410c99b`.
138
 
139
  ## Native OpenCode
140
 
 
232
  | 0 | 14.6% | 40.2% | 14.4% | 6.3% |
233
  | 100 | 24.5% | 50.0% | 29.2% | 10.4% |
234
  | 200 | 27.1% | 62.1% | 32.2% | 9.3% |
235
+ | 300 | 28.5% | 64.4% | 32.2% | 12.1% |
236
+ | 400 | 32.8% | 70.5% | 39.0% | 12.9% |
237
 
238
  ### Harness × difficulty at every checkpoint
239
 
 
251
  | 200 | claude-code | 28.4% | 57.6% (19/33) | 37.3% (44/118) | 8.1% (8/99) |
252
  | 200 | codex | 30.8% | 66.7% (22/33) | 37.3% (44/118) | 11.1% (11/99) |
253
  | 200 | mini-swe-agent | 20.0% | 66.7% (22/33) | 18.6% (22/118) | 6.1% (6/99) |
254
+ | 300 | opencode | 28.0% | 69.7% (23/33) | 30.5% (36/118) | 11.1% (11/99) |
255
+ | 300 | claude-code | 30.4% | 66.7% (22/33) | 33.9% (40/118) | 14.1% (14/99) |
256
+ | 300 | codex | 32.0% | 63.6% (21/33) | 36.4% (43/118) | 16.2% (16/99) |
257
+ | 300 | mini-swe-agent | 23.6% | 57.6% (19/33) | 28.0% (33/118) | 7.1% (7/99) |
258
+ | 400 | opencode | 35.6% | 75.8% (25/33) | 43.2% (51/118) | 13.1% (13/99) |
259
+ | 400 | claude-code | 33.6% | 72.7% (24/33) | 42.4% (50/118) | 10.1% (10/99) |
260
+ | 400 | codex | 33.2% | 69.7% (23/33) | 34.7% (41/118) | 19.2% (19/99) |
261
+ | 400 | mini-swe-agent | 28.8% | 63.6% (21/33) | 35.6% (42/118) | 9.1% (9/99) |
262
 
263
  ### Training history
264
 
265
  | Allocation | First optimizer step | Last optimizer step |
266
  | --- | ---: | ---: |
267
+ | 81075 | 1 | 1000 |
268
 
269
  ### Score provenance
270
 
271
  - Step 0: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/multi4-baseline-20260914/job-78215/canonical_results.json`; SHA256 `8c4f5bced4eff04b0c2e5f41806da9ae1b8a4c0fe356ddf926e1f781c6bb9ac6`.
272
  - Step 100: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000100/scores.json`; SHA256 `08278ea43f152961c9fd3b0b8cc598a0cf5d891b658ef7f5314f95c62ade44d8`.
273
  - Step 200: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000200/scores.json`; SHA256 `718d706a3b61a8a171e567e99084c30f0b4e532d55d9cbe1f38b0a0f88e9741b`.
274
+ - Step 300: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000300/scores.json`; SHA256 `58ccb317e9c50e09afe5211744049ccac6cf8aeb690e9021307591798733118c`.
275
+ - Step 400: `/fsx/adithyaskolavi/projects/trl_prod/experiments/async_grpo_harbor_data_agent/logs/harbor-opencode-only-20260916/checkpoint-evals/step-000400/scores.json`; SHA256 `f473d7a6595f5b461da11eeab86b0265fc0d5a903dd50a7f07c13fe591841071`.
276
 
277
  ## Dashboard metric guide
278
 
checkpoint_scores.csv CHANGED
@@ -119,6 +119,30 @@ Harbor multi-harness,800,codex,hard,8,99,0.08080808080808081
119
  Harbor multi-harness,800,mini-swe-agent,easy,22,33,0.6666666666666666
120
  Harbor multi-harness,800,mini-swe-agent,medium,34,118,0.288135593220339
121
  Harbor multi-harness,800,mini-swe-agent,hard,9,99,0.09090909090909091
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
122
  Native OpenCode,0,opencode,easy,8,33,0.24242424242424243
123
  Native OpenCode,0,opencode,medium,20,118,0.1694915254237288
124
  Native OpenCode,0,opencode,hard,4,99,0.04040404040404041
@@ -287,3 +311,27 @@ Harbor OpenCode-only,200,codex,hard,11,99,0.1111111111111111
287
  Harbor OpenCode-only,200,mini-swe-agent,easy,22,33,0.6666666666666666
288
  Harbor OpenCode-only,200,mini-swe-agent,medium,22,118,0.1864406779661017
289
  Harbor OpenCode-only,200,mini-swe-agent,hard,6,99,0.06060606060606061
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
  Harbor multi-harness,800,mini-swe-agent,easy,22,33,0.6666666666666666
120
  Harbor multi-harness,800,mini-swe-agent,medium,34,118,0.288135593220339
121
  Harbor multi-harness,800,mini-swe-agent,hard,9,99,0.09090909090909091
122
+ Harbor multi-harness,900,opencode,easy,0,33,0.0
123
+ Harbor multi-harness,900,opencode,medium,4,118,0.03389830508474576
124
+ Harbor multi-harness,900,opencode,hard,2,99,0.020202020202020204
125
+ Harbor multi-harness,900,claude-code,easy,24,33,0.7272727272727273
126
+ Harbor multi-harness,900,claude-code,medium,51,118,0.4322033898305085
127
+ Harbor multi-harness,900,claude-code,hard,16,99,0.16161616161616163
128
+ Harbor multi-harness,900,codex,easy,16,33,0.48484848484848486
129
+ Harbor multi-harness,900,codex,medium,24,118,0.2033898305084746
130
+ Harbor multi-harness,900,codex,hard,8,99,0.08080808080808081
131
+ Harbor multi-harness,900,mini-swe-agent,easy,21,33,0.6363636363636364
132
+ Harbor multi-harness,900,mini-swe-agent,medium,44,118,0.3728813559322034
133
+ Harbor multi-harness,900,mini-swe-agent,hard,17,99,0.1717171717171717
134
+ Harbor multi-harness,1000,opencode,easy,2,33,0.06060606060606061
135
+ Harbor multi-harness,1000,opencode,medium,10,118,0.0847457627118644
136
+ Harbor multi-harness,1000,opencode,hard,2,99,0.020202020202020204
137
+ Harbor multi-harness,1000,claude-code,easy,24,33,0.7272727272727273
138
+ Harbor multi-harness,1000,claude-code,medium,53,118,0.4491525423728814
139
+ Harbor multi-harness,1000,claude-code,hard,20,99,0.20202020202020202
140
+ Harbor multi-harness,1000,codex,easy,15,33,0.45454545454545453
141
+ Harbor multi-harness,1000,codex,medium,37,118,0.3135593220338983
142
+ Harbor multi-harness,1000,codex,hard,10,99,0.10101010101010101
143
+ Harbor multi-harness,1000,mini-swe-agent,easy,15,33,0.45454545454545453
144
+ Harbor multi-harness,1000,mini-swe-agent,medium,54,118,0.4576271186440678
145
+ Harbor multi-harness,1000,mini-swe-agent,hard,21,99,0.21212121212121213
146
  Native OpenCode,0,opencode,easy,8,33,0.24242424242424243
147
  Native OpenCode,0,opencode,medium,20,118,0.1694915254237288
148
  Native OpenCode,0,opencode,hard,4,99,0.04040404040404041
 
311
  Harbor OpenCode-only,200,mini-swe-agent,easy,22,33,0.6666666666666666
312
  Harbor OpenCode-only,200,mini-swe-agent,medium,22,118,0.1864406779661017
313
  Harbor OpenCode-only,200,mini-swe-agent,hard,6,99,0.06060606060606061
314
+ Harbor OpenCode-only,300,opencode,easy,23,33,0.696969696969697
315
+ Harbor OpenCode-only,300,opencode,medium,36,118,0.3050847457627119
316
+ Harbor OpenCode-only,300,opencode,hard,11,99,0.1111111111111111
317
+ Harbor OpenCode-only,300,claude-code,easy,22,33,0.6666666666666666
318
+ Harbor OpenCode-only,300,claude-code,medium,40,118,0.3389830508474576
319
+ Harbor OpenCode-only,300,claude-code,hard,14,99,0.1414141414141414
320
+ Harbor OpenCode-only,300,codex,easy,21,33,0.6363636363636364
321
+ Harbor OpenCode-only,300,codex,medium,43,118,0.3644067796610169
322
+ Harbor OpenCode-only,300,codex,hard,16,99,0.16161616161616163
323
+ Harbor OpenCode-only,300,mini-swe-agent,easy,19,33,0.5757575757575758
324
+ Harbor OpenCode-only,300,mini-swe-agent,medium,33,118,0.2796610169491525
325
+ Harbor OpenCode-only,300,mini-swe-agent,hard,7,99,0.0707070707070707
326
+ Harbor OpenCode-only,400,opencode,easy,25,33,0.7575757575757576
327
+ Harbor OpenCode-only,400,opencode,medium,51,118,0.4322033898305085
328
+ Harbor OpenCode-only,400,opencode,hard,13,99,0.13131313131313133
329
+ Harbor OpenCode-only,400,claude-code,easy,24,33,0.7272727272727273
330
+ Harbor OpenCode-only,400,claude-code,medium,50,118,0.423728813559322
331
+ Harbor OpenCode-only,400,claude-code,hard,10,99,0.10101010101010101
332
+ Harbor OpenCode-only,400,codex,easy,23,33,0.696969696969697
333
+ Harbor OpenCode-only,400,codex,medium,41,118,0.3474576271186441
334
+ Harbor OpenCode-only,400,codex,hard,19,99,0.1919191919191919
335
+ Harbor OpenCode-only,400,mini-swe-agent,easy,21,33,0.6363636363636364
336
+ Harbor OpenCode-only,400,mini-swe-agent,medium,42,118,0.3559322033898305
337
+ Harbor OpenCode-only,400,mini-swe-agent,hard,9,99,0.09090909090909091
comparison.pdf CHANGED
Binary files a/comparison.pdf and b/comparison.pdf differ
 
comparison.png CHANGED

Git LFS Details

  • SHA256: 83f2ad27c9f9a55d6848f360645ca51083433a1c837a75eb005c7087dd1a96b4
  • Pointer size: 131 Bytes
  • Size of remote file: 646 kB

Git LFS Details

  • SHA256: 5e9a1ef1e02b29a3206c19fe630f41a5214b87403f6193fb0c0992e1935e73d8
  • Pointer size: 131 Bytes
  • Size of remote file: 691 kB
comparison.svg CHANGED
snapshot.json CHANGED
The diff for this file is too large to render. See raw diff