OpenBMB: coding agents switch strategy in only 2.1% of pairs

Felix Laurent 26 Oct 2026 2 min read

ai agents

OpenBMB analyzed 1,338 complete training trajectories from PostTrainBench and says coding agents almost never reopen their first strategy. Only 2.1% of 3,557 adjacent experiment pairs switch strategy. We read the OpenBMB post and the arXiv paper.

What OpenBMB analyzed

PostTrainBench is a benchmark from the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems. It asks an AI agent to post-train a small base model on its own, with 10 hours on one H100 GPU. The base models are Qwen3-1.7B, Qwen3-4B, SmolLM3-3B and Gemma-3-4B. In the original paper, the best agent reached about 23% across the benchmarks, against 51% for the official instruction-tuned models.

The OpenBMB set covers 7 benchmarks and 4 base models. The agents were Claude Code, Codex CLI and OpenCode. OpenBMB says agents improve over base models on every benchmark. With experience, the average score goes from 33.34 to 48.55.

The paper, "What is Missing from AI Post-Training AI", lists Joy Jia Yin Lim, Xin Huang, Hao Peng, Yaxi Lu, Xin Cong, Zhong Zhang, Maosong Sun and Yankai Lin as authors. It first went up on arXiv on Aug 19 and was revised on Sept 28.

Agents adopt execution tips, not strategy tips

OpenBMB says all execution-level suggestions were adopted. None of the strategy-level suggestions were, such as switching to RL or adding an SFT warmup. The arXiv paper puts the strategy change rate at 2.1% of transitions between adjacent training runs.

The agents also differ in their defaults. OpenBMB says Claude Code starts full-parameter SFT 71.9% of the time, while Codex CLI starts PEFT/LoRA 89.5% of the time. The paper describes this as a default strategy tied to the agent rather than to the task.

More experience does not change strategy

OpenBMB says extra experience and memory raise execution but not strategy. On AIME 2025, about 14M extra tokens solve just one more problem. The paper adds that extra reasoning compute helps early on easier tasks, but it refines the chosen strategy instead of revising it. Human review before training changes which strategy the agent locks into, not whether it locks in.

One sentence mid-run

OpenBMB says one sentence switching to the alternative strategy beats autonomous continuation at the same checkpoint and budget. The gains are GSM8K +17.44pp, HumanEval +10.80pp and AIME 2025 +6.67pp. The arXiv abstract confirms only the top figure, up to 17.44 points.

OpenBMB concludes that the missing ability is the trigger to restart strategy decisions mid-trajectory. It says executing a new strategy is not the problem. The paper links this to recursive self-improvement: agents that post-train other models will need training signals that reward revising a plan, not only carrying it out.

What is not verified

The average score and the per-benchmark gains appear in the OpenBMB post but not in the abstract. They are the lab's own results. The abstract does not name the frontier models behind the agents either.