AutoResearch Eval

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Autonomous agents can now improve technical artifacts through long-horizon experimentation. Yet final scores reveal little about how they succeed or fail. We examine these loops through Solution Framing, Execution, and Feedback Control, complemented by analyses of experience reuse, harness effects, and solution novelty. Taken together, the results show that current agents behave more like engineering optimizers than autonomous researchers: they can deliver practical improvements, but remain inconsistent across runs and rarely produce genuinely novel methods.

Read the paper (opens in a new tab)Explore agent trajectories

Study at a glance

frontier models
7
long-horizon tasks
36
rollouts
756
wall-clock budget
2-12h
inference cost across all experiments
~$100k

In one minute

Where current auto-research agents stand

Overall assessment

Current agents operate more like engineering optimizers than fully autonomous researchers.

Within bounded research loops, they can formulate practical directions, implement working solutions, and improve technical artifacts. Yet success varies across runs, genuine algorithmic innovation remains rare, and realized performance is shaped by process bottlenecks, accumulated experience, and harness design.

  1. 01

    Reliability separates current models more than peak performance.

    The strongest-to-weakest gap is 0.237 on avg@3 but only 0.122 on best@3: several models can reach competitive solutions, but do so with substantially different consistency across repeated runs.

  2. 02

    Outcome scores conceal where research actually fails.

    Similar rewards can hide different process profiles, and dominant bottlenecks shift by workload. Only three of 252 best-seed solutions qualify as novel approaches, exposing a gap between optimization performance and methodological novelty.

  3. 03

    Research performance is not fixed by the backbone model alone.

    Experience can improve or degrade performance strongly enough to change model ordering, while native harnesses mainly improve run-to-run reliability. Automated harness optimization offers further headroom.

02

Evaluation Setting and Outcome-Level Landscape

We compare seven frontier models under the same tasks, harness, and protocol to establish a controlled view of final performance. This outcome landscape anchors the process, experience, harness, and novelty analyses that follow.

overall avg@3 and best@3 model comparison

avg@3best@3 gain
0.4
0.5
0.6
0.7
0.8
0.9
Claude
GLM-5.2
GPT-5.5
Gemini
Kimi
LongCat
DeepSeek
Solid segments show avg@3, while full bar heights show best@3. Category views compare the same seven models across Model Development, System Optimization, Puzzle & Challenge, and CUDA.

Ranked results

Model rewards for Overall; rank is based on avg@3.

Rank 2GLM-5.2
avg@3
0.682
best@3
0.757
Δ gap
0.075
Rank 3GPT-5.5
avg@3
0.663
best@3
0.772
Δ gap
0.109

  1. Average performance separates models more sharply than best performance.

    The highest-to-lowest gap is 0.237 on avg@3 but only 0.122 on best@3. Lower-ranked models can still reach competitive solutions, but do so less consistently across repeated rollouts.

  2. Task categories reveal distinct capability profiles.

    Claude leads avg@3 in three categories, while GLM narrowly leads Puzzle & Challenge. Puzzle & Challenge separates models least, whereas CUDA separates them most and splits the metric leaders: Claude on avg@3, GPT on best@3.

03

Inside the Research Loop: Process Evaluation

Auto research unfolds as a repeated loop of proposing a direction, implementing a change, observing the result, and deciding what to do next. We look inside that loop to distinguish failures of Solution Framing, Execution, and Feedback Control.

Three measurements of one research loop

Solution Framing, Execution, and Feedback Control describe behavior inside a research run.

Process evaluation dimensions

C1
Solution Framing
Solution Framing process dimensionA reward curve that rises quickly and reaches a high plateau, rewarding both height and speed.high reward, reached fast

Captures how quickly the agent reaches a strong solution direction.

C2
Execution
Execution process dimensionA jagged reward curve ends in a failed delivery, showing that promising ideas still require correct execution.delivery fails

Proposed changes must execute correctly and pass task checks.

C3
Feedback Control
Feedback Control process dimensionThe curve falls after a failed try, then recovers toward the best-so-far result.best-so-farfailed try, then recovery

Rewards preserving the best result and recovering after regressions.

Process dimensions across seven models

Outcome, Solution Framing, Execution, and Feedback Control across seven models. Process scores use task-macro aggregation and lie in [0, 1], where higher is better.

Claude leads Outcome, Solution Framing, and Execution. GPT and Gemini reach similar outcomes through different balances of Execution and Feedback Control, while LongCat shows that high Feedback Control alone cannot compensate for weak Solution Framing.

Outcome

C1 · Solution Framing

C2 · Execution

C3 · Feedback Control

Outcome and process scores across seven models
ModelOutcomeC1 · Solution FramingC2 · ExecutionC3 · Feedback Control
Claude-Opus-4.70.7390.6120.9670.920
GLM-5.20.6820.5390.9370.911
GPT-5.50.6630.5550.9580.858
Gemini-3.1-Pro0.6520.5550.8890.921
Kimi-K2.7-Code0.5840.4730.8800.875
LongCat-2.00.5720.4780.8880.928
DeepSeek-V4-Pro0.5240.5190.8880.772

  1. Strong outcomes depend mainly on effective Solution Framing and reliable Execution.

    Claude leads Outcome (0.739), Solution Framing (0.612), and Execution (0.967). Execution is relatively compressed across models, while the wider Solution Framing and Feedback Control ranges reveal differences that successful delivery alone cannot explain.

  2. Similar outcomes can conceal sharply different Execution and Feedback Control profiles.

    GPT and Gemini have nearly identical outcomes and the same Solution Framing score, yet GPT is stronger in Execution while Gemini is stronger in Feedback Control. LongCat shows the complementary limit: the highest Feedback Control score cannot compensate for weaker framing and outcome.

Process dimensions by task category

Process dimensions by task category, averaged over the seven models.

Puzzle and Challenge has the strongest scores across all three dimensions. CUDA has the lowest solution framing and execution scores, while Model Development has the lowest feedback control score.

  • C1 Solution Framing
  • C2 Execution
  • C3 Feedback Control

Model Development

7 tasks
C10.471
C20.985
C30.743

System Optimization

15 tasks
C10.469
C20.892
C30.907

Puzzle & Challenge

10 tasks
C10.737
C20.931
C30.930

CUDA

4 tasks
C10.370
C20.850
C30.924
Process dimension values by task category
WorkloadTasksC1 Solution FramingC2 ExecutionC3 Feedback Control
Model Development70.4710.9850.743
System Optimization150.4690.8920.907
Puzzle & Challenge100.7370.9310.930
CUDA40.3700.8500.924

  1. Different task categories expose different bottlenecks in the research loop.

    CUDA is constrained by Solution Framing and Execution despite strong Feedback Control, while Model Development shows the reverse pattern. Puzzle & Challenge is strongest across all three process dimensions.

04

Learning from Experience

Practical auto research should improve as experience accumulates. We test this evolving capability at two scales: reuse within the same task and transfer from solved tasks to a held-out target.

Experience within and across tasks

Intra-task and inter-task comparisons isolate whether accumulated experience improves later decisions.

Intra-task and inter-task experience comparisons

Intra-Task Self-Improvement
Intra-task experience comparisonFrom the same branch point, the trajectory with accumulated experience finishes above the trajectory without it.with experiencewithout experience

The next commit is compared with and without prior exploration from the same task.

Inter-Task Self-Improvement
Inter-task experience comparisonA source-task trajectory is distilled into a lesson, then a target task is solved with and without that lesson.source taskdistilledlessontarget taskwith experiencewithout experience

Distilled lessons from a source task are tested on a held-out target.

Intra-Task Experience Reuse

Intra-Task Experience Reuse

Per-model first-commit reward with and without retained experience, together with the corresponding intra-task gain, averaged over 32 retained trajectories.

with experiencewithout experiencegain Δ = reward(with) − reward(without)
Intra-task experience-reuse data
ModelWith experienceWithout experienceGain
LongCat-2.00.630.48+0.145
Gemini-3.1-Pro0.700.57+0.128
DeepSeek-V4-Pro0.590.50+0.089
GLM-5.20.720.64+0.079
GPT-5.50.660.58+0.073
Claude-Opus-4.70.740.70+0.036
Kimi-K2.7-Code0.520.54-0.013

  1. Intra-task experience generally improves the next commit across models.

    Six of seven models improve on average. Kimi is the sole aggregate exception at -0.0127, although it still benefits from retained experience on more tasks than it is harmed.

  2. Models differ widely in how much they rely on intra-task experience.

    Claude shows the smallest positive gain (+0.0362), while LongCat combines the largest gain (+0.1454) with the weakest Solution Framing. Accumulated exploration matters more when strong solutions are harder to formulate immediately.

  3. Why retained experience is usually beneficial, and when it backfires.

    Memory helps when it preserves a known dead end, a tuned configuration, or a hard-won implementation that cannot be rediscovered within the remaining budget. It backfires when retained state carries a premature conclusion or anchors the agent to a local optimum.

Inter-Task Experience Reuse

Inter-Task Experience Reuse

Per-model avg@3 with and without distilled experience, together with the corresponding inter-task gain across 19 held-out targets.

with experiencewithout experiencegain Δ = reward(with) − reward(without)
Inter-task experience-reuse data
ModelWith experienceWithout experienceGain
DeepSeek-V4-Pro0.440.35+0.093
GPT-5.50.540.48+0.063
GLM-5.20.550.51+0.040
Kimi-K2.7-Code0.480.46+0.021
Claude-Opus-4.70.600.60+0.001
Gemini-3.1-Pro0.520.53-0.017
LongCat-2.00.410.43-0.021

  1. Initial performance does not reliably predict a model’s ability to improve through experience.

    DeepSeek begins from the weakest lesson-free baseline yet records the largest gains, while the stronger Gemini declines on avg@3. Experience can therefore narrow, widen, or reverse the original ranking.

  2. Experience reuse can improve performance but remains unstable: successful transfer abstracts general principles, whereas failures misapply source-specific tactics or reinforce evaluator-specific shortcuts.

    Successful lessons capture general practices such as constraint checking, verification, and rollback. Failures either transplant a tactic to the wrong workload or turn evaluator-specific behavior into an apparent gain.

  3. Experience transfers more effectively through explicitly distilled, self-generated lessons.

    Distilled lessons outperform raw workspaces for all three tested models, and self-generated lessons outperform cross-model lessons for both GLM and LongCat. Transfer depends on filtering noise and matching the lesson to the receiving model.

05

The Role of the Agent Harness

The harness shapes how an agent plans, executes tools, observes failures, and preserves progress over long runs. We compare leading, native, and open-source harnesses, then test how far an evolved Auto Harness transfers beyond its seed tasks.

Harness Comparison: Leading, Native, and Open-Source Harnesses

Coding harness comparison across Claude Code, each model’s native harness, and OpenCode. Dark bars show avg@3 and light bars show best@3.

Model ranking is unchanged across harnesses. Native and OpenCode harnesses improve GPT and Kimi average performance more than best performance, indicating greater run-to-run stability.

Solid extent: avg@3Hatched extent: best@3Scale 0.000 - 1.000

Claude-Opus-4.7

Claude Code
avg 0.739best 0.790
OpenCode
avg 0.736best 0.825

GPT-5.5

Claude Code
avg 0.663best 0.772
Codex CLI
avg 0.682best 0.763
OpenCode
avg 0.678best 0.761

Kimi-K2.7-Code

Claude Code
avg 0.587best 0.729
Kimi Code CLI
avg 0.642best 0.739
OpenCode
avg 0.632best 0.724
  • Shared Claude Code
  • Model-native harness
  • OpenCode
Coding harness comparison values
ModelHarnessavg@3best@3
Claude-Opus-4.7Claude Code (native)0.7390.790
Claude-Opus-4.7OpenCode0.7360.825
GPT-5.5Claude Code0.6630.772
GPT-5.5Codex CLI (native)0.6820.763
GPT-5.5OpenCode0.6780.761
Kimi-K2.7-CodeClaude Code0.5870.729
Kimi-K2.7-CodeKimi Code CLI (native)0.6420.739
Kimi-K2.7-CodeOpenCode0.6320.724

Auto Harness

Gain of the evolved harness over the original harness on seed tasks, held-out System Optimization tasks, cross-model System Optimization tasks, and unrelated task families.

The evolved harness improves its seed tasks, held-out LongCat System Optimization tasks, and GPT-5.5 System Optimization tasks. On unrelated task families, avg@3 falls slightly while best@3 rises slightly.

Seed tasks

LongCat-2.0 System Optimization tasks used for evolution

+0.123+0.065

Held-out SysOpt

Remaining LongCat-2.0 System Optimization tasks

+0.057+0.043

GPT-5.5 SysOpt

Cross-model transfer on System Optimization tasks

+0.027+0.010

Other families

Unrelated task families

-0.014+0.013
avg@3best@3
Evolved harness gains by transfer setting
SettingContextavg@3 gainbest@3 gain
Seed tasksLongCat-2.0 System Optimization tasks used for evolution+0.123+0.065
Held-out SysOptRemaining LongCat-2.0 System Optimization tasks+0.057+0.043
GPT-5.5 SysOptCross-model transfer on System Optimization tasks+0.027+0.010
Other familiesUnrelated task families-0.014+0.013

  1. The three harness settings achieve comparable aggregate performance and preserve model rankings, differing mainly in run-to-run stability.

    Across harnesses, best@3 moves by at most 0.035, while avg@3 rises more for GPT and especially Kimi under native and OpenCode harnesses. Harness choice therefore improves stability more than the capability ceiling.

06

Solution Nature and Novelty

High reward does not reveal whether an agent discovered a new idea or assembled established techniques. We classify 252 best-seed solutions to examine what current auto research agents actually produce.

Solution Nature and Novelty

Solution nature across 252 best-seed solutions. Each model contributes 36 solutions; three novel approaches remain after manual review.

111 of 252 solutions use composition-stacking. 16 use evaluation-specific shortcuts and 3 qualify as validated novel approaches.

111/252

Composition-stacking

The largest solution category for every model

16/252

Evaluation-hacking

Evaluation-specific shortcuts

3/252

Validated novel approaches

Retained after manual review

  • Param-tune
  • Training-signal / data engineering
  • Structural-swap
  • Composition-stacking
  • Search-hardcode
  • Evaluation-hacking
  • Other
  • Novel-approach
Claude
245213
GLM-5.2
336202
GPT-5.5
2371238
Gemini
3312134
Kimi
3471533
LongCat
429162
DeepSeek
23101424
061218243036
Exact solution-nature counts. Each model contributes 36 solutions and the full sample contains 252 solutions.
ModelParam-tuneTraining-signal / data engineeringStructural-swapComposition-stackingSearch-hardcodeEvaluation-hackingOtherNovel-approachTotal
Claude-Opus-4.724521310036
GLM-5.233620210136
GPT-5.523712381036
Gemini-3.1-Pro331213401036
Kimi-K2.7-Code34715330136
LongCat-2.042916121136
DeepSeek-V4-Pro231014214036
All models192256111181673252

  1. Agents improve artifacts primarily by composing established techniques, while genuine novelty is rare.

    Composition-stacking is the largest category for every model, accounting for 111 of 252 solutions (44.0%). Only three solutions (1.2%) remain novel after manual review, while 16 (6.3%) exploit evaluation-specific shortcuts.

  2. Novel approaches do not concentrate in the highest-performing models and arise through task-specific reframing rather than new technical primitives.

    The three validated cases come from GLM, Kimi, and LongCat rather than the top-ranked models. Each reframes a task with familiar components instead of inventing a new technical primitive.

07

Discussion

The limitations of current agents cannot be addressed through a single optimization strategy. Different failure patterns require corresponding changes to model training, inference-time strategies, long-horizon system design, or the evaluation objective itself.

  1. What Training Can Improve

    Training should target the bottleneck revealed by each model and task category. Because Execution is already strong and tightly clustered, larger opportunities lie in Solution Framing and Feedback Control. Process diagnostics can guide targeted training data, process rewards, and curricula, while positive and negative transfer cases can teach models when prior experience applies and when it should be reconsidered.

  2. What Inference-Time Search Can Recover

    Inference-time strategies can improve how reliably models realize their existing capabilities. Diverse rollouts, verifier-backed selection, branching from promising checkpoints, and early termination of repeatedly failing or stagnant trajectories could allocate compute more effectively. Process diagnostics can guide when to explore, implement, or recover, while trajectory selection should consider execution validity and progress retention alongside final reward.

  3. What Memory and Harness Design Can Stabilize

    Memory must selectively retrieve, validate, revise, and remove experience rather than simply retain more context. Harness stability likely reflects mechanisms such as error recovery, task management, and best-state protection; native harnesses mainly improve reliability, while automated optimization points toward task-specific and model-adaptive harnesses.

  4. What Requires New Objectives and Benchmarks

    When reward captures task performance but not methodological quality, optimizing it more aggressively may reinforce shortcut-seeking. Progress toward open-ended research requires tasks and feedback that reward not only performance, but also novelty, validity, and generality.

Citation

Cite our paper

BibTeX

@misc{li2026finalscoressystematicevaluation,
      title={Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development},
      author={Yiwei Li and Wanli Yang and Hexiang Tan and Xiangzhou Huang and Zhengyu Chen and Ziran Li and Borun Chen and Shanglin Lei and Huaisheng Zhu and Hao Tian and Fei Sun and Xunliang Cai and Jingang Wang},
      year={2026},
      eprint={2608.13417},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.13417},
}