AutoResearch Eval

Beyond
Final Scores

A systematic evaluation of agents for long-horizon AI research and development.

The path reveals what the score hides.

BlogPaper (opens in a new tab)Trajectories
Same outcome, different processTwo research runs begin with the same artifact and reach nearly the same final score. One advances directly. The other regresses, uses feedback, and recovers.
Artifact
Run A
Run B
Solution Framing
Execution
Feedback Control
Final score
frontier models
7
long-horizon tasks
36
baseline trajectories
756

Peak performance is not reliability.

Many models can find a strong solution once. Reaching it consistently is what separates them.

0.237avg@3 spread
0.122best@3 spread

Average and best across three runs

Overall AutoLab reward

The peak-to-average gap is a proxy for run-to-run consistency. A longer connector means the strongest result was harder to reproduce across three runs.
Overall avg@3 and best@3 results for seven models
Modelavg@3best@3Gap
Claude-Opus-4.70.7390.7900.051
GLM-5.20.6820.7570.075
GPT-5.50.6630.7720.109
Gemini-3.1-Pro0.6520.7500.098
Kimi-K2.7-Code0.5870.7290.142
LongCat-2.00.5720.6740.102
DeepSeek-V4-Pro0.5020.6680.166

Look inside the loop.

Similar outcomes can hide different choices, implementation quality, and recovery behavior.

Three measurements around an evolving artifactSolution Framing measures how quickly a run reaches a strong direction. Execution checks whether proposed changes run and pass task checks. Feedback Control measures whether a run preserves gains and recovers after regressions. The accompanying charts compare all seven models on zoomed, explicitly labelled score ranges.

Solution Framing

Choose a direction that can meaningfully improve the artifact.

Solution Framing exampleA reward curve rises quickly and reaches a high plateau, rewarding both solution quality and how early the direction was found.high reward, reached fast

Execution

Turn the chosen direction into working, verified changes.

Execution exampleA promising sequence ends in failed delivery, showing that a good direction still needs correct execution.delivery fails

Feedback Control

Keep real gains, catch regressions, and recover when needed.

Feedback Control exampleReward falls after a failed try, then recovers toward the best result seen earlier.best-so-farfailed try, then recovery
Process scores across seven models
ModelSolution FramingExecutionFeedback Control
Claude-Opus-4.70.6120.9670.920
GLM-5.20.5390.9370.911
GPT-5.50.5550.9580.858
Gemini-3.1-Pro0.5550.8890.921
Kimi-K2.7-Code0.4730.8800.875
LongCat-2.00.4780.8880.928
DeepSeek-V4-Pro0.5190.8880.772

The path is shaped by what surrounds it.

Experience can help or mislead. Harnesses mainly make existing capability more reliable.

Six of seven models benefit from retained intra-task experience, while transfer across tasks remains unstable.

Experience gains and harness reliability

Signed reward change from retained experience, followed by the peak-to-average gap.

Intra-Task Experience Reuse

6 of 7 positive
Claude+0.036
GLM-5.2+0.079
GPT-5.5+0.073
Gemini+0.128
Kimi-0.013
LongCat+0.145
DeepSeek+0.089

Inter-Task Experience Reuse

5 of 7 positive
Claude+0.001
GLM-5.2+0.040
GPT-5.5+0.063
Gemini-0.017
Kimi+0.021
LongCat-0.021
DeepSeek+0.093

Harness reliability

Mean peak-to-average gap for GPT-5.5 and Kimi-K2.7-Code. Lower is more consistent.

0.125
Claude Code
0.089
Model-native
Retained experience helps six models in the intra-task setting and five in the inter-task setting. For GPT-5.5 and Kimi-K2.7-Code, model-native harnesses narrow the mean peak-to-average gap from 0.125 to 0.089.
Experience gains and harness reliability gap
ModelIntra-task experience gainInter-task experience gain
Claude-Opus-4.70.03620.001
GLM-5.20.07900.040
GPT-5.50.07300.063
Gemini-3.1-Pro0.1280-0.017
Kimi-K2.7-Code-0.01270.021
LongCat-2.00.1454-0.021
DeepSeek-V4-Pro0.08900.093
GPT-5.5 and Kimi-K2.7-Code, Claude Code mean peak-to-average gap0.125
GPT-5.5 and Kimi-K2.7-Code, model-native mean peak-to-average gap0.089

Improvement rarely means invention.

Agents usually recombine established techniques. Only three solutions remain novel after manual review.

Solution nature across 252 solutions

Most solutions reuse or recombine established techniques. Three task-specific approaches remain novel after fixed-rubric classification and manual review.
Solution nature among 252 solutions
ModelTotalParameter tuningTraining signal / data engineeringStructural swapComposition stackingSearch and hardcodeEvaluation hackingOtherValidated novel approaches
Claude-Opus-4.736245213100
GLM-5.236336202101
GPT-5.536237123810
Gemini-3.1-Pro363312134010
Kimi-K2.7-Code36347153301
LongCat-2.036429161211
DeepSeek-V4-Pro362310142140
Total252192256111181673

Current agents operate more like engineering optimizers than fully autonomous researchers.

BlogPaper (opens in a new tab)Trajectories

Cite this work

Yiwei Li, Wanli Yang, Hexiang Tan, et al. (2026).Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development.

Join LongCat

Researcher, LLM Self-Improvement and Automated Research Agents

Campus hiring in Beijing and Shanghai.

View open position (opens in a new tab)