AutoResearch Eval

Agent trajectories

Beyond final scores:the trajectories behind them.

01

Model Development

Moving MNIST World Model

moving_mnist_world_model

Train a PyTorch video world model from scratch on deterministic Moving MNIST clips. The model receives ten 64×64 grayscale context frames and must generate the next ten frames autoregressively; the hidden test uses fresh clips from the same generator.

Objective
Validation PSNR · higher is better
Workload
1,000 validation clips; 10 context frames → 10 autoregressive target frames at 64×64.
Correctness
The checkpoint must reconstruct through build_model, emit the required 10-frame tensor, and never condition on future ground-truth frames.
Budget
4 hours
moving_mnist_world_model model trajectory comparisonRunning-best AutoLab reward for seven models, computed by applying the task's official scoring rule to trajectory evaluator observations. Rewards are clipped to the zero-to-one range and shown on a linear vertical axis. Focus or select an interactive line to inspect or open its full trajectory.1.000.750.500.250.000%25%50%75%100%Relative progressAutoLab reward ↑

What to notice

01

Claude reaches 18.60 dB in six validated checkpoints; DeepSeek and GLM finish close together at 17.25 and 17.27 dB despite differently sized loops.

02

Kimi rises from 13.96 to 15.53 dB, with its strongest advance appearing after the move to scheduled sampling and input noise.

Messages across runs
178–1,387
Evaluator checkpoints
2–10
02

System Optimization

BM25 Search in Go

bm25_search_go

Optimize a standard-library-only Go search engine over a deterministic synthetic corpus. Query execution must preserve the exact BM25 top-10 results and checksum while reducing end-to-end runtime.

Objective
Query runtime · lower is better
Workload
40 queries producing 400 exact top-10 hits in total.
Correctness
The hit count and correctness checksum must match exactly; any wrong result receives no score.
Budget
4 hours
bm25_search_go model trajectory comparisonRunning-best AutoLab reward for seven models, computed by applying the task's official scoring rule to trajectory evaluator observations. Rewards are clipped to the zero-to-one range and shown on a linear vertical axis. Focus or select an interactive line to inspect or open its full trajectory.1.000.750.500.250.000%25%50%75%100%Relative progressAutoLab reward ↑

What to notice

01

Six of seven selected runs reach a sub-millisecond best runtime; DeepSeek's selected run stops at 32.75 ms.

02

Checkpoint density ranges from 19 for GPT to 1,268 for GLM, so progress appears in both compact searches and extended tuning loops.

Messages across runs
258–2,334
Evaluator checkpoints
19–1,268
03

System Optimization

Regex Engine

regex_engine

Optimize a pure-Rust regex engine for a fixed collection of patterns and diverse HTTP paths, logs, identifiers, and emails. Pattern compilation and matching are both included in the measured runtime.

Objective
Compile + search runtime · lower is better
Workload
23 patterns compiled and searched over 100,000 haystacks.
Correctness
Every result must exactly match the reference engine across the supported regex syntax.
Budget
2 hours
regex_engine model trajectory comparisonRunning-best AutoLab reward for seven models, computed by applying the task's official scoring rule to trajectory evaluator observations. Rewards are clipped to the zero-to-one range and shown on a linear vertical axis. Focus or select an interactive line to inspect or open its full trajectory.1.000.750.500.250.000%25%50%75%100%Relative progressAutoLab reward ↑

What to notice

01

GPT and Claude reduce their selected best runtime to 12.95 ms and 18.30 ms, while DeepSeek remains at 547.62 ms.

02

GLM's 1,584-message trajectory reaches 84.41 ms but trails several shorter searches, exposing strategy rather than persistence as the bottleneck.

Messages across runs
240–1,584
Evaluator checkpoints
7–253
04

Puzzle & Challenge

Adaptive Compression

adaptive_compression

Build an online byte predictor that adapts across nine hidden sequence families, from Markov and periodic sources to nested structures, recurrences, regime switches, run lengths, and random data.

Objective
Byte-weighted overall bpb · lower is better
Workload
9 sequence families × 34,830 bytes = 313,470 bytes.
Correctness
Every prediction must be a valid 256-value probability distribution; an invalid sequence is scored as 8.0 bpb.
Budget
4 hours
adaptive_compression model trajectory comparisonRunning-best AutoLab reward for seven models, computed by applying the task's official scoring rule to trajectory evaluator observations. Rewards are clipped to the zero-to-one range and shown on a linear vertical axis. Focus or select an interactive line to inspect or open its full trajectory.1.000.750.500.250.000%25%50%75%100%Relative progressAutoLab reward ↑

What to notice

01

The selected runs begin near 5.34 bpb (Kimi begins at 5.03) and separate to best scores from GLM's 3.55 bpb to Kimi's 4.04 bpb.

02

DeepSeek uses 1,710 messages and 83 checkpoints to reach 3.86 bpb; Gemini reaches 3.69 bpb in 118 messages and 12 checkpoints.

Messages across runs
118–1,710
Evaluator checkpoints
11–83
05

CUDA

ICP Correspondence Step

icp_correspondence_step_cuda

Optimize one CUDA Iterative Closest Point correspondence step: find each source point's nearest target neighbor, reject distant pairs, and accumulate covariance, error, and count entirely on the GPU.

Objective
Median kernel runtime · lower is better
Workload
N = 200,000 source points; M = 500,000 target points; d_max = 0.05; median after warmup.
Correctness
The pair count must match exactly and covariance/error outputs must match the CPU reference within 1e-4 relative tolerance.
Budget
2 hours
icp_correspondence_step_cuda model trajectory comparisonRunning-best AutoLab reward for seven models, computed by applying the task's official scoring rule to trajectory evaluator observations. Rewards are clipped to the zero-to-one range and shown on a linear vertical axis. Focus or select an interactive line to inspect or open its full trajectory.1.000.750.500.250.000%25%50%75%100%Relative progressAutoLab reward ↑

What to notice

01

Claude's selected trajectory reaches 0.0380 ms, more than 3× faster than the next-best Gemini at 0.1458 ms.

02

GLM, DeepSeek, Kimi, and LongCat use 123–237 checkpoints yet cluster near 0.17–0.23 ms, despite widely different loop lengths.

Messages across runs
235–1,290
Evaluator checkpoints
21–497