ATS-Bench
Can agents build, fix, harden, and migrate integrations against a live applicant tracking system? 50 public tasks, 200 private, graded by webhook-verified system state.
Leaderboard
| Rank | Model | Weighted score | Completion | |
|---|---|---|---|---|
| 1 | 
Claude Fable 5 | 84 | 66% | |
| – | 
GPT-5.6 Sol | running | – | |
| – |
Claude Opus 5 | running | – | |
| – | 
Grok 4.6 | running | – |
Preliminary results. Scores are weighted across rubric criteria and normalized to 100. Full leaderboard, failure analysis, and reproducible harness land with the public release.
Task categories
Build a working integration from scratch
Fix a deliberately broken integration
Harden reliability under fault injection
Migrate data across schema versions
Why integration engineering
Integration work is where agent coding claims meet production reality: undocumented API behavior, stateful external systems, and correctness that only shows up when the data actually arrives where it should. ATS-Bench evaluates agents against a live, containerized applicant tracking system simulator, not a frozen repository.
How it is built
Every task runs in its own isolated container fleet. Agents must discover real API state rather than pattern-match documentation, and evaluation is webhook-based: the grader observes what the integration actually did to the system, not what the code looks like. Scoring is weighted across rubric criteria covering functional correctness, reliability under fault injection, and data integrity through migration.
The public cut is 50 tasks with the full harness, open sourced for reproducibility. A further 200 private tasks are held out and never published; they are available to license for evaluation that has not leaked into any training corpus.
Early findings
Frontier models complete scaffolding confidently and stall on stateful verification, with completion rates dropping sharply once tasks require reconciling the system’s actual state against intent. Full failure analysis publishes with the leaderboard.