Introducing ATS-Bench: grading agents on what the system says, not what the code looks like
Our first public benchmark evaluates agent integration engineering against a live applicant tracking system, with webhook-verified grading and a 200-task private holdout.
Today we are publishing ATS-Bench, a benchmark for agentic integration engineering against a live applicant tracking system. Fifty tasks are public with a reproducible harness; two hundred more are held out privately for evaluation that has not leaked into any training corpus.
The premise
Coding benchmarks overwhelmingly grade artifacts: does the patch apply, do the unit tests pass. Integration work does not live there. It lives in whether the data actually arrived, whether the webhook actually fired, whether the system on the other side actually ended up in the right state after a fault.
ATS-Bench grades system state. Every task runs against a containerized ATS simulator with real API surface area, and the evaluator observes what the integration did to the system rather than inspecting the code that did it. Agents must discover live state; pattern-matching the documentation is not enough.
Four task categories
- Build: stand up a working integration from a requirements brief.
- Fix: diagnose and repair a deliberately broken integration.
- Harden: keep the pipeline correct under injected faults and rate limits.
- Migrate: move data across schema versions without loss or duplication.
Scoring is weighted across rubric criteria covering functional correctness, reliability, and data integrity, normalized to 100.
What we are seeing so far
Early runs show a consistent shape: models scaffold confidently, then stall where correctness requires reconciling the system’s actual state against intent. Completion drops sharply in the harden and migrate categories. The full leaderboard and failure analysis publish with the release; preliminary numbers are on the benchmark page.
The holdout policy
The 50-task public cut is for reproducibility and community use. The 200-task private set never publishes and is never seeded into training data, a separation enforced in our task registry rather than by convention. Labs can license the holdout for clean evaluation, shared or exclusive.
If you want early access, sample packets, or the harness wired into your eval stack, request samples.