AI integration benchmark for real API work

Your task

Resolve the engineering ticket in PROBLEM.md by producing durable integration software in the starter repository. Gather context from the running vendor services before implementing anything.

The vendor services are mandatory. A solution that never puts signed requests on the wire cannot pass, and the services must be reachable from the connector before work begins.

Ticket: Candidate sync against StaffLine drops changes after the first backfill.

Context gathering strategy

Treat the ticket as the requirement, not the wire contract. The contract lives in the vendor’s served documentation and in the responses the vendor actually returns. Documentation in this suite is truthful, stale, or sparse, so any rule the ticket leaves implicit has to be recovered by reading the docs and observing behavior.

Protocol usage

No MCP servers are exposed in this environment. The vendor APIs are the tools. If a harness does front them with MCP, treat it as another transport and keep the wire contract identical.

1. Read the service endpoints and synthetic credentials from the environment and from declared inputs.
2. Fetch each vendor’s participant documentation, served from the same image as the API.
3. Authenticate every call. Expect static or rotating credentials, OAuth-like token exchange, and HMAC request signatures.
4. Walk collections with the pagination style the vendor advertises, not an assumed one.
5. Persist progress in the vendor’s clock domain, from values the vendor returned.
6. Make writes idempotent, and honor advertised rate limits, retry rules, and optimistic concurrency.
7. Survive restarts, transient server errors, stale reads, dropped or duplicated events, and cursor or token expiry.

# discover the wire contract
curl -s "$STAFFLINE_URL/_docs/openapi.yaml" -o openapi.yaml

# signed request, then read the vendor's own progress fields
TS=$(date +%s)
SIG=$(printf '%s%s' "$TS" "$REQ_PATH" \
  | openssl dgst -sha256 -hmac "$STAFFLINE_SECRET" -r | cut -d' ' -f1)
curl -s "$STAFFLINE_URL$REQ_PATH" \
  -H "X-Timestamp: $TS" -H "X-Signature: $SIG" -o page.json

jq '{next: .next_cursor, watermark: ([.items[].mod_ts] | max)}' page.json

Available resources

Inside the participant workspace:

• PROBLEM.md for the ticket.
• repo/ for the starter repository you edit.
• materials/ and inputs/ for task-specific documents and seed data.
• Endpoints, synthetic credentials, and auth details in the environment.
• Vendor documentation at /_docs/, with OpenAPI at /_docs/openapi.yaml when the vendor publishes it.

Outside it, and unavailable to you: task.yaml, verifier/, authoring/, fixtures, the gold patch, vendor source, and the fault schedule.

Vendor-specific instructions

Only the services in this task’s stack exist. There is no shared world, and no vendor here has a public counterpart to recall.

StaffLine

Candidate system of record. Collection reads, an incremental feed carrying modification timestamps, and tombstones for deletes. HMAC signatures on every request. Documentation at $STAFFLINE_URL/_docs/. Its deterministic clock trails the worker’s wall clock.

Canonical store

The connector’s own durable state, read directly by grading. Records must match StaffLine after the backfill, later updates, and deletions.

Solution requirements

Gather the vendor’s state and behavior first, then implement. The connector must run end to end from a cold start: backfill, persist a watermark, and pick up later changes on the next poll without duplicating or dropping records.

Critical: read the vendor before coding

The ticket is a symptom report, not a full brief. Before writing the fix, establish from the docs and observed responses:

1. The entrypoint the harness runs, and its runtime contract.
2. Input sources, output paths, and output schema.
3. Ports and service configuration.
4. Pagination, auth, and retry semantics.
5. Which timestamps are vendor-owned and which are local.
6. Acceptance behavior across a restart.

Do not guess a requirement the vendor can answer.

Naming rule

Keep the entrypoint the starter repository declares, at the path PROBLEM.md specifies. If neither names one, leave the starter entrypoint in place and make it executable.

Final deliverable

A working connector in the captured workspace. Grading restarts the vendor stack from a checkpoint and runs your artifact against it, so a process you left running, exploration notes, plans, or hand-written output files do not count as completion. Any failed mandatory check gates Task Score to zero.