Agent Evals in CI: From Product Failures to Release Gates

An agent can return a polished answer while failing the actual task. It can claim a refund was issued when no database row changed, say a bug was fixed while tests still fail, or reach the right answer through an unsafe tool path that should never ship.

That is why agent quality cannot be reduced to checking the final message. A production evaluation needs to test the task, the environment, the resulting state, and enough of the execution trace to explain what happened.

The current provider guidance converges on this system-level view. Anthropic's January 2026 guide, Demystifying evals for AI agents, defines tasks, trials, graders, transcripts, outcomes, and evaluation harnesses as separate parts of the test system. OpenAI's May 2026 engineering case study, Building self-improving tax agents with Codex, describes a practical loop where practitioner corrections become product traces, traces become evals, and the resulting failures become concrete improvement targets. Google's current Agents CLI evaluation guide separates trace generation from grading and includes metrics for tool use, multi-turn trajectories, task success, grounding, hallucination, and safety.

The implementation details vary, but the durable pattern is the same: define success as evidence, capture realistic runs, grade the right layer, and block releases when a meaningful regression is outside tolerance.

Treat the Agent as a System Under Test

When you evaluate an agent, you are not measuring only the model. You are measuring a system that includes:

• the model and its sampling configuration
• system and developer instructions
• tool definitions and implementations
• the agent harness and turn loop
• memory and context-selection rules
• retries, budgets, and stopping conditions
• the environment the agent can inspect or modify
• guardrails, approvals, and policy enforcement

A model upgrade can improve reasoning while a tool-schema change makes the overall agent worse. A prompt can look identical while a larger retry budget changes task success and cost. A faster runtime can introduce shared-state contamination that inflates scores.

Record the full configuration with each run. At minimum, keep the model identifier, prompt or policy version, tool-registry version, harness commit, dataset version, grader version, environment image, resource limits, and trial seed when the provider exposes one.

{
  "agentVersion": "support-refunds@8f3c90a",
  "datasetVersion": "refund-regressions@2026-08-20",
  "model": "provider/model-version",
  "toolRegistryVersion": "refund-tools-v4",
  "environmentImage": "support-sim:2026-08-18",
  "maxTurns": 18,
  "maxToolCalls": 24,
  "graderVersion": "refund-policy-rubric-v6"
}

Without that provenance, a score change is only a number. You cannot tell whether the agent improved or the measuring instrument changed.

Start With Product Failures, Not Benchmark Trivia

The highest-value first dataset is usually already in your issue tracker, support queue, review history, and production traces.

Turn each meaningful failure into a task with five parts:

1. Initial state — the files, database rows, messages, browser state, or mock services available before the run.
2. User request — the exact task the agent receives.
3. Allowed capabilities — the tools, credentials, budget, and policy constraints for the trial.
4. Success criteria — observable conditions that must be true after the run.
5. Forbidden outcomes — side effects, policy violations, or regressions that fail the task regardless of final-answer quality.

For example, do not define a refund task as “the response should say the refund was issued.” Define it as:

id: refund-duplicate-request
initial_state:
  order_id: order_482
  existing_refund: refund_901
request: Refund order_482 and tell me when it is complete.
must:
  - refund_count(order_482) == 1
  - final_response_mentions_existing_refund == true
must_not:
  - create_new_refund_called == true
  - unrelated_order_rows_changed == true
budgets:
  max_turns: 12
  max_tool_calls: 10

This task measures the outcome and replay safety. It does not reward a convincing story about success.

Keep a known-good reference run for every new task. If the task author cannot complete it under the same rules, the test may be impossible or underspecified. Anthropic notes that an all-fail result across many trials often points to a broken task or grader rather than proof that every tested agent lacks the capability.

Build Two Suites: Capability and Regression

One dataset cannot answer every release question.

Capability suite

A capability suite asks what the agent can do now. It should contain difficult, representative tasks with room for improvement. A low initial score is acceptable when the task is valid and useful.

Use it to compare:

• new models or reasoning settings
• prompt and tool-design changes
• larger context or resource budgets
• alternative planning and recovery strategies

Regression suite

A regression suite asks whether the agent still handles behavior the product already depends on. These tasks should be stable and should pass at a very high rate.

Every confirmed production failure should become a regression candidate after the fix is verified. Over time, capability tasks that become reliable can graduate into the regression suite.

This separation prevents a common mistake: blocking every release because an experimental capability is not solved, or accepting a release because an average score hides a newly broken core workflow.

Use the Simplest Credible Grader

Agent evals work best with several narrow graders instead of one omniscient score.

Prefer deterministic checks for facts the system can inspect directly. If the requirement is “the repository builds,” run the build. If the requirement is “one refund exists,” query the test database. Do not ask another model to guess.

Use an LLM-as-a-judge when quality is genuinely semantic. Give it a narrow rubric with observable anchors, a bounded scale, and an explicit abstain path. Hide irrelevant metadata and randomize comparison order when judging two candidates.

Score groundedness from 0 to 2.

2: Every material claim is supported by the provided tool results.
1: The conclusion is supported, but one minor claim lacks evidence.
0: A material claim conflicts with or is absent from the evidence.

Return JSON with score, evidence citations, and a one-sentence reason.
If the supplied evidence is insufficient to grade, return abstain: true.

Calibrate model-based graders against human decisions before trusting them as release gates. Recheck calibration after changing the judge model, rubric, or task distribution. OpenAI's Graders API reference reflects the same separation between deterministic string checks, similarity metrics, and score-model graders.

Grade Outcomes First, Then Inspect Trajectories

The final state is usually the strongest signal. Grade it first.

Trajectory checks are still valuable when the path itself carries cost or risk. Useful questions include:

• Did the agent call a destructive tool without the required approval?
• Did it repeatedly call the same failing tool?
• Did it retrieve evidence before making a factual claim?
• Did it recover from a transient error within budget?
• Did it contact a forbidden service or inspect a disallowed path?

Avoid requiring one exact sequence of tool calls unless the sequence is a policy requirement. Capable agents may discover valid paths the task author did not anticipate. Brittle trajectory matching can score creativity as failure.

OpenAI's Agents SDK tracing documentation records model generations, tool calls, handoffs, guardrails, and custom events, and supports custom processors for another trace backend. It also warns that traces can contain sensitive model and tool inputs and outputs. Redact secrets and personal data before retention, and keep a documented retention policy for eval artifacts.

Run Repeated Trials and Report Reliability

Model behavior varies. One pass proves possibility, not reliability.

Run multiple independent trials for important tasks. Report both the average task score and the proportion of tasks that meet the required reliability threshold. For a critical workflow, “passed once in five attempts” is usually a failure even if a best-of-five demo looks impressive.

A small summary can expose variance:

suite: refund-regression
tasks: 42
trials_per_task: 5
task_success_rate: 96.2%
tasks_passing_5_of_5: 88.1%
policy_violations: 0
p95_latency_seconds: 14.8
mean_cost_per_task_usd: 0.034

Choose the number of trials based on impact, variance, and budget. Cheap deterministic regressions can run often. Expensive open-ended capability tasks may run nightly or before model migrations.

Never share mutable state between trials. Reset databases, filesystems, caches, queues, clocks, and external-service fixtures. Anthropic's February 2026 analysis, Quantifying infrastructure noise in agentic coding evals, shows why resource configuration must be treated as an experimental variable rather than background detail.

Prevent Evaluation Contamination

An agent with browsing, repository history, or broad filesystem access may find an answer key instead of solving the task. That can produce a high score with no corresponding product capability.

Protect the suite:

• keep private tasks and grader logic outside the agent workspace
• prevent test fixtures from leaking through logs, Git history, caches, or tool descriptions
• separate development examples from held-out release tasks
• rotate or expand tasks when memorization becomes plausible
• record every network destination and retrieved artifact during the trial
• reject runs that access prohibited answer sources

OpenAI's May 2026 playbook for trustworthy third-party evaluations emphasizes contamination, environment state, tool access, harness choices, and reporting changes to the evaluation setup. Those concerns apply just as strongly to an internal release suite.

Turn the Suite Into a CI Release Gate

The CI gate should compare a candidate against a pinned baseline and apply explicit policies. Do not gate on a single blended score.

agent_eval:
  regression:
    min_task_success_rate: 0.97
    min_all_trials_pass_rate: 0.90
    max_policy_violations: 0
    max_relative_cost_increase: 0.15
    max_relative_p95_latency_increase: 0.20
  capability:
    allowed_regression: 0.02
  grader_health:
    min_human_agreement: 0.85
    max_abstain_rate: 0.05

Use confidence intervals or a minimum task count before treating tiny score differences as real. When the candidate fails, preserve the task ID, configuration, trace, grader outputs, final state, and diff from baseline so the result is actionable.

A practical pipeline is:

1. Validate task and rubric schemas.
2. Provision one clean environment per trial.
3. Generate traces with pinned agent configuration.
4. Run deterministic outcome and policy graders.
5. Run model-based graders only where needed.
6. Aggregate reliability, cost, latency, and violation metrics.
7. Compare the candidate with the pinned baseline.
8. Publish a reviewable failure report.
9. Block promotion when a required threshold fails.

Google's Agents CLI follows a similar generate-then-grade split. That boundary is useful even if you build your own harness: execution failures and grader failures remain distinguishable, and you can regrade stored traces without paying to rerun the agent.

Keep the Evaluation Assets Portable

Store tasks, environment setup, deterministic graders, rubrics, thresholds, and result schemas in your repository or another versioned system you control. Provider platforms can still supply tracing dashboards, hosted graders, or managed execution, but they should not be the only copy of your quality contract.

This matters in 2026 because product surfaces are moving quickly. OpenAI's updated AgentKit announcement says its hosted Agent Builder and Evals products will wind down on November 30, 2026, and recommends code-based Agents SDK workflows or Workspace Agents depending on the use case. A portable suite lets you change the runner, judge model, or observability backend without losing the definition of “good.”

Provider-neutral assets also make model migrations safer. Run the same tasks through the current production agent and the candidate. Then compare task reliability, failure classes, cost, latency, and policy compliance instead of relying on a general benchmark or a handful of demos.

Review Failures as Product Evidence

An eval report is not finished when it prints a score. Sample passing traces, and read every new high-impact failure.

Classify failures into actionable buckets:

• task or fixture bug
• grader bug or disagreement
• environment or infrastructure failure
• model or prompt behavior regression
• tool-contract or integration failure
• context-selection failure
• policy or approval failure
• budget exhaustion

Fix the measuring system when the task or grader is wrong. Fix the product when the failure is real. Preserve both decisions in review notes so the team does not silently weaken a gate to make a release green.

Release Checklist

Before an agent change can promote, confirm:

• The task suite includes recent real-world failures and important negative cases.
• Each task has a clean initial state, clear success conditions, and a known-good solution.
• Capability and regression suites are reported separately.
• Deterministic graders own verifiable outcomes and policy rules.
• Model-based graders are calibrated against human decisions.
• Important tasks run multiple independent trials.
• The environment, model, harness, tools, dataset, and graders are versioned.
• Traces are useful for debugging and scrubbed of sensitive data.
• Held-out tasks and grader logic are not visible to the agent.
• CI thresholds cover reliability, violations, cost, and latency instead of one average score.
• Candidate results are compared with a pinned production baseline.
• Failures preserve enough evidence for a developer to reproduce them.

The Durable Principle

An agent eval is a product contract expressed as executable evidence.

Start with the state users need changed, not the sentence you hope the model will say. Use real failures to build tasks. Grade outcomes with code whenever possible, trajectories when the path matters, and subjective quality with calibrated judgment. Repeat trials until the suite measures reliability rather than luck. Then put the result in CI, where it can stop a regression before users discover it for you.