Risk-Calibrated AI Code Review: From Agent PRs to Resolved Findings

AI can review more code than a person can inspect line by line, but volume is not the same as assurance. An automated reviewer can find a real boundary bug and still miss the broken workflow around it. It can also produce a plausible comment that costs more to investigate than the change itself.

The useful operating model is risk-calibrated review: decide how much scrutiny a change deserves, give the reviewer only the context needed to judge it, require evidence for findings, and keep acceptance or merge authority outside the reviewing agent.

The tools are moving in that direction. GitHub made Lite and Balanced review effort levels generally available in August 2026, explicitly tying review depth to change complexity and sensitivity. Later that month, Copilot code review expanded to bot-authored and very large pull requests and added resolution reasons for comments. GitHub also supports agent skills and read-only MCP context in code review.

Those capabilities improve coverage. They do not remove the need to decide what deserves deeper review, validate every material finding, or require an independent owner for the final merge decision.

Start With a Change-Risk Contract

Do not choose review depth from lines changed alone. Classify the change by what can go wrong.

Increase scrutiny when the author is an agent or bot, the diff is unusually large, generated files hide the meaningful change, tests are weak, or the repository instructions are incomplete. These signals do not prove the change is unsafe. They tell you that a shallow review is unlikely to be enough.

A compact review contract should name:

• intended behavior and explicit non-goals
• affected users, data, systems, and trust boundaries
• required checks and evidence
• rollback or recovery path
• reviewer depth and required human owners
• conditions that block merge

Without that contract, the reviewer is forced to infer product intent from the diff. That is where confident but irrelevant feedback begins.

Bound the Review Context

Repository context improves review only when it is relevant and trustworthy.

Start with the issue or specification, the diff, nearby tests, repository instructions, and the interfaces the change crosses. Add external documentation or service-catalog context only when it resolves a concrete question. GitHub's code review documentation describes how repository instructions, skills, and MCP servers can extend the reviewer's context, but it also warns that AI feedback still needs validation.

Treat retrieved context as evidence, not authority. An issue body, documentation page, tool result, or MCP response may be stale, incomplete, or maliciously written. It must not silently override the review contract or grant permission to mutate an external system.

For review integrations:

• prefer read-only tools
• expose the smallest relevant dataset
• exclude secrets, private source, customer data, and unrelated conversations
• record which instructions, skills, and external sources were used
• fail closed when required context is missing

GitHub's current agentic review surface limits MCP tools to read-only use. That is a sound default: a reviewer should inspect evidence, not repair the code or resolve its own comments while it is still judging the change.

Review in Layers

One pass should not pretend to answer every question. Use layers that move from objective evidence toward broader judgment.

1. Contract and Scope

Confirm the diff implements the stated outcome without adding unrelated behavior. Flag missing acceptance criteria, hidden generated changes, undocumented migrations, and scope that spans multiple independent releases.

2. Correctness and Failure Paths

Trace inputs, state transitions, outputs, errors, retries, timeouts, and cleanup. Look for invalid assumptions at boundaries: null or empty input, partial writes, stale state, duplicate delivery, concurrency, and downstream failure.

3. Security and Privacy

Inspect trust boundaries, authorization checks, credential exposure, injection paths, logging, data retention, and destructive operations. A prompt instruction is not an enforceable security control; verify the host, policy, database, and deployment configuration that actually restrict behavior.

4. Tests and Release Evidence

Check whether tests prove the changed behavior rather than only executing the new code. Match every material risk to a test, static check, browser verification, migration read-back, preview, or production gate.

5. Maintainability

Review naming, duplication, documentation, performance, and operability after correctness and safety. Avoid drowning a blocking defect in cosmetic suggestions.

Make Every Finding Adjudicable

A useful finding can be independently checked. It includes:

1. Location — the smallest affected file, line, component, or boundary.
2. Failure scenario — the concrete input or sequence that triggers the problem.
3. Impact — what a user, operator, system, or release will experience.
4. Evidence — code path, test result, contract, or authoritative documentation.
5. Smallest credible fix — a bounded correction, not a speculative redesign.
6. Verification — how to prove the correction works.

Use severity to describe impact and likelihood, not writing style. Separate blockers from suggestions and questions.

Then adjudicate each comment explicitly:

• Addressed — the change or evidence resolves the finding.
• Won't fix — the risk is accepted, out of scope, or intentionally deferred; record the owner and rationale.
• Incorrect — the finding rests on a false assumption; cite the evidence that disproves it.
• Needs evidence — the claim may be valid, but the reviewer cannot decide with the available context.

GitHub's August 27 release added Addressed, Won't fix, and Incorrect as Copilot comment resolution reasons. Use the same discipline even when the review tool does not expose structured reasons. Silent resolution destroys feedback about false positives, recurring defects, and accepted risk.

Re-Review the Change That Will Merge

The first review applies to one commit. New pushes can repair a finding while introducing another defect.

After material changes:

1. refresh the diff against the intended base branch
2. rerun the focused checks tied to the findings
3. request another review at the required depth
4. confirm every blocking thread has evidence-backed adjudication
5. verify approvals and status checks apply to the final head commit

GitHub documents an option to automatically request re-review on new pushes. Automation is useful, but it should reinforce a branch policy rather than become the only merge gate.

Also audit defaults instead of assuming a platform feature enabled them. In August 2026, GitHub stopped Code Quality from automatically adding Copilot as a reviewer, leaving teams to configure automatic review intentionally. A missing review can look exactly like a clean review if nobody checks the timeline.

Use a Second Opinion Carefully

A second model or reviewer is most valuable when it uses a different lens, not when it repeats the first prompt.

Ask the second pass to challenge:

• the highest-risk assumptions
• boundary and failure-path coverage
• evidence behind blocking findings
• reasons a finding may be a false positive
• gaps in tests and release verification

The August VS Code agent-session release introduced an experimental /rubber-duck second-opinion command that asks a complementary model to surface missed details and edge cases. Whether the second pass is automated or human, disagreement is a signal to inspect evidence, not a vote to average.

Measure Review Quality, Not Comment Volume

Useful review metrics include:

• confirmed blocking findings by risk class
• false-positive or Incorrect resolution rate
• findings reopened after a new push
• defects found after merge that review should have caught
• time from finding to evidence-backed resolution
• percentage of high-risk changes with the required depth and owner approval

Do not optimize for number of comments, acceptance rate, or a single quality score. A reviewer that creates many low-value comments can slow delivery while making risk harder to see.

Production Checklist

• The change has an explicit outcome, non-goals, and risk classification.
• Review depth matches security, data, operational, and cross-system risk.
• Context is relevant, provenance-aware, and read-only where possible.
• Findings include a failure scenario, impact, evidence, fix, and verification.
• Blocking findings are separated from suggestions and questions.
• Every resolved comment has an explicit reason and supporting evidence.
• Material pushes trigger focused checks and re-review of the final head.
• An independent owner retains acceptance and merge authority.
• High-risk changes have rollback or recovery evidence.
• Review quality is measured by escaped risk and adjudication, not comment count.

The Operating Principle

AI code review is a decision-support system, not a merge authority.

Match depth to risk. Bound the context. Make findings testable. Record why each comment was resolved. Re-review the commit that will actually ship. Then let an accountable owner decide whether the evidence is strong enough to merge.