AI Tooling

Learn the operating system for AI development

Agents, skills, planning, branching, and quality gates explained as one system for building faster without losing project direction.

Operating model

Plan
01

Shape intent into specs, non-goals, and verification criteria.

Context
02

Load repo rules, architecture notes, and task-specific memory before edits.

Execute
03

Use agents and skills on scoped feature branches with focused checks.

Promote
04

Move through dev, test, and prod only after checks and review.

14 topics found

qualitypractice

Quality

Copilot default-policy governance

On October 22, 2026, GitHub's default policy for new Copilot features begins applying to eligible generally available capabilities that enterprises or organizations left unconfigured. GitHub separately governs released-model defaults and now exposes model-policy state through REST endpoints for enterprises, organizations, teams, and agent apps. Treat these defaults as inherited production configuration: inventory every scope, distinguish explicit decisions from delegation, keep preview opt-ins separate, and verify effective availability without assuming one policy field proves a user's access.

copilotpolicygovernanceenterprisemodel-access

Principles

  • Keep feature defaults, model defaults, preview opt-ins, model selection, and end-user access as separate controls with separate evidence.
  • Treat Unconfigured, Delegate to default policy, and delegation to an organization, team, or app as meaningful states rather than synonyms for enabled.
  • Preserve explicit enable or disable decisions and apply a default only to eligible generally available capabilities governed by that scope.
  • Read policy inheritance from the controlling enterprise through its selected management path before judging a lower-level setting or proposing a change.

Steps

  • Record the Copilot plan, management mode, enterprise and organization boundaries, accountable owners, default policies, and evidence timestamp.
  • Inventory features and models by release stage, stored state, controlling scope, applicable default, delegated target, editability, and provider-documented exception.
  • Resolve each effective policy from the enterprise downward, keeping preview choices and explicit exceptions outside the general default calculation.
  • Compare the policy map with a permitted synthetic user's observed availability only when access testing is authorized, without using private prompts or repositories.
  • Review new releases and unconfigured counts on a fixed cadence; propose scoped changes with blast radius, rollback, and verification for separate approval.

Avoid

  • Calling an inherited or delegated state an explicit administrative decision.
  • Assuming the default availability for released models selects the model a user receives in a session.
  • Applying the generally available feature default to preview capabilities or provider-documented exceptions.
  • Reading one REST field or settings page as proof that a particular user can access a capability.
  • Changing a broad enterprise default to fix one organization's exception without reviewing the wider blast radius.
qualitypractice

Quality

Typed AI decision systems

OpenAI released the Decisions API in public beta on October 6, 2026 for fast predicate, fixed-choice, and rubric-score answers over text or inline images. The dedicated endpoint returns probabilities or confidence instead of generated prose, but a typed response is not a production policy: teams still need observable criteria, application-specific labeled examples, cost-sensitive thresholds, explicit fallback and refusal paths, privacy review, and a versioned rollout that can survive beta changes.

decisions-apiclassificationevaluationthresholdsreliability

Principles

  • Use a decision endpoint when the application needs a predicate, one value from a fixed set, or an ordered rubric score; use Structured Outputs or function calling when the application needs a custom object, explanation, or action request.
  • Treat returned probabilities and confidence as model evidence, not as permission or a universal guarantee; choose application thresholds from labeled examples and the real costs of false positives and false negatives.
  • Make fallback categories, refusals, low-confidence review, and unavailable-service behavior explicit parts of the product contract rather than exceptional afterthoughts.
  • Version the model, question wording, choice descriptions, score levels, thresholds, dataset, and downstream action policy together so a changed result can be traced and rolled back.

Steps

  • Define each observable decision, its allowed answer type, stable name, input boundary, excluded use cases, and the least consequential downstream action it may influence.
  • Build a labeled dataset with ordinary, boundary, ambiguous, adversarial, refusal, and subgroup examples; keep a held-out set for threshold and regression decisions.
  • Compare the candidate endpoint with the current rule or model on error costs, review coverage, latency, input-token cost, refusal handling, and operational failure behavior.
  • Set separate accept, reject, and human-review bands from measured outcomes, then log the versioned decision contract and outcome without retaining unnecessary sensitive input.
  • Shadow production traffic, inspect slice-level errors, canary a reversible low-risk action, and promote only when quality and operational gates pass with an owned rollback path.

Avoid

  • Reading a high probability or confidence value as proof that the answer is correct or calibrated for your population.
  • Combining dependent questions in one request even though a later question needs the earlier answer as input.
  • Defining choices without an other or review route when real inputs fall outside the supplied taxonomy.
  • Automatically taking high-impact actions from a beta endpoint without authorization checks, human review, or a deterministic policy layer.
  • Assuming endpoint availability means inference occurs in the requested region or that every image URL and file reference is supported.
qualitypractice

Quality

AI API key lifecycle governance

OpenAI added project-key expiration and maximum-lifetime controls on September 10, 2026, followed by organization and project controls over which kinds of API keys may be created on September 15. Those controls improve the next credential but do not repair the current inventory: creation restrictions apply only to new keys, and an expiration date does not prove that every workload can rotate safely. Govern keys as owned production dependencies with scoped identities, planned overlap, observable cutover, and verified revocation.

api-keyscredentialssecurityrotationgovernance

Principles

  • Separate who may create a key, what the key may access, how long it remains valid, where it is stored, and how its use is observed; each is a different control.
  • Prefer workload-owned service accounts and environment-specific projects over shared personal keys whose ownership changes with a human account.
  • Treat maximum lifetime as a backstop, not a rotation mechanism; replacement, cutover, verification, and revocation still need an operating procedure.
  • Audit existing credentials explicitly because a policy for newly created keys does not shorten, rescope, or revoke keys that already exist.

Steps

  • Inventory each key by provider, project, environment, owner, workload, permissions, storage location, creation date, expiration, last use, and emergency contact without copying secret values.
  • Set organization defaults for allowed key identities and maximum lifetime, then make project policy equally or more restrictive for production workloads.
  • Replace shared or overbroad keys with workload identities and the narrowest verified permissions; isolate development, staging, production, and disaster-recovery credentials.
  • Rotate with a bounded overlap: issue the replacement, deploy consumers, prove traffic moved, revoke the predecessor, and test that the old credential now fails safely.
  • Review unused, ownerless, non-expiring, and exception credentials on a fixed cadence, preserving evidence of approval, cutover, revocation, and incident response.

Avoid

  • Assuming a new creation policy changes existing API keys.
  • Setting an expiration date without alerting early enough to replace and verify the key.
  • Using one credential across local development, CI, staging, and production.
  • Revoking the predecessor before every intended consumer has demonstrably moved.
  • Logging, exporting, or placing raw key values in the audit inventory or an AI prompt.
qualityorchestration

Quality

Agent cost-quality governance

GitHub Copilot now combines prompt-level auto-routing tiers with organization and enterprise budget-increase requests. Those controls solve different problems: a routing tier expresses a cost, quality, and latency preference for each prompt, while session limits and account budgets bound consumption at different scopes. Govern them as one system so routine work stays efficient, complex work can earn more capacity, and every exception has an owner, reason, expiration, and outcome review.

agentscostbudgetsmodel-routinggovernance

Principles

  • Separate prompt routing, per-session soft limits, per-user budgets, and account spending limits; none is a substitute for the others.
  • Allocate more capability from task risk and measured outcomes, not seniority, urgency language, or a default preference for the largest model.
  • Treat budget increases as time-bounded operating decisions with an accountable approver and evidence that the additional spend restored useful work.
  • Pair usage metrics with delivery and quality evidence so lower spend is not mistaken for better engineering economics.

Steps

  • Inventory agent surfaces, billing owners, model policies, routing defaults, session-limit support, user budgets, spending limits, and available usage evidence.
  • Define task classes with a default routing tier, starting session limit, quality gate, and an explicit condition for escalating capability or credits.
  • Set layered limits from the narrowest useful scope outward, and confirm which limits are soft, hard, pooled, alert-only, or capable of blocking work.
  • Route exhausted-budget requests to the paying account with a reason, requested amount, task owner, expiration, and fallback if approval is denied.
  • Review cost per accepted outcome, rework, blocked sessions, and exception frequency on a fixed cadence; adjust one control at a time and preserve the baseline.

Avoid

  • Calling an auto-routing tier a fixed model selection or guaranteed price.
  • Assuming a soft session limit prevents a final in-flight model call from exceeding the configured amount.
  • Treating a notification threshold as a hard spending stop without verifying the enforcement setting.
  • Approving permanent user-budget increases for a temporary incident or sprint.
  • Optimizing credit consumption without checking whether accepted changes, review load, or defect rates got worse.
agentsorchestration

Orchestration

Managed agent API adoption

OpenAI's Agents API entered public beta on September 10, 2026 as a managed Codex harness for durable cloud agents. It manages agent sessions, orchestration, context compaction, and recovery while the application still chooses tools and an execution environment. Adopt that split deliberately: the API can run the agent, but your application still owns user authorization, business state, side-effect receipts, retention decisions, and release gates.

agents-apimanaged-harnesssessionsproduction

Principles

  • Treat the managed harness as an execution dependency, not as the system of record for business workflow or authorization.
  • Choose hosted, self-hosted, partner, or no environment from data boundaries, required capabilities, isolation, latency, and operational ownership.
  • Record session, environment, artifact, and external side-effect identifiers in application state so retries and audits do not depend on a transcript.
  • Confirm retention and residency requirements before adoption; a self-hosted sandbox does not make the current Agents API eligible for Zero Data Retention.

Steps

  • Map the workload's files, tools, secrets, network paths, side effects, latency target, and retention obligations before choosing an environment.
  • Define a versioned session contract with instructions, model, tool allowlist, environment, concurrency limit, budgets, and terminal outcomes.
  • Keep end-user authorization, approvals, idempotency records, and business workflow transitions in the application control plane.
  • Consume session events into durable application records, correlate artifacts and tool effects, and make resumed work conditional on the latest accepted state.
  • Run outcome, recovery, isolation, retention, and cost evaluations before a canary; roll back to the prior execution path on a material regression.

Avoid

  • Assuming API-managed recovery proves an external write did or did not happen.
  • Selecting a self-hosted sandbox while ignoring that session state still passes through the managed API.
  • Enabling subagents without a workload-specific concurrency budget or shared-filesystem conflict plan.
  • Treating public beta behavior, limits, or data controls as a permanent production contract.
  • Giving the agent broad tools or vault access because the runtime itself is managed.
qualitypractice

Quality

Agent data-access boundaries

A coding agent can receive information through repository files, editor context, terminal output, and connected services. Content exclusion, MCP server admission, tool availability, execution permission, and sandboxing constrain different parts of that path. GitHub's September 2, 2026 release brings content exclusions to the Copilot app and CLI for Business and Enterprise customers; it does not make every agent surface equivalent. Audit the exact client, mode, identity, and data path before trusting a policy label.

agentsdata-accesscontent-exclusionmcppolicy

Principles

  • Document policy-surface coverage: one enabled setting is not proof of enforcement across every editor, terminal, and hosted agent.
  • Keep content exclusions separate from filesystem access and downstream service authorization; an admitted integration still needs narrowly scoped credentials.
  • Treat tool availability as a selection boundary and execution permission as an invocation boundary. Neither replaces the other.
  • Use synthetic evidence and preserve unknowns instead of opening protected files or declaring universal protection.

Steps

  • Inventory sensitive data classes and all routes into agent context without reading protected payloads.
  • Build a client-and-mode matrix with policy owner, effective settings, provider support, verification date, and evidence for each data path.
  • For Copilot, check documented indirect editor-context, symlink, and remote-filesystem limitations; follow the documented policy refresh procedure before testing.
  • Inspect managed MCP endpoint or command matchers, deny conflicts, and credential scope. Do not use a renamable server label as a security identity.
  • With test authorization, pair an allowed synthetic fixture with a denied one and inspect enforcement evidence. A missing marker in the final answer alone is not proof of denial.
  • Record confirmed gaps, untested surfaces, a responsible owner, and a retest plan using Agent Data Access Audit. Stop testing on unexpected exposure.

Avoid

  • Assuming the September app/CLI release implies support in every editor agent mode.
  • Confusing similarly named registry policies and enterprise managed MCP settings.
  • Treating a configured allowlist as proof that every tool on an admitted server is authorized.
  • Using private code, real tokens, or customer records as audit fixtures.
  • Silently weakening policy or retrying through another tool after a denial.
agentsfoundation

Foundation

What agents are

Agents are tool-using AI workers that can inspect a project, decide which tools to call, make bounded changes, run checks, and iterate toward a defined outcome. Treat them like capable teammates with a clear task contract: they need context, allowed actions, verification requirements, and a stopping point so their speed serves the project instead of drifting away from it.

agentstoolsautonomycontext

Principles

  • Treat agents as bounded executors, not independent owners.
  • Give agents clear inputs, allowed tools, verification gates, and stop conditions.
  • Keep durable project rules outside chat so every agent starts from the same contract.

Steps

  • Define the outcome in one concrete sentence.
  • Load the project contract and only the docs relevant to the task.
  • Let the agent inspect before editing.
  • Require verification output before accepting work.

Avoid

  • Letting the agent infer product direction from a vague prompt.
  • Skipping context load because the change looks small.
  • Accepting a diff without checking the behavior it changes.
skillspractice

Practice

When to build skills

Skills are worth building when the same agent behavior keeps showing up across projects, repos, or teams and the workflow has rules stable enough to document. A good skill packages triggers, constraints, examples, scripts, references, and expected outputs so an agent can repeat the work reliably without rediscovering the process every time.

skillsreusepromptsautomation

Principles

  • Build a skill when the workflow repeats and has stable rules.
  • Keep skill instructions specific enough to prevent drift but short enough to be followed.
  • Prefer scripts and templates inside skills when repetition is mechanical.

Steps

  • Name the workflow the skill owns.
  • Write trigger conditions and non-goals.
  • Add command recipes, references, and example outputs.
  • Test the skill on a real repo task before treating it as standard.

Avoid

planningpractice

Practice

Planning without drift

Planning without drift means turning loose ideas into execution-ready specs before agents start changing the codebase. The plan should preserve product intent through scope, non-goals, acceptance criteria, branch naming, verification gates, and review notes, giving every contributor the same target even when implementation moves quickly.

planningspecsscoperoadmap

Principles

  • Use brainstorms for discovery and specs for execution.
  • Approve scope before implementation begins.
  • Record non-goals as aggressively as goals.

Steps

  • Capture the idea in Plan or the planning source of truth.
  • Promote only clear work into an approved spec.
  • Create a feature branch named after the spec.
  • Verify against acceptance criteria before promotion.

Avoid

  • Treating a brainstorm as an implementation contract.
  • Adding adjacent features while already inside a feature branch.
  • Skipping non-goals because the desired outcome feels obvious.
workfloworchestration

Workflow

Feature to prod cycle

A clean feature-to-prod cycle separates local implementation from integration, staging validation, and production release decisions. Feature branches contain focused work, `dev` absorbs reviewed integration, `test` proves the release candidate, and `prod` becomes the intentional deployment boundary instead of whatever branch happened to build successfully.

branchesdeploymentvercelrelease

Principles

  • Feature branches isolate implementation.
  • `dev` is for integration, `test` is for release validation, and `prod` is the production promotion branch.
  • Production deploys should come from a known release branch, not an arbitrary local state.

Steps

  • Branch from the current integration base with `feature/<slug>`.
  • Open a PR from feature to `dev` after lint, build, and review.
  • Promote `dev` to `test` for staging validation.
  • Promote `test` to `prod` for production deployment.
  • Tag the release or merge back to the long-lived default branch after production is healthy.

Avoid

  • Deploying production directly from a feature branch.
  • Letting `dev`, `test`, and `prod` diverge for long periods.
  • Promoting without rerunning checks at each boundary.
workfloworchestration

Orchestration

Orchestrating large work

Large-scale AI development works when the effort is split into small, independently verifiable slices that share one source of truth. Specs define the target, Brain captures durable discoveries, agents execute scoped branches, and handoffs summarize behavior and risks so parallel work compounds instead of creating incompatible assumptions.

orchestrationmulti-agentsliceshandoff

Principles

  • Split broad work into independently verifiable slices.
  • Give each agent one slice and one source of truth.
  • Use Brain for durable discoveries and Plan for execution scope.

Steps

  • Start from an approved spec.
  • Create slices by boundary, risk, or user-facing workflow.
  • Run agents on separate feature branches when work can conflict.
  • Merge only after checks and a human-readable summary of shipped behavior.

Avoid

  • Parallelizing before the architecture boundary is clear.
  • Letting each agent invent its own acceptance criteria.
  • Using chat history as the only handoff artifact.
qualityorchestration

Quality

Quality gates for AI work

AI-assisted changes need explicit quality gates because fast implementation can hide broken behavior, weak assumptions, or undocumented drift. Baseline lint and build checks catch obvious failures, while browser verification, review, preview deployments, and durable notes make sure the shipped result matches the intended workflow instead of merely compiling.

verificationreviewtestingci

Principles

  • Make the smallest check mandatory and broader checks conditional on risk.
  • Review behavior, not just generated code.
  • Capture failed commands when they reveal repeatable traps.

Steps

  • Run focused checks while implementing.
  • Run the repo baseline gate before review.
  • Use preview deployments for user-facing routes.
  • Promote durable lessons into Brain or docs after closeout.

Avoid

  • Assuming generated code is correct because it compiles.
  • Letting UI changes ship without a browser check.
  • Forgetting to update docs when workflow surfaces change.
qualityorchestration

Quality

Eval-driven agent development

Eval-driven agent development turns product requirements and real failures into executable tasks before agent behavior is changed. Strong suites isolate every trial, grade observable outcomes with deterministic checks, use calibrated model judges only for subjective qualities, repeat important tasks to measure reliability, and preserve traces so regressions become reproducible engineering work instead of arguments about whether an agent merely feels better or worse.

evalsagentstracingci

Principles

  • Evaluate the model, harness, tools, environment, and policy as one versioned system.
  • Grade end-state outcomes with code whenever the requirement is directly observable.
  • Keep capability experiments separate from high-reliability regression gates.

Steps

  • Convert a real product failure into a task with clean initial state and explicit success criteria.
  • Run independent repeated trials and capture the full configuration plus trace.
  • Combine deterministic outcome graders with calibrated rubric graders where judgment is unavoidable.
  • Compare the candidate with a pinned baseline and block promotion on material regressions.

Avoid

  • Grading only the final response while ignoring whether the environment actually changed.
  • Requiring one exact tool path when several safe solutions can produce the correct outcome.
  • Letting answer keys, shared state, or infrastructure differences contaminate the result.
workfloworchestration

Orchestration

Conversation-to-PR handoffs

Collaborative agent sessions can turn a Slack or Teams discussion into an issue, code change, or pull request without losing the decisions that shaped the work. The safe pattern is to treat the thread as source material, extract a bounded implementation brief, confirm the repository and branch, preserve the agent's identity on every artifact, and require an independent review before merge.

collaborative-agentshandoffcontextpull-requests

Principles

  • Treat the conversation as evidence, not as an automatically approved specification.
  • Separate who can contribute context from who is authorized to trigger repository changes.
  • Name the target repository and base branch instead of relying on a shared default.
  • Preserve app identity and require an independent human review for agent-authored artifacts.

Steps

  • Summarize the thread into a goal, non-goals, decisions, unknowns, and acceptance criteria.
  • Confirm the repository, base branch, issue, write-authorized owner, and data that may enter agent context.
  • Start one collaborative agent session for one task and keep steering inside its dedicated thread or code channel.
  • Track the generated issue, branch, or pull request as the durable handoff artifact.
  • Review the diff, validation evidence, provenance, and required approvals before merge.

Avoid

  • Sending a long channel history that contains secrets, private data, or unrelated instructions.
  • Assuming every participant who can steer the discussion is allowed to authorize code changes.
  • Letting a channel's default repository or default branch silently choose the change target.
  • Treating a visible collaborative session as a substitute for code review and required checks.