Agent Observability: From Tool Traces to Adoption Metrics

An agent can finish a task while still being operationally unhealthy. It may retry the same tool five times, send sensitive prompt content to a trace backend, burn most of its tokens on recovery, or produce a change that fails review. A green final status hides all of that.

Useful agent observability connects three different questions:

1. What did the agent do? Trace the workflow, model calls, plans, handoffs, and tool executions.
2. Did the system operate safely and efficiently? Measure latency, errors, retries, approvals, cost, and privacy controls.
3. Did the work create value? Join runtime telemetry to reviews, merges, deployments, incidents, or another real outcome.

These layers are becoming easier to standardize. GitHub's August 2026 Copilot usage metrics update added per-agent activity grouped by a stable agent_id. OpenTelemetry moved its GenAI conventions into a dedicated repository with operations for agents, workflows, plans, model calls, retrieval, and tools. The OpenAI Agents SDK exposes built-in traces and spans for the same kind of execution graph.

The opportunity is not to collect everything. It is to create a small, durable telemetry contract that helps operators answer real questions without copying private work into another system.

Separate Adoption, Runtime, and Outcome Data

Do not force every question into one event stream.

GitHub's current usage schema is a good example of the first layer. Its totals_by_3rd_party_agent records identify recognized agent apps by agent_id, display them with agent_name, and report job starts plus aggregated session counts. The field reference warns that the nested job-start count is different from the top-level prompt count. Those values should not be summed or treated as equivalent.

That data can tell you whether an agent is being tried. It cannot tell you whether the agent made a good change. Runtime and outcome evidence must supply the rest.

Start With a Stable Identity Contract

Human-readable names change. Build joins on stable identifiers and reserve names for display.

A practical identity set includes:

• agent.id: stable ID for the agent implementation or product
• agent.name: readable label that may change
• agent.version: prompt, skill, policy, or application version
• workflow.name: low-cardinality workflow such as pull_request_review
• workflow.version: version of the orchestration contract
• run.id: unique invocation ID
• trace.id: end-to-end correlation ID
• task.id: durable link to the issue, job, or work item
• environment: development, test, or production

Do not use a user-written task title, repository path, prompt, or full URL as a metric label. Those values have high or unbounded cardinality and can make metric storage expensive or unusable. Keep them in controlled trace attributes or logs only when policy permits.

GitHub's recommendation to group on agent_id instead of the mutable agent_name is broadly useful: identifiers are for joins; labels are for people.

Model the Execution as a Trace

A trace should describe one end-to-end unit of work. Spans describe operations inside it.

The current OpenTelemetry GenAI agent span conventions define a useful vocabulary:

• invoke_workflow for the orchestration boundary
• invoke_agent for an agent invocation
• plan when the framework can reliably identify planning or task decomposition
• model inference spans for generation calls
• retrieval or memory spans for context access
• execute_tool for tool execution

A coding workflow might produce this hierarchy:

invoke_workflow pull_request_change
├── invoke_agent implementation_agent
│   ├── plan implementation_agent
│   ├── generate_content gpt-model
│   ├── execute_tool read_file
│   ├── execute_tool apply_patch
│   └── execute_tool run_tests
└── invoke_agent review_agent
    ├── generate_content review-model
    └── execute_tool inspect_diff

The exact tree will vary by framework. Preserve the causal structure: model calls and tools should be children of the turn, agent, or workflow that caused them. That makes it possible to explain a slow or failed run instead of staring at an unordered list of events.

Propagate Context Across Boundaries

Agent work often crosses a web request, queue, worker, sandbox, tool server, and third-party API. If each component starts a new trace, the most important causal links disappear.

Propagate the trace context through every trusted boundary. OpenTelemetry's context propagation guidance uses the W3C traceparent header by default so a downstream span can retain the upstream trace and parent relationship. For queues or durable jobs, store the trace context with the job envelope and restore it when the worker resumes.

Treat incoming context from untrusted systems as untrusted input. Validate or discard forged headers, and do not place credentials, personal data, customer content, or internal authorization details in propagated baggage.

Capture Decisions, Not Every Thought

The most useful agent trace usually records:

• operation type and duration
• agent, workflow, model, and tool identifiers
• token or usage totals when the provider returns them
• retry and timeout counts
• approval requested, granted, denied, or expired
• tool result status and a bounded result summary
• policy or guardrail decision
• artifact, commit, PR, or deployment reference
• terminal outcome and error type

It usually should not record raw prompts, full model outputs, tool arguments, tool results, source files, email bodies, access tokens, or hidden reasoning.

The OpenTelemetry GenAI span specification marks instructions and message payloads as opt-in and notes that they may be large or exceed backend limits. OpenAI's Python Agents SDK also documents that trace and diagnostic payloads can contain sensitive model or tool data and provides configuration controls to suppress them.

Make payload capture deny-by-default in production. If a debugging session requires content:

1. Use a non-production environment and synthetic data.
2. Enable capture for a short, documented window.
3. Redact secrets and personal data before export.
4. Restrict access and retention.
5. Confirm the override is disabled afterward.

Redaction must happen before data leaves the process or trusted collector. Deleting a secret from the dashboard after export is incident response, not a privacy control.

Use Metrics for Trends, Traces for Explanations

Metrics should be low-cardinality and stable enough to compare over time. Useful agent metrics include:

Reliability

• successful, failed, cancelled, and timed-out runs
• tool error rate by tool ID
• retry rate and retries per successful run
• approval denial and expiration rate
• runs resumed after interruption

Efficiency

• end-to-end run duration
• model and tool duration distributions
• tokens per run and per accepted outcome
• tool calls per run
• queue time versus execution time

Governance

• runs by agent and workflow version
• sensitive-payload capture enabled count
• policy blocks and approval requests
• unrecognized or unversioned agents
• traces missing task or artifact correlation

Outcome

• reviewed changes per run
• accepted or merged changes per run
• deployment success and rollback rate
• human correction or rework rate
• incidents linked to agent-produced artifacts

Avoid a single "agent productivity score." It hides tradeoffs and invites gaming. A faster agent that creates more rework is not necessarily better. Keep the component measures visible and compare them within the same workflow, task class, and review policy.

Join Telemetry to Evidence

The outcome join is where observability becomes operationally useful.

When a run creates an artifact, record both directions when possible:

• the trace stores the issue, commit, PR, build, or deployment ID
• the artifact stores the originating run or trace ID in metadata, a check, or an audit record

Do not put raw trace IDs into high-cardinality metrics. Store them on traces, structured logs, and artifact metadata, then query metrics to find the affected time window and traces to investigate specific runs.

A lightweight run receipt can look like this:

{
  "run_id": "run_01J...",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "task_id": "issue-482",
  "agent_id": "code-change-agent",
  "agent_version": "2026.08.27",
  "workflow_name": "pull_request_change",
  "artifact": {
    "type": "pull_request",
    "id": "repo#917"
  },
  "verification": ["lint", "test", "build"],
  "outcome": "awaiting_review"
}

This receipt contains correlation and status, not the prompt or source code.

Build Dashboards Around Decisions

Every dashboard should support a decision.

Start with a small set. If nobody can name the action a chart triggers, remove the chart.

A Safe Rollout Sequence

1. Inventory workflows and owners. Identify each production agent, its tools, data classes, and outcome system.
2. Define stable IDs. Version agents and workflows before collecting metrics.
3. Add one root trace. Correlate it with the durable task or job ID.
4. Instrument model and tool boundaries. Record duration, status, and bounded metadata.
5. Disable payload capture. Test redaction before sending telemetry to a backend.
6. Propagate context. Preserve causality through workers, sandboxes, and trusted tools.
7. Emit low-cardinality metrics. Start with reliability, latency, usage, and policy exceptions.
8. Join one real outcome. For coding agents, use review, merge, deployment, or rollback evidence.
9. Test failure modes. Include timeouts, retries, resume, duplicate work, rejected approvals, and export failure.
10. Review retention and access. Trace data deserves the same governance as application logs and support data.

Production Checklist

• Stable agent, workflow, version, run, task, and trace identifiers exist.
• Trace structure distinguishes workflows, agents, models, plans, and tools.
• Context survives queues, workers, and sandbox boundaries.
• Prompt, output, tool payload, and baggage capture is deny-by-default.
• Redaction happens before export.
• Metric labels are bounded and low-cardinality.
• Retries, approvals, policy blocks, and terminal outcomes are visible.
• At least one external outcome is joined to the run.
• Telemetry export failure cannot fail the user workflow.
• Retention, access, and deletion policies are documented and tested.

The Operating Principle

Agent observability is not transcript storage. It is a causal record of bounded operations, stable identities, safety decisions, and outcomes.

Trace enough to explain behavior. Measure enough to see trends. Join enough external evidence to know whether the work mattered. Keep raw content out unless a tightly controlled debugging need justifies it.