Observe agent workflows without keeping everything they read

An assistant reports that a document lookup failed. The operator wants to know whether the cause was authorization, retrieval, a timeout or a bad tool choice. Logging the entire conversation may seem like the fastest way to answer. It can also copy customer documents, credentials and private instructions into a system with different access and retention rules.

Good observability starts with the questions operations needs to answer. Capture the smallest useful evidence for those questions, then design a controlled path for deeper investigation when necessary.

Separate operational telemetry from diagnostic content

Operational telemetry should show which component handled a request, what operation was attempted, how long it took and how it ended. That does not usually require the full tool arguments or result body.

Diagnostic content is different. A redacted request fragment or a synthetic reproduction may be useful for a specific investigation. Give such capture an owner, purpose, access boundary and expiry rather than making it a permanent default.

OpenTelemetry's sensitive-data guidance emphasizes data minimization and makes clear that the implementer must understand what instrumentation collects. Installing an instrumentation library does not establish that its emitted fields are appropriate for your application.

Review automatic HTTP, database and exception instrumentation as carefully as your own logging. A safe custom event can coexist with a request logger that captures the same secret through a URL, header or exception message.

Design an event around an outcome

For a document-search operation, useful application attributes might look like this:

{
  "event": "tool.operation.finished",
  "app.tool.name": "search_documents",
  "app.tool.revision": "3",
  "app.operation.kind": "read",
  "app.policy.decision": "allow",
  "app.outcome": "upstream_timeout",
  "app.duration_ms": 4200,
  "app.request.reference": "<opaque correlation reference>"
}

These names are an illustrative application convention, not official MCP or OpenTelemetry semantic-convention fields. Adapt them to your telemetry system and document the meaning of each outcome.

The result count is deliberately absent: an upstream_timeout is not a completed search with zero matches. A schema that forces a number for every outcome can create misleading dashboards. Reserve a count of zero for a completed operation that actually returned no records.

Do not put individual request references, document identifiers or arbitrary model-generated tool names into metric labels without controlling their cardinality. Aggregate metrics need a bounded vocabulary. Detailed correlations belong in access-controlled traces or events designed to hold them.

Trace the operation across boundaries

Give a workflow a correlation reference and connect the host, MCP service and downstream request where your instrumentation supports that relationship. Record both the model-facing tool name and the actual downstream operation type, using bounded identifiers.

Distinguish a tool invocation from a business effect. One invocation might fail before reaching the downstream service, create a job that completes later, or return an unknown outcome after a disconnect. A single “tool success” counter cannot explain all three.

For state-changing operations, record the accepted operation reference and final status in the application system of record. Telemetry is valuable for investigation, but a sampled trace should not be the only evidence that a customer-visible write occurred.

Treat propagation as disclosure

Context that travels between services can leave your intended boundary. OpenTelemetry's baggage documentation warns that baggage can propagate to downstream services and has no built-in integrity guarantee.

Keep secrets, raw user text and sensitive identifiers out of baggage. Treat incoming baggage as untrusted metadata, not an authorization source. A header claiming a tenant or user identity should not override the identity established by the application's authentication layer.

Review egress behavior when integrating a third-party service. An attribute stored only in a private trace backend has a different exposure path from a value automatically attached to outbound requests. “It is only telemetry” does not describe who receives it.

Redact before export, then verify it worked

Prefer not collecting sensitive fields to collecting and later trying to recognize them. Use an allowlist of application attributes where practical. Scrubbing can remain a second control for unexpected content, especially exception paths.

Test redaction using synthetic canaries that resemble the kinds of data you intend to exclude: an authorization header, an email address, a document excerpt and a secret embedded in a query parameter. Trigger successful requests, validation errors, downstream failures and timeouts. Inspect what reaches the actual telemetry destination.

Hashing is not automatically anonymization. Predictable identifiers can sometimes be guessed and compared. If correlation is necessary, define the threat model for the chosen transformation, control access to any secret used in it and decide how long correlations should remain possible.

Choose retention by purpose

A short-lived diagnostic capture, an operational latency metric and an authorization audit event serve different purposes. Give them separate retention and access policies. Do not inherit a long default retention period merely because a storage plan makes it convenient.

Document who can enable deeper capture, whether the affected user or tenant must be informed under your policy, and how capture is stopped. Include deletion and access review in the operational procedure. A temporary debugging switch that nobody remembers to disable becomes a permanent data pipeline.

During an incident, preserve the evidence needed for the investigation through the approved process. Do not respond by indiscriminately turning on full prompt and response logging for every customer.

Review dashboards for honest interpretation

Build a small set of views around answerable questions: Are authorized tasks completing? Which dependency is timing out? Are permission denials increasing? Are writes ending with unresolved outcomes? Is a new tool revision producing different error categories?

Show missing telemetry as missing, not as a healthy zero. Explain sampling and collection gaps where they affect interpretation. A decline in logged errors may reflect a broken exporter rather than an improved service.

The practical target is an operator who can locate a failing boundary without opening a customer's private document. When deeper evidence is necessary, it should be collected deliberately. That gives the system enough memory to be operated responsibly without making every user interaction part of a permanent debugging archive.