6.6 Observability & Audit

What you'll learn
The four observation axes: structured logs, metrics, traces, session replay
The minimum metrics every production deployment should emit
When session replay is the only thing that lets you diagnose a "why did the agent do that?" incident

Agents are classic "long-tail bug" workloads — fine 90% of the time, then 10% of incomprehensible behavior. No observability = no diagnosis = no improvement.

Four observation axes

┌─────────────────────────────────────────────┐
│ 1. Structured logs: what happened?           │
│    agentao.log + your app logs               │
├─────────────────────────────────────────────┤
│ 2. Metrics: how much, how fast, how expensive? │
│    calls / latency / tokens / failure rate   │
├─────────────────────────────────────────────┤
│ 3. Event stream: per-session replay           │
│    AgentEvent archive                        │
├─────────────────────────────────────────────┤
│ 4. Distributed tracing: end-to-end per request │
│    OpenTelemetry                             │
└─────────────────────────────────────────────┘

Axis one: structured logs

The built-in agentao.log

Defaults to <working_directory>/agentao.log. It's very thorough:

Every LLM request/response (full content — untruncated, but credential-shaped strings are redacted to [REDACTED:<kind>] on write — plus tokens, model)
Every tool call with args and result
MCP server start/stop
Plugin hook dispatch
Context compression triggers

This is your most important debugging tool. In production:

Mount it on a persistent volume (survives restarts)
Rotate daily + keep 7–30 days
Scrub (see 6.5)
Isolate per tenant (natural with working_directory)

Take over Agentao's logger

By default LLMClient.__init__ mutates logging.getLogger("agentao") — sets level to DEBUG, attaches a RotatingFileHandler for <wd>/agentao.log, and evicts its own marker-tagged handlers on re-construction. Hosts that want to own their logging stack should inject a logger explicitly so that mutation never runs:

python

import logging
from agentao import Agentao

# Your own logger — JSON handler shipped to Loki / CloudWatch / ELK
import pythonjsonlogger.jsonlogger as jl
my_logger = logging.getLogger("myapp.agentao")
handler = logging.StreamHandler()
handler.setFormatter(jl.JsonFormatter())
my_logger.addHandler(handler)
my_logger.setLevel(logging.INFO)

agent = Agentao(
    api_key=..., base_url=..., model=...,
    working_directory=workdir,
    logger=my_logger,            # ← skips package-root mutation
)

When you pass logger=, LLMClient short-circuits before building any file handler — so even the default <wd>/agentao.log is not created. For a fully silent agent, use a NullHandler:

python

quiet = logging.getLogger("myapp.agentao")
quiet.addHandler(logging.NullHandler())
quiet.propagate = False
agent = Agentao(..., logger=quiet)

Gotcha

Adding a handler to getLogger("agentao") without also passing logger= works, but the package-root level still gets forced to DEBUG and the rolling agentao.log file still gets written alongside your handler. To fully suppress the file, either inject your own logger (above) or pass log_file=None when building LLMClient yourself.

Full reference (knob matrix, code anchors, LLMClient direct path) in docs/guides/embedding.md §2 → "Optional: silencing or redirecting agentao.log".

Essential fields

Inject business context via on_event:

python

def on_event(ev):
    logger.info("agent_event", extra={
        "event_type": ev.type.value,
        "session_id": current_session_id(),
        "tenant_id": current_tenant_id(),
        "user_id": current_user_id(),
        **ev.data,
    })

session_id / tenant_id / user_id are the most-used filter fields when debugging.

Axis two: metrics

Must-have metrics

Metric	Type	Meaning
`agent.turn.count`	counter	Turns per chat()
`agent.turn.duration_ms`	histogram	Per-turn duration
`agent.tool.calls`	counter, by tool	Tool invocations
`agent.tool.failures`	counter, by tool	Tool failures
`agent.tool.duration_ms`	histogram, by tool	Tool latency
`agent.llm.tokens.prompt`	counter	Prompt tokens
`agent.llm.tokens.completion`	counter	Completion tokens
`agent.llm.tokens.cached`	counter	Prompt-cache hits
`agent.llm.errors`	counter, by error_type	LLM errors
`agent.confirm.requests`	counter, by outcome	Confirm request / allow / reject / timeout
`agent.max_iterations.hits`	counter	Bailouts triggered

Prometheus template

python

from prometheus_client import Counter, Histogram

turn_dur = Histogram("agent_turn_duration_ms", "Turn duration",
                     buckets=[100, 500, 1000, 3000, 10_000, 30_000])
tool_calls = Counter("agent_tool_calls", "Tool invocations", ["tool", "status"])

def on_event(ev):
    if ev.type == EventType.TOOL_COMPLETE:
        tool_calls.labels(tool=ev.data["tool"], status=ev.data["status"]).inc()

start = time.time()
reply = agent.chat(msg)
turn_dur.observe((time.time() - start) * 1000)

Alert thresholds

Metric	Typical threshold
Tool failure rate > 10%	Tools broken or misconfigured
LLM 5xx rate > 2%	Vendor issue
max_iterations hits > 5%	Agent stuck pattern
Cache hit rate < 30%	System prompt is churning
Confirm timeout rate > 10%	UI issue or user drop-off

Axis three: session replay

Agentao can record each session's runtime timeline as append-only JSONL under .agentao/replays/. Enable it per project:

bash

/replay on

This writes .agentao/settings.json:

json

{
  "replay": {
    "enabled": true,
    "max_instances": 20
  }
}

Recording starts on the next session. Existing replay files remain readable after /replay off.

Replay enables:

Session replay (reconstruct issue scene in UI)
Retroactive debugging (see where the LLM made the wrong call)
Compliance audit (user X had the agent do Z at time Y)

Commands

bash

/replay list            # list replay instances (also the default of bare /replay)
/replay on | /replay off  # toggle recording (persists to .agentao/settings.json)
/replay show <id>       # grouped render
/replay show <id> --raw
/replay show <id> --turn <turn_id>
/replay show <id> --kind tool_
/replay show <id> --errors
/replay tail <id> 50
/replay prune

Replay files are separate from saved sessions: save_session / load_session restore conversation state, while replay records what the runtime did.

Capture depth

Default replay capture includes turn boundaries, user messages, assistant chunks, tool lifecycle, permission decisions, sub-agent lifecycle, errors, state changes, and compact LLM deltas.

Deep capture flags live under replay.capture_flags in .agentao/settings.json:

Flag	Default	Risk
`capture_llm_delta`	`true`	Normal replay history delta
`capture_full_llm_io`	`false`	Full provider payloads; sensitive
`capture_tool_result_full`	`false`	Full tool output; may be large or sensitive
`capture_plugin_hook_output_full`	`false`	Full plugin hook output

Custom archive hook

Use the built-in replay recorder first. Add a custom on_event archiver only when you need to send selected events into your own audit pipeline:

python

def audit_event(ev):
    if ev.type in {EventType.TOOL_COMPLETE, EventType.ERROR}:
        audit_log.info("agent_event", extra={
            "type": ev.type.value,
            "session_id": session_id,
            "tenant_id": tenant.id,
            **ev.data,
        })

transport = SdkTransport(on_event=audit_event)

Axis four: distributed tracing

When the agent is embedded in your web service, one user request can span:

Browser → your API → Agent.chat() → LLM API → Agent → custom tool → database

OpenTelemetry stitches these into one trace:

python

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/chat")
async def chat(req: ChatRequest):
    with tracer.start_as_current_span("user_chat") as span:
        span.set_attribute("user.id", req.user_id)
        span.set_attribute("session.id", req.session_id)
        with tracer.start_as_current_span("agent_chat"):
            reply = await asyncio.to_thread(agent.chat, req.message)
        return {"reply": reply}

Inner spans: use the event stream, don't monkey-patch

There is no need to wrap LLMClient or Tool.execute — the runtime already emits its lifecycle as events. Two entry points, different trade-offs:

Entry point	What you get	Stability
`Agentao.events()` / `add_host_event_observer()`	Typed `HostEvent`: tool / sub-agent / permission decisions, carrying `session_id`, `turn_id`, and RFC3339 start/end timestamps	The `agentao.host` public contract
`Agentao(transport=SdkTransport(on_event=...))` / `transport.subscribe()`	Every `AgentEvent`, including LLM calls	Not under the host stability boundary; fields may change between versions

Use the first for tool spans — ToolLifecycleEvent's fields are already span-shaped. LLM spans currently need the second: the HostEvent union has no LLM event (agentao/host/models.py:158).

python

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from agentao.transport import EventType, SdkTransport

tracer = trace.get_tracer(__name__)
open_spans = {}

def on_event(ev):
    if ev.type == EventType.LLM_CALL_STARTED:
        span = tracer.start_span("chat")
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", ev.data["model"])
        open_spans[ev.data["attempt"]] = span
    elif ev.type == EventType.LLM_CALL_COMPLETED:
        span = open_spans.pop(ev.data["attempt"], None)
        if span is None:
            return
        for attr, key in (
            ("gen_ai.usage.prompt_tokens", "prompt_tokens"),
            ("gen_ai.usage.completion_tokens", "completion_tokens"),
            ("gen_ai.response.finish_reason", "finish_reason"),
        ):
            if ev.data.get(key) is not None:   # all three are None on a failed call
                span.set_attribute(attr, ev.data[key])
        if ev.data["status"] == "error":
            span.set_status(Status(StatusCode.ERROR, ev.data["error_class"]))
        span.end()

agent = Agentao(working_directory=".", transport=SdkTransport(on_event=on_event))

Which event supplies which attribute

Attribute	Source event	Payload key
`gen_ai.system`	—	set it yourself, per provider
`gen_ai.request.model` / `.temperature` / `.max_tokens`	`LLM_CALL_STARTED`	`model` / `temperature` / `max_tokens`
`gen_ai.usage.prompt_tokens` / `completion_tokens`	`LLM_CALL_COMPLETED`	same names
`gen_ai.response.finish_reason`	`LLM_CALL_COMPLETED`	`finish_reason` (`None` on failure)
Tool span name / span id / start-end	`ToolLifecycleEvent`	`tool_name` / `tool_call_id` / `started_at`, `completed_at`

Follow OpenTelemetry GenAI semantic conventions.

Two traps

A raw AgentEvent carries no timestamp and no turn_id. TURN_BEGIN's payload is just user_message, so a subscriber stamps the time on receipt and pairs turns positionally between TURN_BEGIN and TURN_END. The typed HostEvent carries both — prefer the first entry point wherever it covers you.
Listeners run synchronously on the emitting thread, and anything they raise is logged and swallowed (agentao/transport/broadcast.py:42). Your exporter must sit behind an OTel batch processor; do not do network I/O inside the callback, or you are adding latency to every turn.

Agentao itself has no OpenTelemetry dependency and ships no built-in exporter. Everything above is host-side wiring — opentelemetry-* is installed in your application. That boundary is deliberate; for how peer harnesses divide it four different ways, see docs/design/otel-peer-survey.zh.md.

Audit and compliance

Must-keep audit events

Scenario	Trigger	Retention
User starts a session	Agent construction	90–365 days
User approves dangerous tool	confirm_tool = True	180–365 days
Permission rule denied	decide = DENY	90 days
Agent modified user data	Business tool execution	Business-defined (usually 1–7 years)
User requests "forget me"	memory.clear_all	Indefinite (compliance evidence)

Scrubbing and retention

Audit logs should not be scrubbed (you lose evidence), but must be encrypted-at-rest with strict access control.

Under compliance regimes: append-only storage (WORM) required.

Minimum viable observability

On a budget:

agentao.log → per-tenant files, daily rotate, 14d retention
prometheus_client → the 5 key metrics, Grafana dashboard
Built-in replay JSONL → .agentao/replays/, tune replay.max_instances
No OpenTelemetry

Enough for 99% of small/mid SaaS. Add APM when you scale.

TL;DR

Four axes: structured logs (agentao.log), metrics (Prometheus / StatsD), traces (OpenTelemetry), session replay.
Minimum metrics: tool call rate by name, tool failure rate, LLM 5xx rate, confirm-timeout rate, turn duration p50/p95/p99, max-iterations hit rate.
Session replay is the killer feature — when "why did the agent do X?" comes up, replay deterministically with replay_config= and step through.
Cost monitoring is a first-class observable: track tokens-per-turn and tokens-per-tenant; sudden 2× spikes usually mean a model swap or skill change.

→ 6.7 Resource Governance & Concurrency

6.6 Observability & Audit ​

Four observation axes ​

Axis one: structured logs ​

The built-in agentao.log ​

Take over Agentao's logger ​

Essential fields ​

Axis two: metrics ​

Must-have metrics ​

Prometheus template ​

Alert thresholds ​

Axis three: session replay ​

Commands ​

Capture depth ​

Custom archive hook ​

Axis four: distributed tracing ​

Inner spans: use the event stream, don't monkey-patch ​

Which event supplies which attribute ​

Two traps ​

Audit and compliance ​

Must-keep audit events ​

Scrubbing and retention ​

Minimum viable observability ​

TL;DR ​

6.6 Observability & Audit

Four observation axes

Axis one: structured logs

The built-in agentao.log

Take over Agentao's logger

Essential fields

Axis two: metrics

Must-have metrics

Prometheus template

Alert thresholds

Axis three: session replay

Commands

Capture depth

Custom archive hook

Axis four: distributed tracing

Inner spans: use the event stream, don't monkey-patch

Which event supplies which attribute

Two traps

Audit and compliance

Must-keep audit events

Scrubbing and retention

Minimum viable observability

TL;DR