Agent Observability: What to Log So You Can Debug Failures After the Fact

The frustrating part of a production agent failure is almost never diagnosing it — it’s that the information needed to diagnose it was never captured, and the exact conversation that caused the failure is gone by the time you’re looking. Fixing this doesn’t require an observability platform; it requires deciding, before something breaks, what a debugging session will actually need.

The minimum viable schema

Four things, logged per tool call, are enough to reconstruct most failures without guessing:

import json
import time
import uuid

def log_tool_call(run_id, call_name, arguments, result, duration_ms, error=None):
    entry = {
        "run_id": run_id,               # ties every call in one agent run together
        "timestamp": time.time(),
        "tool": call_name,
        "arguments": arguments,         # exact inputs, not a paraphrase
        "result_summary": str(result)[:500],  # truncated, not the full payload
        "duration_ms": duration_ms,
        "error": error,
    }
    with open("agent_calls.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

The two fields people skip and then regret: exact arguments, not a summary of what the agent “intended” to do — if a tool call fails because of a malformed argument, you need the actual malformed value, not your code’s guess at what it meant. And duration, which is the cheapest possible signal for “this call hung” or “this call is unusually slow this time,” visible in the log without needing to reproduce the failure live.

Log the decision, not just the outcome

A tool call succeeding or failing tells you what happened. It doesn’t tell you why the model chose that tool over another one — and reconstructing that after the fact from the raw conversation is exactly the debugging tax this section removes:

def log_agent_turn(run_id, available_tools, model_response, chosen_tool):
    entry = {
        "run_id": run_id,
        "timestamp": time.time(),
        "tools_in_scope": [t["name"] for t in available_tools],  # what it could have picked
        "chosen_tool": chosen_tool,
        "raw_response_text": model_response.content[:1000],      # any reasoning it exposed
    }
    with open("agent_turns.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

tools_in_scope matters specifically because of the “too many similar tools” failure mode from the hallucinated tool-call field guide — without logging what was available, you can’t later confirm whether a wrong tool choice was a real model error or a predictable consequence of an overcrowded tool list you can only see by reconstructing it from memory.

What NOT to log, and why it’s not just a storage concern

• Full, untruncated tool results. Logging a complete 50KB API response on every call turns your log into something too large to actually search through when you need it. Truncate to what’s useful for pattern-matching a failure, not archival.
• Raw user input containing sensitive data, unredacted. If the agent handles anything personal or credentialed, logging arguments verbatim means your debug log is now a second place that data lives, with its own retention and access problem you didn’t sign up to own.
• Everything, “just in case.” A log nobody can query because it’s undifferentiated noise is barely better than no log — the goal is a log a person can grep through during an actual incident, not a complete transcript of everything that happened.

The test for whether your logging is actually sufficient

Pick a real failure from memory — a tool call that returned something wrong, an agent that looped, a wrong tool chosen. Ask: could I have diagnosed this from the log alone, without being able to reproduce it live? If the honest answer is no, that’s the specific field to add — not a general “log more,” but the one concrete piece of information that specific failure was missing.