AgentCraftpractical AI/LLM engineering, by LogicPond

Why Your AI Agent Forgets Things Mid-Task (and Three Memory Patterns That Fix It)

If you’ve built anything beyond a single-turn chatbot, you’ve hit this: your agent correctly identifies a user’s preference, a file path, or a decision on turn 3, and by turn 40 it’s acting like that never happened. This isn’t a bug in the model — it’s a predictable consequence of how context windows work, and it has a small number of well-understood fixes.

The actual mechanism

An LLM has no persistent state between calls. Every request you send is the entire conversation so far, re-read from scratch. “Memory” in a running agent is really just: whatever text is still inside that window by the time the model reads it.

Three things push information out of effective memory even when the raw tokens are technically still in the window:

  1. Truncation. Once you hit the context limit, something has to go — usually the oldest turns, which is exactly where early decisions live.
  2. Dilution. Even within the window, a fact stated once on turn 3 competes with thousands of tokens of tool output, retries, and scaffolding by turn 40. Models attend less reliably to a single mention buried in noise than to something recent or repeated.
  3. Summarization loss. If your framework auto-compacts old turns into a summary (many do, to manage cost), the summarizer itself decides what’s “important” — and it often drops the specific detail you needed.

None of this is about the model being bad at “remembering.” It’s about what’s literally present, and how prominently, at inference time.

Pattern 1: Structured scratchpad state (cheapest, fixes the most common case)

For most agents, the fix isn’t a database — it’s forcing important facts out of freeform conversation and into an explicit, structured block that gets re-injected on every turn.

# state that survives every turn, independent of conversation history
agent_state = {
    "user_prefs": {"timezone": "America/Edmonton", "units": "metric"},
    "decisions": ["using SQLite, not Postgres — no ops overhead needed"],
    "open_tasks": ["fix the flaky retry test"],
}

def build_system_prompt(state):
    return f"""You are a coding assistant.

Known facts you must not re-derive or contradict:
{json.dumps(state, indent=2)}
"""

This works because the state block is always near the top of the prompt, on every call — it never ages out and never competes with 40 turns of tool noise. The tradeoff: you (or the agent, via a tool call) have to decide what’s worth promoting into this block. It doesn’t scale to “remember everything,” and it shouldn’t try to.

Pattern 2: Retrieval on demand (for facts too numerous to keep resident)

When there’s more state than fits comfortably in a scratchpad — a large codebase, a long project history — the standard move is to store facts externally and pull back only what’s relevant to the current turn, via embedding search or even plain keyword search over a local file.

The mistake people make here is reaching for a vector database on day one. For most single-user agents, a flat file plus a cheap similarity check is enough:

import json, difflib

def recall(query, memory_path="memory.jsonl", top_k=3):
    facts = [json.loads(l) for l in open(memory_path)]
    scored = sorted(
        facts,
        key=lambda f: difflib.SequenceMatcher(None, query, f["text"]).ratio(),
        reverse=True,
    )
    return scored[:top_k]

It’s crude compared to real embeddings, but it costs nothing to run, has zero infra, and for a lot of agent workloads (a personal assistant, a small codebase) it’s accurate enough. Upgrade to real vector search only once you’ve confirmed keyword/fuzzy matching is actually the bottleneck — not before.

Pattern 3: Explicit checkpointing before compaction

If your framework compacts/summarizes old context automatically, don’t let it decide unsupervised what survives. Before a compaction event, run a cheap, targeted pass that extracts anything load-bearing into Pattern 1’s scratchpad first:

def pre_compaction_hook(conversation_history):
    extraction_prompt = (
        "Extract only: user preferences stated, irreversible decisions made, "
        "and open commitments. Ignore everything else. Output as JSON."
    )
    extracted = llm_call(extraction_prompt, conversation_history)
    agent_state.update(extracted)
    # only now let the framework summarize/drop the raw turns

This turns compaction from a lossy black box into a controlled handoff: the summarizer is still free to compress the narrative, but the facts you actually depend on already live somewhere that doesn’t get compressed away.

Which one do you actually need?

Most production agents end up running all three at once. The failure mode to watch for is reaching for retrieval (Pattern 2) as the first fix, when the actual bug is that three important facts just needed to live in a 200-token block that never leaves the prompt.

← Back to all articles