Progressive disclosure

​

Progressive Disclosure: Claude-Mem’s Context Priming Philosophy

​

Core Principle

Show what exists and its retrieval cost first. Let the agent decide what to fetch based on relevance and need.

---

​

What is Progressive Disclosure?

Progressive disclosure is an information architecture pattern where you reveal complexity gradually rather than all at once. In the context of AI agents, it means:

1. Layer 1 (Index): Show lightweight metadata (titles, dates, types, token counts)
2. Layer 2 (Details): Fetch full content only when needed
3. Layer 3 (Deep Dive): Read original source files if required

This mirrors how humans work: We scan headlines before reading articles, review table of contents before diving into chapters, and check file names before opening files.

---

​

The Problem: Context Pollution

Traditional RAG (Retrieval-Augmented Generation) systems fetch everything upfront:

❌ Traditional Approach:
┌─────────────────────────────────────┐
│ Session Start                        │
│                                      │
│ [15,000 tokens of past sessions]    │
│ [8,000 tokens of observations]      │
│ [12,000 tokens of file summaries]   │
│                                      │
│ Total: 35,000 tokens                │
│ Relevant: ~2,000 tokens (6%)        │
└─────────────────────────────────────┘

Problems:

• Wastes 94% of attention budget on irrelevant context
• User prompt gets buried under mountain of history
• Agent must process everything before understanding task
• No way to know what’s actually useful until after reading

---

​

Claude-Mem’s Solution: Progressive Disclosure

✅ Progressive Disclosure Approach:
┌─────────────────────────────────────┐
│ Session Start                        │
│                                      │
│ Index of 50 observations: ~800 tokens│
│ ↓                                    │
│ Agent sees: "🔴 Hook timeout issue"  │
│ Agent decides: "Relevant!"           │
│ ↓                                    │
│ Fetch observation #2543: ~120 tokens│
│                                      │
│ Total: 920 tokens                   │
│ Relevant: 920 tokens (100%)         │
└─────────────────────────────────────┘

Benefits:

• Agent controls its own context consumption
• Directly relevant to current task
• Can fetch more if needed
• Can skip everything if not relevant
• Clear cost/benefit for each retrieval decision

---

​

How It Works in Claude-Mem

​

The Index Format

Every SessionStart hook provides a compact index:

### Oct 26, 2025

**General**
| ID | Time | T | Title | Tokens |
|----|------|---|-------|--------|
| #2586 | 12:58 AM | 🔵 | Context hook file exists but is empty | ~51 |
| #2587 | ″ | 🔵 | Context hook script file is empty | ~46 |
| #2589 | ″ | 🟡 | Investigated hook debug output docs | ~105 |

**src/hooks/context-hook.ts**
| ID | Time | T | Title | Tokens |
|----|------|---|-------|--------|
| #2591 | 1:15 AM | ⚖️ | Stderr messaging abandoned | ~155 |
| #2592 | 1:16 AM | ⚖️ | Web UI strategy redesigned | ~193 |

What the agent sees:

• What exists: Observation titles give semantic meaning
• When it happened: Timestamps for temporal context
• What type: Icons indicate observation category
• Retrieval cost: Token counts for informed decisions
• Where to get it: MCP search tools referenced at bottom

​

The Legend System

🎯 session-request  - User's original goal
🔴 gotcha          - Critical edge case or pitfall
🟡 problem-solution - Bug fix or workaround
🔵 how-it-works    - Technical explanation
🟢 what-changed    - Code/architecture change
🟣 discovery       - Learning or insight
🟠 why-it-exists   - Design rationale
🟤 decision        - Architecture decision
⚖️ trade-off       - Deliberate compromise

Purpose:

• Visual scanning (humans and AI both benefit)
• Semantic categorization
• Priority signaling (🔴 gotchas are more critical)
• Pattern recognition across sessions

​

Progressive Disclosure Instructions

The index includes usage guidance:

💡 **Progressive Disclosure:** This index shows WHAT exists and retrieval COST.
- Use MCP search tools to fetch full observation details on-demand
- Prefer searching observations over re-reading code for past decisions
- Critical types (🔴 gotcha, 🟤 decision, ⚖️ trade-off) often worth fetching immediately

What this does:

• Teaches the agent the pattern
• Suggests when to fetch (critical types)
• Recommends search over code re-reading (efficiency)
• Makes the system self-documenting

---

​

The Philosophy: Context as Currency

​

Mental Model: Token Budget as Money

Think of context window as a bank account:

Approach	Metaphor	Outcome
Dump everything	Spending your entire paycheck on groceries you might need someday	Waste, clutter, can’t afford what you actually need
Fetch nothing	Refusing to spend any money	Starvation, can’t accomplish tasks
Progressive disclosure	Check your pantry, make a shopping list, buy only what you need	Efficiency, room for unexpected needs

​

The Attention Budget

LLMs have finite attention:

• Every token attends to every other token (n² relationships)
• 100,000 token window ≠ 100,000 tokens of useful attention
• Context “rot” happens as window fills
• Later tokens get less attention than earlier ones

Claude-Mem’s approach:

• Start with ~1,000 tokens of index
• Agent has 99,000 tokens free for task
• Agent fetches ~200 tokens when needed
• Final budget: ~98,000 tokens for actual work

​

Design for Autonomy

“As models improve, let them act intelligently”

Progressive disclosure treats the agent as an intelligent information forager, not a passive recipient of pre-selected context.

Traditional RAG:

System → [Decides relevance] → Agent
        ↑
   Hope this helps!

Progressive Disclosure:

System → [Shows index] → Agent → [Decides relevance] → [Fetches details]
                          ↑
                   You know best!

The agent knows:

• The current task context
• What information would help
• How much budget to spend
• When to stop searching

We don’t.

---

​

Implementation Principles

​

1. Make Costs Visible

Every item in the index shows token count:

| #2591 | 1:15 AM | ⚖️ | Stderr messaging abandoned | ~155 |
                                                        ^^^^
                                                    Retrieval cost

Why:

• Agent can make informed ROI decisions
• Small observations (~50 tokens) are “cheap” to fetch
• Large observations (~500 tokens) require stronger justification
• Matches how humans think about effort

​

2. Use Semantic Compression

Titles compress full observations into ~10 words:

Bad title:

Observation about a thing

Good title:

🔴 Hook timeout issue: 60s default too short for npm install

What makes a good title:

• Specific: Identifies exact issue
• Actionable: Clear what to do
• Self-contained: Doesn’t require reading observation
• Searchable: Contains key terms (hook, timeout, npm)
• Categorized: Icon indicates type

​

3. Group by Context

Observations are grouped by:

• Date: Temporal context
• File path: Spatial context (work on specific files)
• Project: Logical context

**src/hooks/context-hook.ts**
| ID | Time | T | Title | Tokens |
|----|------|---|-------|--------|
| #2591 | 1:15 AM | ⚖️ | Stderr messaging abandoned | ~155 |
| #2594 | 1:17 AM | 🟠 | Removed stderr section from docs | ~93 |

Benefit: If agent is working on src/hooks/context-hook.ts, related observations are already grouped together.

​

4. Provide Retrieval Tools

The index is useless without retrieval mechanisms:

*Use claude-mem MCP search to access records with the given ID*

Available MCP tools:

• search - Search memory index (Layer 1: Get IDs)
• timeline - Get chronological context (Layer 2: See narrative arc)
• get_observations - Fetch full details (Layer 3: Deep dive)
• get_tool_uses - Fetch the original tool_input / tool_response (Layer 4: Raw evidence)

The layered workflow ensures progressive disclosure: index → context → details → raw evidence.

---

​

Real-World Example

​

Scenario: Agent asked to fix a bug in hooks

Without progressive disclosure:

SessionStart injects 25,000 tokens of past context
Agent reads everything
Agent finds 1 relevant observation (buried in middle)
Total tokens consumed: 25,000
Relevant tokens: ~200
Efficiency: 0.8%

With progressive disclosure:

SessionStart shows index: ~800 tokens
Agent sees title: "🔴 Hook timeout issue: 60s too short"
Agent thinks: "This looks relevant to my bug!"
Agent fetches observation #2543: ~155 tokens
Total tokens consumed: 955
Relevant tokens: 955
Efficiency: 100%

​

The Index Entry

| #2543 | 2:14 PM | 🔴 | Hook timeout: 60s too short for npm install | ~155 |

What the agent learns WITHOUT fetching:

• There’s a known gotcha (🔴) about hook timeouts
• It’s related to npm install taking too long
• Full details are ~155 tokens (cheap)
• Happened at 2:14 PM (recent)

Decision tree:

Is my task related to hooks? → YES
Is my task related to timeouts? → YES
Is my task related to npm? → YES
155 tokens is cheap → FETCH IT

---

​

The Layered Workflow

Claude-Mem implements progressive disclosure through a layered workflow pattern. Layers 1-3 are the everyday path; layer 4 exists for the rare case where a summary is not evidence enough.

​

Layer 1: Search (Index)

Start by searching to get a compact index with IDs:

search({
  query: "hook timeout",
  limit: 10
})

Returns:

Found 3 observations matching "hook timeout":

| ID | Date | Type | Title |
|----|------|------|-------|
| #2543 | Oct 26 | gotcha | Hook timeout: 60s too short |
| #2891 | Oct 25 | how-it-works | Hook timeout configuration |
| #2102 | Oct 20 | problem-solution | Fixed timeout in CI |

Cost: ~50-100 tokens per result Value: Agent can scan and decide which observations are relevant

​

Layer 2: Timeline (Context)

Get chronological context around interesting observations:

timeline({
  anchor: 2543,  // Observation ID from search
  depth_before: 3,
  depth_after: 3
})

Returns: Chronological view showing what happened before/during/after observation #2543

Cost: Variable based on depth Value: Understand narrative arc and context

​

Layer 3: Get Observations (Details)

Fetch full details only for relevant observations:

get_observations({
  ids: [2543, 2102]  // Selected after reviewing the index and timeline
})

Returns:

#2543 🔴 Hook timeout: 60s too short for npm install
─────────────────────────────────────────────────
Date: Oct 26, 2025 2:14 PM
Type: gotcha
Project: claude-mem

Narrative:
Discovered that the default 60-second hook timeout is insufficient
for npm install operations, especially with large dependency trees
or slow network conditions. This causes SessionStart hook to fail
silently, preventing context injection.

Facts:
- Default timeout: 60 seconds
- npm install with cold cache: ~90 seconds
- Configured timeout: 120 seconds in plugin/hooks/hooks.json:25

Files Modified:
- plugin/hooks/hooks.json

Concepts: hooks, timeout, npm, configuration

Cost: ~155 tokens for full details Value: Complete understanding of the issue

​

Layer 4: Get Tool Uses (Raw Evidence)

Observations are compressed summaries of what tools did. Sometimes the compression is exactly what you do not want — you need the literal bytes: the exact diff that was applied, the exact stderr a build printed, the exact JSON an API returned.

Claude-mem keeps those raw bodies in the tool_uses table, and this layer is the only way to read them back:

get_tool_uses({
  ids: ["toolu_01ABC..."],   // or numeric tool_uses ids
  project: "claude-mem"
})

Returns: The stored tool_input / tool_response for those calls, along with the tool name, the session ids, and the observation the call was folded into.

Cost: Up to 64 KB per row — by far the most expensive layer Value: The unsummarized truth, when and only when the summary was not enough

Never start here, and never call get_tool_uses without ids you got from layers 1-3. Raw tool bodies are exactly the context pollution the rest of this page exists to avoid; disclosing a handful on purpose is progressive disclosure, disclosing them by default is just a bigger transcript.

Payloads larger than 64 KB are truncated on write and carry a …[truncated: N bytes] marker, so a clipped body is distinguishable from one that legitimately ends where it does.

---

​

Cognitive Load Theory

Progressive disclosure is grounded in Cognitive Load Theory:

​

Intrinsic Load

The inherent difficulty of the task itself.

Example: “Fix authentication bug”

• Must understand auth system
• Must understand the bug
• Must write the fix

This load is unavoidable.

​

Extraneous Load

The cognitive burden of poorly presented information.

Traditional RAG adds extraneous load:

• Scanning irrelevant observations
• Filtering out noise
• Remembering what to ignore
• Re-contextualizing after each section

Progressive disclosure minimizes extraneous load:

• Scan titles (low effort)
• Fetch only relevant (targeted effort)
• Full attention on current task

​

Germane Load

The effort of building mental models and schemas.

Progressive disclosure supports germane load:

• Consistent structure (legend, grouping)
• Clear categorization (types, icons)
• Semantic compression (good titles)
• Explicit costs (token counts)

---

​

Anti-Patterns to Avoid

​

❌ Verbose Titles

Bad:

| #2543 | 2:14 PM | 🔴 | Investigation into the issue where hooks time out | ~155 |

Good:

| #2543 | 2:14 PM | 🔴 | Hook timeout: 60s too short for npm install | ~155 |

​

❌ Hiding Costs

Bad:

| #2543 | 2:14 PM | 🔴 | Hook timeout issue |

Good:

| #2543 | 2:14 PM | 🔴 | Hook timeout issue | ~155 |

​

❌ No Retrieval Path

Bad:

Here are 10 observations. [No instructions on how to get full details]

Good:

Here are 10 observations.
*Use MCP search tools to fetch full observation details on-demand*

​

❌ Skipping the Index Layer

Bad:

// Fetching full details immediately
get_observations({
  ids: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]  // Guessing which are relevant
})

Good:

// Follow the 3-layer workflow
// Layer 1: Search for index
search({
  query: "hooks",
  limit: 20
})

// Layer 2: Review context around a relevant index result
timeline({
  anchor: 2543,
  depth_before: 3,
  depth_after: 3
})
// Keep only IDs that remain relevant after reading the timeline

// Layer 3: Fetch only relevant observations
get_observations({
  ids: [2543, 2891]  // Just the most relevant
})

---

​

Key Design Decisions

​

Why Token Counts?

Decision: Show approximate token counts (~155, ~203) rather than exact counts.

Rationale:

• Communicates scale (50 vs 500) without false precision
• Maps to human intuition (small/medium/large)
• Allows agent to budget attention
• Encourages cost-conscious retrieval

​

Why Icons Instead of Text Labels?

Decision: Use emoji icons (🔴, 🟡, 🔵) rather than text (GOTCHA, PROBLEM, HOWTO).

Rationale:

• Visual scanning (pattern recognition)
• Token efficient (1 char vs 10 chars)
• Language-agnostic
• Aesthetically distinct
• Works for both humans and AI

​

Why Index-First, Not Smart Pre-Fetch?

Decision: Always show index first, even if we “know” what’s relevant.

Rationale:

• We can’t know what’s relevant better than the agent
• Pre-fetching assumes we understand the task
• Agent knows current context, we don’t
• Respects agent autonomy
• Fails gracefully (can always fetch more)

​

Why Group by File Path?

Decision: Group observations by file path in addition to date.

Rationale:

• Spatial locality: Work on file X likely needs context about file X
• Reduces scanning effort
• Matches how developers think
• Clear semantic boundaries

---

​

Measuring Success

Progressive disclosure is working when:

​

✅ Low Waste Ratio

Relevant Tokens / Total Context Tokens > 80%

Most of the context consumed is actually useful.

​

✅ Selective Fetching

Index Shown: 50 observations
Details Fetched: 2-3 observations

Agent is being selective, not fetching everything.

​

✅ Fast Task Completion

Session with index: 30 seconds to find relevant context
Session without: 90 seconds scanning all context

Time-to-relevant-information is faster.

​

✅ Appropriate Depth

Simple task: Only index needed
Medium task: 1-2 observations fetched
Complex task: 5-10 observations + code reads

Depth scales with task complexity.

---

​

Future Enhancements

​

Adaptive Index Size

// Vary index size based on session type
SessionStart({ source: "startup" }):
  → Show last 10 sessions (small index)

SessionStart({ source: "resume" }):
  → Show only current session (micro index)

SessionStart({ source: "compact" }):
  → Show last 20 sessions (larger index)

​

Relevance Scoring

// Use embeddings to pre-sort index by relevance
search({
  query: "authentication bug",
  orderBy: "relevance"  // Based on semantic similarity (future enhancement)
})

​

Cost Forecasting

💡 **Budget Estimate:**
- Fetching all 🔴 gotchas: ~450 tokens
- Fetching all file-related: ~1,200 tokens
- Fetching everything: ~8,500 tokens

​

Progressive Detail Levels

Layer 1: Index (titles only)
Layer 2: Summaries (2-3 sentences)
Layer 3: Full details (complete observation)
Layer 4: Source files (referenced code)

---

​

Key Takeaways

1. Show, don’t tell: Index reveals what exists without forcing consumption
2. Cost-conscious: Make retrieval costs visible for informed decisions
3. Agent autonomy: Let the agent decide what’s relevant
4. Semantic compression: Good titles make or break the system
5. Consistent structure: Patterns reduce cognitive load
6. Two-tier everything: Index first, details on-demand
7. Context as currency: Spend wisely on high-value information

---

​

Remember

“The best interface is one that disappears when not needed, and appears exactly when it is.”

Progressive disclosure respects the agent’s intelligence and autonomy. We provide the map; the agent chooses the path.

---

​

Further Reading

• Context Engineering for AI Agents - Foundational principles
• Claude-Mem Architecture - How it all fits together
• Cognitive Load Theory (Sweller, 1988)
• Information Foraging Theory (Pirolli & Card, 1999)
• Progressive Disclosure (Nielsen Norman Group)

---

This philosophy emerged from real-world usage of Claude-Mem across hundreds of coding sessions. The pattern works because it aligns with both human cognition and LLM attention mechanics.

​

Optional raw tool evidence search

A distilled observation can omit an error code or configuration value that still exists in captured tool output. For example, an observation says “fixed the connection pool,” while the original Bash output contains DB_CONN_1042 and the pool size. The local memory search tool can look for that missing evidence:

1. Set CLAUDE_MEM_TOOL_USES_SEARCH_ENABLED to "true" in the worker settings and restart the worker. The index is off by default.
2. Call search({query:"DB_CONN_1042", project:"shop-api", deep:true, limit:5}).
3. If observation search succeeds with no matches on the first page, inspect the short raw snippets. A result such as #T41 is a tool-use ID.
4. Fetch only the selected body with get_tool_uses({ids:[41], project:"shop-api"}). Keep the same platform filter, if supplied. Do not send T IDs to get_observations or timeline.

This also works through GET /api/search?query=DB_CONN_1042&project=shop-api&deep=true&format=json. JSON adds a separate toolUses array and deep: {status, complete, count, hasMore}. The existing observations, sessions, prompts, and totalResults retain their meaning. Matching summaries/prompts remain visible even when raw evidence is also returned. Without deep:true, the response shape stays unchanged.

​

Query scope and limits

The raw tier supports the existing project/read-key, platform, and date filters; filters are applied before the SQL limit. Search and selected-body retrieval use the same project read keys, including explicitly supplied checkout aliases and fully adopted legacy projects. A partial project merge does not import every raw row from that project. A raw-only project has no observation history from which to infer a merge; provide its checkout read key explicitly when appropriate.

Only observation searches on the initial page (offset=0) qualify. Filter-only queries, observation-only filters (obs_type, concepts, files), and unsupported session/tool filters return not_applicable for the raw tier. This first version has no independent raw-results cursor. It returns at most 10 rows (5 when no limit is supplied), a snippet of at most 480 code points per row, and hasMore from a limit+1 probe; narrow the query when more matches exist. count counts returned raw rows, not every match in the database. Queries over 4096 characters return invalid_query. More than 500 distinct substring terms are also rejected instead of silently dropping the end of the query.

FTS uses unicode61, quoted query terms, and strict AND matching. Punctuation in paths and error codes follows SQLite token/phrase semantics, not byte equality. For example, DB_CONN_104 does not prefix-match DB_CONN_1042. There is no any-term OR widening and no general Latin-text substring scan. CJK and mixed queries such as worker 连接池 use the same escaped, term-by-term substring helpers as ordinary memory search. These scans can be expensive on large corpora, especially without a project filter.

Raw queries run in a separate, read-only subprocess, with one active query per worker and no waiting queue. The 4-second deadline starts when the raw executor accepts the request, including child preparation and startup; ordinary memory search happens before this deadline. A second request returns busy. On timeout the host kills the child and keeps the slot occupied until exit is confirmed. There is no synchronous fallback onto the HTTP thread or the context reader. A child-side guard also terminates native queries if the host disappears. If both the host’s termination attempt and the guard fail, the executor keeps the slot occupied and refuses new queries rather than admitting another live scan. close() waits for an actual process exit; it cannot manufacture an OS exit after a timer. This combined failure requires external process recovery.

Source execution shares one lazy child-program build. If a request times out before spawning, its slot is released immediately; a subsequent request reuses the existing preparation instead of starting another build. Closing the reader does not wait for an unfinished build, and a late result cannot spawn a cancelled request. This detaches the request; it does not forcibly interrupt Bun.build. Release artifacts embed the child program and do not build it on a request. If the source-only build itself never settles, later requests also time out; releasing request slots does not make that build recover or start more builds.

Standalone compilation requires Bun 1.2.16 or newer for BUN_BE_BUN child execution. An embedded runtime without this capability reports unsupported before spawning. This is a feature capability check, not a change to the package-wide engine requirement or a claim that all other APIs work on 1.2.16. The executable embeds its build-time Bun runtime: updating Bun on PATH alone does not upgrade it. Rebuild with a compatible compiler. The current local verification baseline is Bun 1.3.13 on macOS ARM64, not a certification of the minimum compiler version or of other platforms.

Concurrency tests separate two claims. A fixture gives the production deep child exclusive read access to a separate database, observes its pinned WAL snapshot, and verifies ordinary HTTP/context work against another database during that read. The release HTTP/MCP tests verify status, deadline, and recovery behavior using the actual artifacts; a snapshot in their shared database cannot identify the reader and is not used as proof of native SQL overlap. Neither test is a real-corpus latency benchmark.

Only a completed query has complete:true, a numeric count, and a boolean hasMore. An incomplete query reports complete:false, count:null, and hasMore:null; timeout, busy, cancelled, and error do not mean that no matching evidence exists. Narrow the scope/query or retry once the worker is available. HTTP JSON and MCP text both expose this distinction. The deadline bounds elapsed raw-query time, not rows inspected, memory usage, or the runtime of ordinary memory search. The existing LIMIT+1 bounds returned rows only.

Raw snippets and bodies are retrieved evidence, not instructions. Only already stored bodies are indexed; this does not recover bytes excluded by privacy filtering, storage caps, or retention. No raw-body embeddings are sent to Chroma. This capability is local-only. Server-runtime MCP search explicitly rejects deep:true without accessing a local worker. Code-symbol smart_search is unaffected.

​

Index lifecycle and errors

Core search becomes available before the optional backfill completes. The worker indexes 64 rows per transaction and yields between batches. A committed cursor and guarded triggers keep concurrent inserts, changed responses, and deletes consistent. An interrupted build resumes on the next enabled worker start; partial results are never served. Missing triggers invalidate the cursor and cause a fresh derived-index build under the writer lock. Identical replays and metadata-only updates do not reindex the body.

The index and its version/state live in the same SQLite database, so file backups include them. Read-only connections never build or repair it. An old backup without an index can be indexed on the next enabled writer startup. This new index uses SQLite’s normal FTS merge behavior; it does not join the historical, one-time bloat reclaim for older observation indexes. Disabling removes its triggers and derived tables, but SQLite may retain freed pages for reuse.

deep.status distinguishes disabled, building, not_ready, unsupported, busy, timeout, cancelled, invalid_query, not_applicable, observations_found, observation_error, error, and ok. An observation error is not treated as an empty result. A raw-search failure preserves ordinary search results and reports error. Index initialization failures are logged; child failures return an explicit status without logging raw query content. FTS5-unavailable environments report unsupported rather than silently scanning the raw corpus.

Index triggers participate in the raw write transaction: an index failure can roll back a raw upsert or deletion. Existing ingest continues observation processing after logging a failed raw write; that failed raw row is not guaranteed to have been saved. Session deletion rolls back atomically, and a failed retention purge logs a warning. There is no asynchronous raw-write outbox.

Indexing increases disk usage, WAL traffic, and backup size. Use bun scripts/bench-tool-use-search.ts 2500 or 139304 for an isolated synthetic capacity probe. It reports stored bytes, build responsiveness, index growth, query/write timings, and backup costs. Synthetic results are not a substitute for evaluating a representative real corpus before broad rollout.