3 Perception: What your agent sees determines what it does

Perception is presented as the main bottleneck in agent performance: what matters is not how much information an agent can accept, but how well it selects, prioritizes, and shapes that information before reasoning begins. The chapter opens with a failure case in which an agent was given far too much repository context, causing it to focus on irrelevant material and miss the actual defect. From this, the central idea emerges that a context window is not a neutral storage space but a constrained attention system, so loading the wrong evidence can make an otherwise capable model act confidently on the wrong basis.

The chapter then develops a framework for managing this constraint through four perception patterns. Context triage sorts evidence into priority tiers so the most important material enters first and deferrable material stays out until needed. Semantic compaction reduces accumulated history while preserving decision-critical details, especially failure evidence that would otherwise be lost. Progressive discovery explores unknown environments in stages, starting broad and cheap, then focusing on relevant files or sources, and finally deepening through references. Multi-modal fusion chooses the representation that best preserves the evidence needed for the next decision, whether that means keeping an image, converting structured data into a compact table, or using text. Across all four, the goal is to maximize signal, minimize waste, and avoid iatrogenic context.

To make the ideas operational, the chapter emphasizes observability and gives concrete implementation guidance through the evolving Argus reviewer. It recommends tracking re-read ratio, token spend per successful outcome, and attention-zone coverage to detect when perception policies are helping or hurting. It also shows how these patterns can be composed into a full pipeline that gathers context, filters it, compacts it, and keeps it aligned with the task as the session evolves. By upgrading Argus from a blind prompt wrapper into a context-aware analyst, the chapter argues that strong agent design depends on deliberate perception choices that determine what the system sees, what it can remember within the session, and ultimately what it can do well.

Perception quality as a function of context fill: quality rises with information, peaks, then degrades as attention dilutes. The curve illustrates the concept; the peak’s position varies by task and model.

Context Triage priority tiers. Information flows downward from most critical to most deferrable. Percentages show an example budget and require workload calibration.

Semantic compaction at three levels. Each level preserves more recent turns and compresses everything before them.

Progressive discovery's three-phase strategy applied to a debugging task.

Multi-modal fusion decision flow. Text and structured data are converted to compact text representations.

The perception pipeline.

• Perception is a core architectural discipline of agent engineering. A well-chosen context can allow a capable model to outperform a stronger model whose evidence is buried in noise.
• Context Triage (Perception × Route) classifies information into four priority tiers (P0 through P3) and fills the window from highest priority down. The key design decision is what can be deferred safely, who sets that policy, and when the agent may promote an item for retrieval.
• Semantic Compaction (Perception × Chain) compresses accumulated context at three levels (tool clearing, conversation summarization, progressive summarization). Preserve decision-critical evidence until the relevant decision is resolved; losing failure evidence can cause the agent to repeat mistakes.
• Progressive Discovery (Perception × Loop) builds context through a forage-focus-deepen cycle. Place exploration cost deliberately in per-task search, session setup, or maintenance of a persistent index.
• Multi-Modal Fusion (Perception × Parallel) chooses a token-efficient representation for each input. Keep images when spatial information is the signal; convert symbolic or nonspatial evidence to compact text when the decision does not require the original form.
• The four patterns can compose into a perception pipeline that shifts from acquisition (early cycles, dominated by Progressive Discovery) to management (late cycles, dominated by Semantic Compaction). A system selects only the stages its evidence and task require.
• Three metrics help measure perception quality: repeated retrieval, token spend per successful outcome, and reliable use of critical evidence. Establish workload and model baselines, then calibrate thresholds against them.
• The underlying principle is to aim the context window at decision-relevant evidence.

FAQ