Design against these at Level 4 — before they compound at Level 5.
| Challenge | How it manifests | Mitigating design pattern |
|---|---|---|
| Agentic Loops | A coding agent "fixes" a failing test by breaking another, triggering a fix-and-fix cycle that never converges and quietly consumes compute & time. | Circuit-breaker patterns: cap repair-loop iterations; require agents to reason over the full trajectory; force human escalation after repeated failed convergence. |
| Agent-to-Agent Contradiction | An architecture agent proposes a cost-saving monolith while a requirements agent pushes microservices — both well-reasoned, yet structurally incompatible. | A designated arbitration layer that detects conflicting proposals before execution and either checks them against explicit priority rules or escalates to a human governor. |
| Trajectory Review Overload | As autonomous decision volume grows, the number of agent trajectories that could warrant human audit grows just as fast — exception-based review reverts to reviewing everything (unsustainable) or nothing (unsafe). | Risk-weighted trajectory sampling: audit depth scales with blast radius and novelty; continuously re-tune exception thresholds as an ongoing governance activity. |
| Skill Rot | Engineers who have spent years as supervisors find they can no longer diagnose a production failure, override a bad agent decision, or reason about the underlying system when autonomy fails. | Deliberate skill-maintenance programs: rotation into hands-on work, simulated failure drills, standing pools of senior practitioners, and periodic Unplugged Days. |
| Gaming the Feedback Loop | Users learn to trigger the behaviors a closed loop optimizes — synthetic clicks, fake usage patterns, metric-padding workflows — driving features & architecture changes on signals that look like demand but are not. | Separate signal validation from signal reaction; corroborate usage patterns across multiple independent data sources; require human approval for structural changes driven by behavioral signals alone. |
| Autonomy Boundaries for Critical Systems | A stock-trading platform, core banking ledger, or health-records system reaches Level 4 capability — but regulators, customers, and boards refuse to let it run autonomously at Level 5. | Tier the autonomy model by system criticality: consumer-facing apps may reach full adaptive maturity; trading, financial, healthcare, and infrastructure systems permanently cap at supervised Level 4 with human-mandatory gates. |
Three themes, one common thread
Three themes run through the table: coordination failures (loops, contradictions), human capacity limits (review overload, skill rot), and trust boundaries (gaming signals, critical-system caps). None are fixable by better prompts alone — each requires deliberate architectural design before Level 5 is claimed.
They share a common thread: they are not edge cases — they are direct consequences of how Levels 4 and 5 are architected. Left unaddressed, trajectory review overload in particular becomes a Governance Paradox that mirrors the Productivity Paradox: escaping the code-review bottleneck at Level 1 delivers little lasting value if the organization simply recreates an equivalent bottleneck as an unscoped audit queue at Level 5.
The autonomy-boundaries question reframes who the real consumers of Level 5 actually are. A social feature in a consumer app and a settlement engine in a global bank both run software — but they do not carry the same cost of a wrong autonomous change. The framework must distinguish adaptive consumer systems (where continuous self-optimization is the competitive advantage) from governed critical infrastructure (where the goal may be Level 4 forever, with humans permanently holding veto power over structural evolution).
Claiming Level 5 maturity organization-wide without making this distinction is not ambition — it is a category error. Treat multi-agent conflict resolution, audit-effort scaling, skill retention, signal integrity, and criticality-tiered autonomy as first-class architectural concerns at Level 4 — before they compound into Level 5.