The agent's monitors are the spine of the hub.

#00 The problem we're solving

The user has an agent in production. They want it to keep getting better against the things they care about — accuracy, cost, latency, robustness, drift, capability.

The mechanism is monitors. Each one:

• Tracks one metric (accuracy / cost / latency / robustness / …).
• Knows the user's target (e.g. 95% accuracy by Jul 14).
• Knows the user's budget (e.g. don't ship anything that pushes cost over $0.10/task).
• Listens to signals — production tasks, HITL corrections, golden-set replay, synthetic edges.
• Surfaces work when it has a fix to propose, or escalates when it needs help.

The interface gives the user the tools to do three jobs:

1. Train — provide labels, corrections, golden examples. (Datasets tab.)
2. HITL optimize — review monitor findings, approve / reject / rollback. (Findings tab.)
3. Audit — read the immutable log of every change, signal, and rollback. (Audit tab.)

Each monitor runs in one of two operational modes:

Why both modes: the trust ladder. Users start in HITL-Approve, see the monitor working well, and graduate it to Continuous when they trust it. Trust is per-monitor — Accuracy can be Continuous while Cost is still HITL.

#01 Sub-navigation — which tabs make sense

The per-agent Learning route had Findings · Tuning · Audit · Datasets as historical labels. After round-1 sparring, here's what each tab is and which to keep:

Flow → tab mapping:

#02 Locked principles

Every variant in this doc is checked against these.

#03 Flow map

Hot path (thick stroke) = first-time owner setting up the first goal. Clay nodes = open forks. Dashed edges = loops or the alternative spine.

A1 · Empty hub — locked

First-run state. The hero CTA replaces today's "Tool optimisation" card; the 4 empty axis tiles replace today's "Test & eval / Agent tuning" Coming-soon pillars. Goals not pillars (P1, P2).

A2 · Configure (setup affordance) — LOCKED on Variant B (modal)

The customization screen that was missing from the dashboard. Same form across every variant — what differs is where the form lives. Locked on Variant B (centered modal) in round 1: "helps focus on configuration." The other variants are kept visible with their reasons so future readers see the alternatives explored.

Trade-off table

A3 · 1-of-4 (between-state) — locked

Between "I clicked save" and "Beam has data" there's a several-minute gap. The Accuracy tile's "Gathering first measurement…" state is the only signal the goal actually landed in a real measurement loop. P8.

A4 · Edit affordance — LOCKED: same modal as setup + inline mode-pill

Edit-existing-configuration uses the same centered modal as A2 setup, populated with current values + an edit-history meta strip. For the most frequent edit (mode change → graduation Approve → Continuous), a 1-click inline mode-pill popover lives on the monitor tile. The wireframe below still shows the right-rail drawer pattern — slated for redraw next round to match the locked A2 modal.

B1 · Monitor hub (multi-monitor dashboard) — locked · in Figma

Two weeks in. Full mission. All 4 monitors populated. Per-tile ⋯ opens the edit modal (same as A2).

B2 · Monitor detail (drill-in · Open Accuracy) — locked

Per-monitor full page. KPI + sparkline, scoped findings, side-rail with monitor settings + Focus tools (pulled from live Flow page node-quality). Pending: flatter drill-in variants per round-1 feedback — full-page nav reads as too deep. Three flatter bets to draw next: inline expand-in-place, sticky detail panel under the hub, or "no drill-down" (everything important on the tile).

C1 · Combined Inbox — primary · locked round 1

Promoted from alternative spine → primary surface. Confirmed in round 1: "I very much like the inbox. This is also not bad, because this is something we got as feedback in the user interview as well. If there is something, just surface it up." The inbox is where the monitors push things that need the user — it's the partner surface to the monitor hub, not a fallback.

Cockpit vs Inbox — when each wins

C2 · Finding detail — 3 variants · awaiting pick

What happens when the user clicks a finding from the C1 inbox to review it. This is the HITL decision screen — where the user actually says ship / reject. The form is the same in every variant (diff + validation + decision bar); what differs is where the screen lives and how it interrupts the inbox.

D1 · Continuous mode — autonomous visibility

When a monitor is in Continuous mode, the user isn't blocked — the monitor ships safe changes itself. But the user still wants visibility into what's happening. This is the monitor's surface when no human decision is required.

E1 · Datasets — verification & training data

The substrate the monitors verify against. Three sources: production (promoted runs), upload (JSONL / CSV), synthetic (rule-generated edge cases). Each monitor picks a dataset to verify against in its config (see A2 modal · Verify field).

F1 · Audit log — immutable history

Every change, every signal, every rollback. Read-only. Exportable. Filterable. This is the page someone reaches when they need to answer "what happened, who did it, when, and can we revert it?"

#14 Sub-nav variants — 3 options · awaiting pick

The sub-nav itself has a fork. The current pick in #01 was object-based — but two alternatives are worth surfacing so you can react.

#15 Metrics philosophy — accuracy isn't enough

From round-2 sparring: "Accuracy is a good metric, but initially when the agent is trained on a small use case, its accuracy is gonna be high. Then as we increase the scope of the agent… its accuracy will go down, even though the agent became better."

Right. Bare accuracy rewards staying narrow. An agent at 95% on 5 task types looks "better" than one at 80% on 50 — but the second is doing 8× the work. Three ways the UI can honor this:

How a Coverage monitor reads on the hub

#16 A4 · Edit existing monitor — modal redraw (replaces S5 drawer)

Same centered modal as A2 (locked round 1) — populated with current values, edit-history strip at the top, mode-graduation hint inline, plus a small Delete monitor action in the footer that A2 setup doesn't need.

#17 E2 · Dataset detail — examples + usage + pass rate

Click into a dataset from E1. Three sub-tabs: Examples (the rows), Used by (which monitors verify against it), Pass rate (per-monitor accuracy on this dataset over time).

#18 E3 · Curate — build / add examples to a dataset

Add examples to a dataset from three sources: promote from production runs (most common — user sees a failure in the inbox, decides "we should test for this"), upload JSONL/CSV, or generate synthetic edges. Per example: label, must-pass flag, edge-case flag.

#19 E4 · Use this dataset — 3 actions

From E2's "Use this dataset →" CTA. A dataset can be used three different ways: