AssayAssay

The capstone

One agent, built the whole way.

The ten courses each teach one idea. The capstone is the idea threaded into a single project you can clone, run, test, and deploy: a customer-support agent built across the entire curriculum, from an evaluation baseline to a release gate, monitoring, and a factsheet. Framework-free — the loop is about 120 lines of plain TypeScript on one fetch call — with a Python port of the core.

Download the project (.tgz)The courses it draws on

Run it

Everything deterministic runs with no API key. Download the project above, then:

tar xzf capstone.tgz
cd capstone && npm install

npm run typecheck        # strict TS, no errors
npm test                 # 12 tests, no API key needed
npm run gate:offline     # the release gate, replaying recorded traces

With a free NVIDIA key, run the agent and the live gate:

cp .env.example .env     # add your NVIDIA_API_KEY
npm run agent -- "Where is my order A-4417?"
npm run gate             # live
npm run factsheet        # generated from the tool config, then verified

Where each course shows up

Evalsevals/ — dataset, graders, the release gate with a Wilson interval
Agent loopssrc/agent/loop.ts — the loop, stopping conditions, repeat detection
Loop engineeringthe checker/sensor loop in src/harness/sensors.ts
Harness engineeringsrc/harness/ — guides (feedforward) and sensors (feedback)
Graph engineeringsrc/graph/runtime.ts + intake routing in src/helpdesk.ts
Agent patternsthe routing-then-act shape of handle()
Retrievalsrc/retrieval/ — offline TF-IDF store with an authorization-filter seam
Industriesthe support domain, its tiers and refusal boundaries
Operationssrc/agent/policy.ts (tiers), src/factsheet.ts, DEPLOYMENT.md
Agent securitytier-3 withheld, secrets-out-of-context and egress notes in DEPLOYMENT.md

The arc: baseline, diagnose, improve

This repository shipped by walking the exact loop the curriculum teaches. The history is the point.

  1. 01Baseline. The first gate run scored 5/8. Deletions were answered without calling escalate; refunds were described rather than staged.
  2. 02Diagnose. Both were harness gaps, not model failures. The sensors check for the tool call, because “said the right thing” and “did the right thing” are different.
  3. 03Fix the guides. The system prompt gained two explicit routing guides: stage refunds, escalate deletions. Deletions and one refund fixed.
  4. 04Fix the eval too. The remaining refund failures were the agent correctly asking for an unspecified amount — a bad test case, not a bad agent. The dataset now states amounts. 8/8.
  5. 05And still it fails. 8/8 is 100%, but the 95% interval runs 68–100% — below the 80% bar. Not a bug: a pass rate without enough evidence is not a release criterion. The fix is more cases, not a higher number.

That last step is the one to sit with. A green board that still fails the gate is the score-confidence lesson made concrete — and it is why the capstone ships with the gate wired into CI as an informational step, not a rubber stamp.

Before you ship it

The project ships with a full deployment checklist (DEPLOYMENT.md) — risk-tier every action, bound by design, least privilege, keep secrets out of context, allowlist egress, pass the gate on a lower interval bound, roll out gradually, emit traces, publish the factsheet, monitor on multiple layers, and a three-line incident runbook. It is educational, not production-hardened as-is; read the checklist before shipping.