AI Product Manager · Agents, workflows and evals
LLM products fail quietly. I ship the workflow and the checks that catch it.
Fig. 0The count
86 of 167 outputs failed a check.
All 86 caught before anything was published.
CrossSource v0.1 and v0.3 · mirror-eval · screener-eval
failed a check passed
Each square is one model output a check read, from the three case studies on this site. Squares are grouped by outcome, so a square's position is not an item id. Hover or arrow through a square to see which harness, what it was, and which check read it.
I'm Zoeb Nomi, a product manager at Instead, an AI-native tax research and planning platform. I work on a multi-step tax workflow from both sides: I test whether its steps stay coherent from first input to final memo, and I cut its run time from 10 to 15 minutes to under a minute. In tax research, a wrong citation isn't a UX bug. It's a compliance risk.
30 min · intro call
What my agents shipped
Fig. 0 counts the checks. This counts what shipped, on a date set when the work was filed.
-
12
pieces of work shipped since the system went live on 10 Nov
-
90%
on or before the due date set at filing
9 of 10 with a due date
-
$0.07
API spend per shipped item, failures included
Every item passes a written acceptance check and a secret scan before it reaches me, and I review and merge each one myself: median 4.2 hours from ready to merged.
Shipped, then measured
Fig. 0, the first band. The 15 CrossSource judge verdicts checked against a person are what let R2 be read as a number. A check is only real if it produces a number someone else can read: three of them here, from three different systems.
-
R1 · Run time
<1 min
Datum 10–15 min before change about −90% production runs, count not public
run time of a multi-step tax workflow, after scoping edits to changed inputs
Instead, production
-
R2 · Citation precision
0.994 ✓
Datum 0.981 baseline change +0.013 n = 25
citation precision with strict citation prompting; recall 0.760
CrossSource v0.1
-
R3 · Revenue
$2.7M MRR
Datum 0, from zero change +$2.7M MRR 28 US enterprise clients
from zero, Background Verification on the Checkr API
Keka HR
Builds
Fig. 0, four of the six bands. Four of the six bands on the sheet came out of two builds below, CrossSource and screener-eval; the other two came out of mirror-eval, on the full ledger. A number means little without the system it checked. Every build is drawn the same way, so you can compare them.
What I built, how it is guarded, what it was measured against.
Select a row to open it.
01TradeFrame[guarded automation]A personal trading system where an AI judgment layer proposes trades and a deterministic layer decides what may run.Measuredproposevalidateconfirmexecute and record
- Result
- Every decision is scored against a do-nothing baseline; returns are reported net of all costs.
- Guarded
- Hard rails and breachable bands; a kill switch per book that can stop new risk but never exits or stops; a person confirms each US order; India orders run only after dry-run validation.
- Shipped
- Shipped Aug 2026 (before ship-date tracking)
02cheap-tier-legal-tax-hr[counted]Where a 10-cent model can sit in a tax, legal or HR workflow, and what has to back it up.Measured99 of 1001,414 expert-labelled items
- Result
- 5 models, 1,414 expert-labelled items. At a stated confidence of 0.9 or more, Haiku was sure on 28 of 100 answers and right on 99 of 100 of those; three other cheap models were sure on 92 to 93 of 100 and wrong on 15 to 16 of 100 of those.
- Guarded
- Compares four routings per domain: one cheap model, a confidence gate, a second reader, a frontier referee.
- Shipped
- Shipped Oct 2026 (before ship-date tracking)
03CrossSource[judged]An open harness that checks whether a legal RAG answer's citations support its claims.Measured0.981 → 0.994n = 25
- Result
- v0.1: precision with strict citation prompting, n = 25, recall 0.760 capped by retrieval; judge checked blind against a person, 15/15. v0.3: 150 items over 45 opinions, two blind model raters, kappa 0.98 / 0.84.
- Guarded
- n/a: measurement only
- Shipped
- Shipped 20 Oct 2026, v0.3 (before ship-date tracking)
04screener-eval[counted]A paired, repeated test of what an LLM résumé screener actually rewards.Measured22 points885 calls
- Result
- 885 calls over 21 postings, 2 screeners, 5 repeats. Employer swaps and deleted links moved the score about a point or less; which screener read it moved it 22 points.
- Guarded
- n/a: measurement only
- Shipped
- Shipped Sep 2026 (before ship-date tracking)
One system where the AI proposes and code decides, and three of my five eval harnesses. Every new build is added with the date I planned to ship it and the date I did.
How I work
Fig. 0 is 01, drawn. Decide what good output means, then count it: every square on the sheet is one such count. 02 to 04 are the work around it.
-
Measure
Decide what good output means, then count it. At Instead: flag, classify by failure mode, review weekly, on a four-part rubric of answerability, accuracy, citation quality and actionability.
-
Trace
Every answer can be followed back to its source, and every decision leaves a record. At Instead every cited source carries its document ID and sub-type; in TradeFrame every decision is scored against doing nothing.
-
Guard
The model proposes; rules decide what runs. A step runs without a person only when its miss rate holds under the bar across releases, and any judge that scores it has been checked against people.
-
Ship
Put it in front of users, then keep measuring. A failure taxonomy says whether the next fix belongs to the prompt, the retrieval or the workflow.
A rule for autonomy. A step runs without a human only when its miss rate on the labelled sample holds under the bar across releases, and any judge used to score it has its own measured agreement with humans.
PM who ships code
Fig. 0, in public. Every harness behind the sheet is a public repo. These are the commits behind every build, by day.
Every build above is a public repository. This is the last year of commits, live from GitHub.
PM who ships code
Back to Fig. 0. The one cell I cannot draw yet: your product. Fig. 0 ↑
NOW: PM AT INSTEAD
BEFORE: $2.7M MRR FROM ZERO AT KEKA HR
If you're hiring a PM for an agent or LLM product, I'd like to hear what it has to get right.
30 min · intro call