AI Product Manager · Agents, workflows and evals

LLM products fail quietly. I ship the workflow and the checks that catch it.

Fig. 0The count

86 of 167 outputs failed a check.

All 86 caught before anything was published.

CrossSource v0.1 and v0.3 · mirror-eval · screener-eval

Fig. 0The audit sheet

failed a check passed

Each square is one model output a check read, from the three case studies on this site. Squares are grouped by outcome, so a square's position is not an item id. Hover or arrow through a square to see which harness, what it was, and which check read it.

Zoeb Nomi
Fig. 1. Zoeb Nomi, Bengaluru.

I'm Zoeb Nomi, a product manager at Instead, an AI-native tax research and planning platform. I work on a multi-step tax workflow from both sides: I test whether its steps stay coherent from first input to final memo, and I cut its run time from 10 to 15 minutes to under a minute. In tax research, a wrong citation isn't a UX bug. It's a compliance risk.

30 min · intro call

What my agents shipped

Fig. 0 counts the checks. This counts what shipped, on a date set when the work was filed.

  • 12

    pieces of work shipped since the system went live on 10 Nov

  • 90%

    on or before the due date set at filing

    9 of 10 with a due date

  • $0.07

    API spend per shipped item, failures included

Every item passes a written acceptance check and a secret scan before it reaches me, and I review and merge each one myself: median 4.2 hours from ready to merged.

Next: what my builds were measured on

Shipped, then measured

Fig. 0, the first band. The 15 CrossSource judge verdicts checked against a person are what let R2 be read as a number. A check is only real if it produces a number someone else can read: three of them here, from three different systems.

  • R1 · Run time

    <1 min

    Datum 10–15 min before change about −90% production runs, count not public

    run time of a multi-step tax workflow, after scoping edits to changed inputs

    Instead, production

  • R2 · Citation precision

    0.994 ✓

    Datum 0.981 baseline change +0.013 n = 25

    citation precision with strict citation prompting; recall 0.760

    CrossSource v0.1

  • R3 · Revenue

    $2.7M MRR

    Datum 0, from zero change +$2.7M MRR 28 US enterprise clients

    from zero, Background Verification on the Checkr API

    Keka HR

Next: the builds these readings came from

Builds

Fig. 0, four of the six bands. Four of the six bands on the sheet came out of two builds below, CrossSource and screener-eval; the other two came out of mirror-eval, on the full ledger. A number means little without the system it checked. Every build is drawn the same way, so you can compare them.

What I built, how it is guarded, what it was measured against.

Select a row to open it.

  • 01TradeFrame[guarded automation]A personal trading system where an AI judgment layer proposes trades and a deterministic layer decides what may run.Measuredproposevalidateconfirmexecute and record
    Result
    Every decision is scored against a do-nothing baseline; returns are reported net of all costs.
    Guarded
    Hard rails and breachable bands; a kill switch per book that can stop new risk but never exits or stops; a person confirms each US order; India orders run only after dry-run validation.
    Shipped
    Shipped Aug 2026 (before ship-date tracking)

    Read the guide →Repo →

  • 02cheap-tier-legal-tax-hr[counted]Where a 10-cent model can sit in a tax, legal or HR workflow, and what has to back it up.Measured99 of 1001,414 expert-labelled items
    Result
    5 models, 1,414 expert-labelled items. At a stated confidence of 0.9 or more, Haiku was sure on 28 of 100 answers and right on 99 of 100 of those; three other cheap models were sure on 92 to 93 of 100 and wrong on 15 to 16 of 100 of those.
    Guarded
    Compares four routings per domain: one cheap model, a confidence gate, a second reader, a frontier referee.
    Shipped
    Shipped Oct 2026 (before ship-date tracking)

    Read the note →Repo →

  • 03CrossSource[judged]An open harness that checks whether a legal RAG answer's citations support its claims.Measured0.981 → 0.994n = 25
    Result
    v0.1: precision with strict citation prompting, n = 25, recall 0.760 capped by retrieval; judge checked blind against a person, 15/15. v0.3: 150 items over 45 opinions, two blind model raters, kappa 0.98 / 0.84.
    Guarded
    n/a: measurement only
    Shipped
    Shipped 20 Oct 2026, v0.3 (before ship-date tracking)

    Case study →Repo →Citable DOI →

  • 04screener-eval[counted]A paired, repeated test of what an LLM résumé screener actually rewards.Measured22 points885 calls
    Result
    885 calls over 21 postings, 2 screeners, 5 repeats. Employer swaps and deleted links moved the score about a point or less; which screener read it moved it 22 points.
    Guarded
    n/a: measurement only
    Shipped
    Shipped Sep 2026 (before ship-date tracking)

    Case study →Repo →

One system where the AI proposes and code decides, and three of my five eval harnesses. Every new build is added with the date I planned to ship it and the date I did.

Next: the method, one step at a time

How I work

Fig. 0 is 01, drawn. Decide what good output means, then count it: every square on the sheet is one such count. 02 to 04 are the work around it.

  1. Measure

    Decide what good output means, then count it. At Instead: flag, classify by failure mode, review weekly, on a four-part rubric of answerability, accuracy, citation quality and actionability.

  2. Trace

    Every answer can be followed back to its source, and every decision leaves a record. At Instead every cited source carries its document ID and sub-type; in TradeFrame every decision is scored against doing nothing.

  3. Guard

    The model proposes; rules decide what runs. A step runs without a person only when its miss rate holds under the bar across releases, and any judge that scores it has been checked against people.

  4. Ship

    Put it in front of users, then keep measuring. A failure taxonomy says whether the next fix belongs to the prompt, the retrieval or the workflow.

A rule for autonomy. A step runs without a human only when its miss rate on the labelled sample holds under the bar across releases, and any judge used to score it has its own measured agreement with humans.

From Nobody Reports the Misses →

Next: the code behind them

PM who ships code

Fig. 0, in public. Every harness behind the sheet is a public repo. These are the commits behind every build, by day.

Every build above is a public repository. This is the last year of commits, live from GitHub.

PM who ships code

Back to Fig. 0. The one cell I cannot draw yet: your product. Fig. 0 ↑

NOW: PM AT INSTEAD

BEFORE: $2.7M MRR FROM ZERO AT KEKA HR

If you're hiring a PM for an agent or LLM product, I'd like to hear what it has to get right.

30 min · intro call