Builds

What I built, and how I know it works

Each row is one build. It says what the system does, the rule that stops it doing the wrong thing, the number it was measured on, and where the code is. Builds started after 10 Nov 2026 also show the date I planned to ship and the date I did.

Guarded
the rule or person that stops a bad action.
Measured
the number, its denominator, and how the scorer was checked.
Shipped
planned vs actual, where recorded.

Systems

agents, workflows, guarded automation

01

TradeFrame

[guarded automation]

A personal trading system where an AI judgment layer proposes trades and a deterministic layer decides what may run.

Fig. 1Who decides what runs. Dashed: proposes only. Solid: decides.
AI judgment layerProposes trades. Never executes.
Deterministic layerValidates against hard rails and breachable bands.
US bookA person confirms each order.
India bookRuns only after dry-run validation.
Kill switch, per bookCan stop new risk. Never exits or stops.
Decision logEvery decision scored against a do-nothing baseline; returns net of all costs.
Guarded
Hard rails and breachable bands; a kill switch per book that can stop new risk but never exits or stops; a person confirms each US order; India orders run only after dry-run validation.
Measured
Every decision is scored against a do-nothing baseline; returns are reported net of all costs.
Shipped

Shipped Aug 2026 (before ship-date tracking)

Repo →Guide →

Eval harnesses

five harnesses

Why this matters: a harness is how a step earns the right to run without a person.

[judged] scored by an LLM judge that was checked against people · [counted] counted, no judge · [null] the judge failed its check, so the result is reported as bands.

  • 01cheap-tier-legal-tax-hr[counted]Where a 10-cent model can sit in a tax, legal or HR workflow, and what has to back it up.GuardedCompares four routings per domain: one cheap model, a confidence gate, a second reader, a frontier referee.Measured99 of 1001,414 expert-labelled itemsShippedShipped Oct 2026 (before ship-date tracking)
    Result
    5 models, 1,414 expert-labelled items. At a stated confidence of 0.9 or more, Haiku was sure on 28 of 100 answers and right on 99 of 100 of those; three other cheap models were sure on 92 to 93 of 100 and wrong on 15 to 16 of 100 of those.

    Read the note →Repo →

  • 02CrossSource[judged]An open harness that checks whether a legal RAG answer's citations support its claims.Guardedn/a: measurement onlyMeasured0.981 → 0.994n = 25 · 15/15 blind agreement with a personShippedShipped 20 Oct 2026, v0.3 (before ship-date tracking)
    Result
    v0.1: precision with strict citation prompting, n = 25, recall 0.760 capped by retrieval; judge checked blind against a person, 15/15. v0.3: 150 items over 45 opinions, two blind model raters, kappa 0.98 / 0.84.

    Case study →Repo →Citable DOI →

  • 03screener-eval[counted]A paired, repeated test of what an LLM résumé screener actually rewards.Guardedn/a: measurement onlyMeasured22 points885 callsShippedShipped Sep 2026 (before ship-date tracking)
    Result
    885 calls over 21 postings, 2 screeners, 5 repeats. Employer swaps and deleted links moved the score about a point or less; which screener read it moved it 22 points.

    Case study →Repo →

  • 04mirror-eval[null]A harness that measures what four AI search engines say about a person, twice.Guardedn/a: measurement onlyMeasured13 → 6363 search probes · judges 3/40 and 12/40, so bands onlyShippedShipped Sep 2026 (before ship-date tracking)
    Result
    332 probes per wave. Perplexity cited the canonical site on 13 → 63 of 63 search probes; Claude 0 → 0. LLM judges matched blind human labels on 3/40 and 12/40, so the taxonomy is reported as bands.

    Case study →Repo →Release v1.0 →

  • 05decisions-api-legal-quiz[counted]544 LegalBench questions through OpenAI's Decisions API, to see if any confidence level makes checking safe to skip.GuardedFinding: no confidence threshold in this data was safe to stop checking.Measured88%n = 544ShippedShipped Oct 2026 (before ship-date tracking)
    Result
    right at 90% stated confidence, n = 544.

    Read the essay →Repo →

Method

Three harnesses, one method.

Five harnesses are listed above; three of them share one method and are compared here side by side: legal RAG citations, what AI search says about a person, and LLM résumé screeners. Each row is one harness: the question it answers, the number it stands on, and how it was run.

Build the judge. Validate it blind against a human. Count what you can. Report the rest as bands.

Fig. 2 Judge verdicts that matched a blind human label.
  • CrossSource judge 15 of 15 [judged]
  • mirror-eval, judge 1 3 of 40 [null]
  • mirror-eval, judge 2 12 of 40 [null]
  • screener-eval no judge [counted]

CrossSource v0.1, stratified blind sample, hand-graded. mirror-eval: two LLM judges from different model families against 40 blind labels. screener-eval has no judge.

Table 1. The three harnesses compared
Harness QuestionJudgeCounted metricSampleVerdict
CrossSource When the RAG cites a court opinion, does that opinion support the claim?v0.3: two blind, independent model raters (kappa 0.98 citation_check, 0.84 qa_citation), disagreements plus a 10% audit sample (30 items in all) adjudicated by me. v0.1: LLM judge blind-validated against a human, 100% (15/15)v0.1 (n = 25): claim-level citation precision 0.981 → 0.994; recall 0.760, capped by retrievalv0.3: 150 items · 45 opinions. v0.1: 22 opinions · 25 questionsv0.1: judge passed and caught a harness bug; prompting buys precision, retrieval owns recall
mirror-eval What do AI search engines say about me, and is it true and sourced to something I control?Two LLM judges from different model families: failed blind validation, 3/40 and 12/40Probes citing an owned surface, of 63 search-mode probes per engine: counted, judge-free4 engines · 63 search probes each · 2 wavesPerplexity 13 → 63, Claude 0 → 0; index access explains the split
screener-eval Do employer names and evidence links change what an LLM résumé screener scores?None: nothing is judged by a modelPaired delta in fit score (B − A), 10,000-resample bootstrap 95% CI1 résumé · 21 postings · 2 screeners · 885 callsEmployer swap and link deletion each moved the score by about a point or less; which screener read it moved it 22

One judge passed and caught a bug. One judge failed and the study survived on counts. One needed no judge at all. It is the same method I use on anything I build before it runs unchecked.

Contributions

pull requests to other people's tooling

  • 01TaxCalcBench PR #157An audit of a public tax-calculation benchmark's 101 test cases against the official instructions.Guardedn/aMeasured2 answer-key errors and 5 input conflicts found; an opt-in scoring flag proposed. Open, not merged.ShippedShipped Oct 2026 (before ship-date tracking)
  • 02eyecite PR #360A fix to legal-citation extraction for "Id." references past explanatory parentheticals.GuardedRegression tests for three review findings.MeasuredExtraction benchmark: 0 gains, 0 losses on 15,988 citations. Open, not merged.ShippedShipped Oct 2026 (before ship-date tracking)

If a row has no number, it says why. Nothing on this page is a demo.

Shipping numbers

since 10 Nov 2026

What my agents shipped, counted.

Every piece of work I hand to my agent system is filed with an acceptance line and a due date. This section counts what landed, what it cost and how long it waited for me.

  • 12

    pieces of work shipped since the system went live on 10 Nov

  • 90%

    on or before the due date set at filing

    9 of 10 with a due date

  • $0.07

    API spend per shipped item, failures included

  • 4.2 h

    median time from ready to merged by me

A shipped item costs $0.07 in API spend, failures included.

Every item passes a written acceptance check and a secret scan before it reaches me, and I review and merge each one myself: median 4.2 hours from ready to merged. acceptance check passed: 12 of 12

Table 2. Six of the 12 items
Week KindAgent minSlip (d)API $Model
9 Nov judgement3.7−20Claude plan
9 Nov judgement19.6+10Claude plan
9 Nov mechanical30.5−30.05GLM
9 Nov mechanical13.700.13GLM
9 Nov mechanical26.8−20.13GLM
9 Nov mechanical6.4−10.11GLM

No titles, project names or links: the work is private. Slip is days against the due date set at filing; negative is early. Late is not marked in red: red is for annotation and action.

Pilot figures, 10 to 11 Nov 2026. Pilot: two days of data; updates when the tracker runs.

NOW: PM AT INSTEAD

BEFORE: $2.7M MRR FROM ZERO AT KEKA HR

If you're hiring a PM for an agent or LLM product, I'd like to hear what it has to get right.

30 min · intro call