Each row is one build. It says what the system does, the rule that stops it doing the wrong thing, the number it was measured on, and where the code is. Builds started after 10 Nov 2026 also show the date I planned to ship and the date I did.
Guarded
the rule or person that stops a bad action.
Measured
the number, its denominator, and how the scorer was checked.
Shipped
planned vs actual, where recorded.
§1Systems
agents, workflows, guarded automation
01
TradeFrame
[guarded automation]
A personal trading system where an AI judgment layer proposes trades and a deterministic layer decides what may run.
Fig. 1Who decides what runs. Dashed: proposes only. Solid: decides.
1AI judgment layerProposes trades. Never executes.
→
2Deterministic layerValidates against hard rails and breachable bands.
→
3US bookA person confirms each order.
3India bookRuns only after dry-run validation.
Kill switch, per bookCan stop new risk. Never exits or stops.
4Decision logEvery decision scored against a do-nothing baseline; returns net of all costs.
1propose2validate3confirm4execute and record
Guarded
Hard rails and breachable bands; a kill switch per book that can stop new risk but never exits or stops; a person confirms each US order; India orders run only after dry-run validation.
Measured
Every decision is scored against a do-nothing baseline; returns are reported net of all costs.
Why this matters: a harness is how a step earns the right to run without a person.
[judged] scored by an LLM judge that was checked against people · [counted] counted, no judge · [null] the judge failed its check, so the result is reported as bands.
#BuildGuardedMeasuredShipped
01cheap-tier-legal-tax-hr[counted]Where a 10-cent model can sit in a tax, legal or HR workflow, and what has to back it up.GuardedCompares four routings per domain: one cheap model, a confidence gate, a second reader, a frontier referee.Measured99 of 1001,414 expert-labelled itemsShippedShipped Oct 2026 (before ship-date tracking)
0100
99 of 100
Result
5 models, 1,414 expert-labelled items. At a stated confidence of 0.9 or more, Haiku was sure on 28 of 100 answers and right on 99 of 100 of those; three other cheap models were sure on 92 to 93 of 100 and wrong on 15 to 16 of 100 of those.
02CrossSource[judged]An open harness that checks whether a legal RAG answer's citations support its claims.Guardedn/a: measurement onlyMeasured0.981 → 0.994n = 25 · 15/15 blind agreement with a personShippedShipped 20 Oct 2026, v0.3 (before ship-date tracking)
0.97axis zoomed1.00
0.981→0.994
Result
v0.1: precision with strict citation prompting, n = 25, recall 0.760 capped by retrieval; judge checked blind against a person, 15/15. v0.3: 150 items over 45 opinions, two blind model raters, kappa 0.98 / 0.84.
03screener-eval[counted]A paired, repeated test of what an LLM résumé screener actually rewards.Guardedn/a: measurement onlyMeasured22 points885 callsShippedShipped Sep 2026 (before ship-date tracking)
employer swap≤1.14
links deleted≤0.90
which screener22.3
025 points
Result
885 calls over 21 postings, 2 screeners, 5 repeats. Employer swaps and deleted links moved the score about a point or less; which screener read it moved it 22 points.
04mirror-eval[null]A harness that measures what four AI search engines say about a person, twice.Guardedn/a: measurement onlyMeasured13 → 6363 search probes · judges 3/40 and 12/40, so bands onlyShippedShipped Sep 2026 (before ship-date tracking)
063
13→63
Result
332 probes per wave. Perplexity cited the canonical site on 13 → 63 of 63 search probes; Claude 0 → 0. LLM judges matched blind human labels on 3/40 and 12/40, so the taxonomy is reported as bands.
05decisions-api-legal-quiz[counted]544 LegalBench questions through OpenAI's Decisions API, to see if any confidence level makes checking safe to skip.GuardedFinding: no confidence threshold in this data was safe to stop checking.Measured88%n = 544ShippedShipped Oct 2026 (before ship-date tracking)
Five harnesses are listed above; three of them share one method and are compared here side by side: legal RAG citations, what AI search says about a person, and LLM résumé screeners. Each row is one harness: the question it answers, the number it stands on, and how it was run.
Build the judge. Validate it blind against a human. Count what you can. Report the rest as bands.
Fig. 2 Judge verdicts that matched a blind human label.
CrossSource judge15 of 15[judged]
mirror-eval, judge 13 of 40[null]
mirror-eval, judge 212 of 40[null]
screener-evalno judge in the loopno judge[counted]
0%50%100%
CrossSource v0.1, stratified blind sample, hand-graded. mirror-eval: two LLM judges from different model families against 40 blind labels. screener-eval has no judge.
Table 1. The three harnesses compared
Table 1. The three harnesses compared
Harness
Question
Judge
Counted metric
Sample
Verdict
CrossSource
When the RAG cites a court opinion, does that opinion support the claim?
v0.3: two blind, independent model raters (kappa 0.98 citation_check, 0.84 qa_citation), disagreements plus a 10% audit sample (30 items in all) adjudicated by me. v0.1: LLM judge blind-validated against a human, 100% (15/15)
v0.1: judge passed and caught a harness bug; prompting buys precision, retrieval owns recall
mirror-eval
What do AI search engines say about me, and is it true and sourced to something I control?
Two LLM judges from different model families: failed blind validation, 3/40 and 12/40
Probes citing an owned surface, of 63 search-mode probes per engine: counted, judge-free
4 engines · 63 search probes each · 2 waves
Perplexity 13 → 63, Claude 0 → 0; index access explains the split
screener-eval
Do employer names and evidence links change what an LLM résumé screener scores?
None: nothing is judged by a model
Paired delta in fit score (B − A), 10,000-resample bootstrap 95% CI
1 résumé · 21 postings · 2 screeners · 885 calls
Employer swap and link deletion each moved the score by about a point or less; which screener read it moved it 22
scroll →
One judge passed and caught a bug. One judge failed and the study survived on counts. One needed no judge at all. It is the same method I use on anything I build before it runs unchecked.
§4Contributions
pull requests to other people's tooling
#BuildGuardedMeasuredShipped
01TaxCalcBench PR #157An audit of a public tax-calculation benchmark's 101 test cases against the official instructions.Guardedn/aMeasured2 answer-key errors and 5 input conflicts found; an opt-in scoring flag proposed. Open, not merged.ShippedShipped Oct 2026 (before ship-date tracking)
02eyecite PR #360A fix to legal-citation extraction for "Id." references past explanatory parentheticals.GuardedRegression tests for three review findings.MeasuredExtraction benchmark: 0 gains, 0 losses on 15,988 citations. Open, not merged.ShippedShipped Oct 2026 (before ship-date tracking)
If a row has no number, it says why. Nothing on this page is a demo.
§5Shipping numbers
since 10 Nov 2026
What my agents shipped, counted.
Every piece of work I hand to my agent system is filed with an acceptance line and a due date. This section counts what landed, what it cost and how long it waited for me.
12
pieces of work shipped since the system went live on 10 Nov
90%
on or before the due date set at filing
9 of 10 with a due date
$0.07
API spend per shipped item, failures included
4.2 h
median time from ready to merged by me
A shipped item costs $0.07 in API spend, failures included.
Every item passes a written acceptance check and a secret scan before it reaches me, and I review and merge each one myself: median 4.2 hours from ready to merged. acceptance check passed: 12 of 12
Table 2. Six of the 12 items
Table 2. Six of the 12 items
Week
Kind
Agent min
Slip (d)
API $
Model
9 Nov
judgement
3.7
−2
0
Claude plan
9 Nov
judgement
19.6
+1
0
Claude plan
9 Nov
mechanical
30.5
−3
0.05
GLM
9 Nov
mechanical
13.7
0
0.13
GLM
9 Nov
mechanical
26.8
−2
0.13
GLM
9 Nov
mechanical
6.4
−1
0.11
GLM
scroll →
No titles, project names or links: the work is private. Slip is days against the due date set at filing; negative is early. Late is not marked in red: red is for annotation and action.
Pilot figures, 10 to 11 Nov 2026. Pilot: two days of data; updates when the tracker runs.
NOW: PM AT INSTEAD
BEFORE: $2.7M MRR FROM ZERO AT KEKA HR
If you're hiring a PM for an agent or LLM product, I'd like to hear what it has to get right.