Writing
Notes on eval methodology, LLM-as-judge reliability, and RAG output quality — drawn from CrossSource and production eval work. And, occasionally, an industry thesis written so it can be wrong.
Shorter and more often: Notes
Listen, and read along
The judge caught a bug I didn't
On validating LLM-as-judge evals — and why a 15/15 blind agreement and a broken instrument can live in the same pipeline.
Writing · Eval
544 yes/no legal questions, one API call each, and the question that matters to anyone routing work on a confidence score: is there a number above which you can stop checking? Not in this data.
· 7 min read · 9:46 audio
Eval methodology · Regulated work
Wrong AI answers get caught. Missed items don't. How I'd count false passes in six kinds of regulated work.
· 5 min read · 8:14 audio
Industry thesis · Mobility
A family business from Hubli built a fifty-year moat in Indian transport and just placed the largest private Volvo order in the country. What that says about premium intercity bus travel — and the ₹50,000 experiment I am running to find out whether I actually understand it.
· 11 min read · 15:46 audio
Eval methodology · screener-eval
I ran my own résumé through two LLM screeners against 21 open AI PM roles, five times each, with the employers swapped for fictional ones and the links deleted. Neither mattered. Something else did.
· 3 min read · 3:52 audio
Eval methodology · mirror-eval
What four AI search engines say about me, measured twice — and the three things that only showed up because the numbers were counted rather than scored.
· 5 min read · 5:55 audio
Eval methodology · CrossSource
On validating LLM-as-judge evals — and why a 15/15 blind agreement and a broken instrument can live in the same pipeline.
· 2 min read · 2:06 audio