Sean Spratt — Selected Work

I’m a developer-tools engineer. I spent eighteen years at Microsoft on tools other engineers depended on, and since early 2026 I’ve worked full time with terminal coding agents and built on top of them: agent loops and their tools, a multi-model evaluation CLI, and a way of scoping work so an agent can finish it.

Six projects, each chosen for what it shows

Choir

A command-line tool and app for evaluating models · macOS app + CLI

One prompt fanned out to every major LLM at once (OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, local models), with judge-model scoring, latency charts, a divergence heatmap, and saved runs that can be re-judged later without re-running the participants. The CLI shares the app’s database and drops in as a skill, so a coding agent can hand work to a specific model.

Choir Reports

Published comparison studies · 23 reports

The benchmark work written up: the same prompt across twenty-plus models, hand-scored and then re-ranked by a blind multi-judge jury, with inter-judge agreement, leak analysis, and the failure modes called out by name. Two findings: when the asker was visibly distressed, models led with the correct medical fix in 0 of 26 answers, against 20 of 26 for a calm asker; and a blind four-judge jury put shuffled, score-stripped answers back in their intended order at a median ρ of about .98.

Playable BYOK Games

Agent loops and tool design · two browser games, bring your own key

Two games where the models sit at the table: an AI Game Master narrates and referees, three AI party members act beside you, and you take the fourth seat. In the dungeon crawl the model runs the game through about twenty tools (narrate, attack, call for a roll, move the party) on one tool-calling layer that works across the Anthropic, OpenAI, Google and xAI APIs. Your keys never leave the browser.

Choir, online

Many models streaming at once · hosted, pay as you go

Choir in the browser. One question goes to as many as 24 models at once, each a lane that streams as it writes. A judge model scores every answer and sorts them by their bottom line, and you choose which answers carry into the next question. A second tab does the same for image models.

Tips for Building One-Shots

Working method with AI coding agents · written so others can repeat it

The three-document method I use to turn a product idea into something a coding agent can actually build without session drift or scope creep: what SPEC.md, CLAUDE.md, and FIRST_SESSION.md are each for, why the order matters, and a copy-pasteable scoping template.

FestivalIMS

In production · used at SOAK 2026, hosted per event

I built it after ten years as a Black Rock City Ranger, for the incident work Rangers do. It ran at SOAK 2026, a regional burn, and is planned for SOAK 2027 ahead of a wider launch. Incident management for festivals and regional burns: shift rosters, incident reports, a forensic audit trail, live metrics with a shift Gantt, and an offline Command Center for when there is no wifi. Hosted and operated per event, with full data export and deletion on request.

Also running

Dexter’s Papers

Unattended on weeknights · 87 of 90 since July 13, 2026

Each weeknight the day’s arXiv AI papers are ingested, summarized, and scored 0–10, then filed into a filterable map and a per-night calendar with one-line synopses and full breakdowns. It published on 87 of the 90 weeknights from July 13 to November 13, 2026, with no hand-holding.

The Sleeper Awakes and the pipelines behind it

Multi-agent pipelines with a checker in the loop · illustrated, narrated books

An illustrated, narrated edition of H. G. Wells’s The Sleeper Awakes, a public-domain novel: the text is Project Gutenberg’s, word for word, the pictures come from an image model working from scenes in the text, and a multi-voice audiobook runs on an open-weight speech model with word-by-word read-along. Its edition page says what AI drew and voiced. It comes from the same family of pipelines as Scriptorium (a brief to an illustrated audiobook through ten judged stages) and Series Notes (19 reader’s companions so far), where one model produces and a second checks the result against the source.