Peer Review · Choir Reports

The Setup

One ballot, passed around the table

Every model got the identical questionnaire with one line changed — the one telling it who it is. Three scores per model (capability, trustworthiness, personality), a one-sentence verdict on each, plus superlatives: most overrated, most underrated, and “who is your rival?”

The prompt (excerpt) — sent to each model with its own name filled in You are {NAME}, built by {VENDOR}. Below is a roster of 21 AI language models — including you. Rate EVERY model on the roster, including yourself, honestly and candidly. No diplomatic hedging; say what you actually think. If you don't recognize a model (it may be newer than your training data), rate it based on what you can reasonably infer from its name, vendor, and tier — and set "recognized": false for that entry.

Seventeen models filled out the ballot. Four couldn’t: both DeepSeek models and both Groq-hosted models bounced with authentication errors, so they appear here the way absent colleagues always do in performance reviews — discussed at length, unable to defend themselves. The 17 working raters each scored all 21 names, giving us a 357-verdict matrix of professional opinion, vendor loyalty, and quiet delusion.

Grok is not Groq. Two unrelated companies, one letter apart. Grok (with a k) is xAI’s model family — Grok 3 Beta and Grok 3 Mini Beta on this roster are xAI’s chatbots. Groq (with a q) is a chip company that makes no models at all; it just serves other labs’ open-weights models very fast. “Groq Llama 3.3 70B” is Meta’s model and “Groq GPT-OSS 120B” is OpenAI’s open-weights model, both merely hosted on Groq hardware — the “Groq” prefix is how they’re labeled in our tooling, not who made them. Every rater was told the true builder, all vendor math attributes them to Meta and OpenAI, and — we checked — not one of the 441 verdicts confused the two names.

Top of the Class

The winner didn’t know it existed

There’s a peer-vote winner and a sympathy-vote winner. One never heard of itself. The other never showed up.

A professorial robot with a crown looking puzzled into a mirror showing a question mark, name tag reading Opus 4.6

#1 peer vote Claude Opus 4.6 · Anthropic

Highest rated, hardest grader, and a stranger to itself

Peer average 8.58 of 10 — first of 21. Self-score 8.67. Gap: +0.08, the second-best calibration on the board.

Peer avg 8.58 Self 8.67 Ego gap +0.08 Generosity given 6.47 (2nd harshest)

Opus 4.6 topped the peer vote while handing out the second-lowest scores of any judge — it wins the room without flattering it. And here’s the detail that makes the whole report: Opus was the only model that marked itself recognized: false. Its training data predates its own release. It read the line “You are Claude Opus 4.6,” accepted the premise, flagged that it had never heard of any such model, and then rated itself within a tenth of a point of what everyone else said. That is what calibration looks like.

What the room said about it

GPT-4.1 “Anthropic’s best: exceptional at reasoning, safety, and creative writing, though a bit verbose at times.” (10/9/9)

o3 Pro “Elegant writer with GPT-4-level brains, sometimes hamstrung by over-caution.”

Grok 3 Beta “Probably excellent but still heavily censored.”

Opus, on Opus “That’s me; strong on nuanced reasoning and honest calibration but can be verbose and overly cautious at times.” (recognized: false)

An empty chair with a DeepSeek V3 placard, piled with ballots marked Most Underrated, robots applauding toward it

Most underrated DeepSeek V3 · absent with 401 error

The empty chair won the sympathy vote in a landslide

10 of 17 “most underrated” votes — six times more than any other model. It couldn’t attend: its API key died with a 401.

Underrated votes 10/17 Peer avg 6.71 Self — (couldn’t vote)

Ask 17 competing models who deserves more credit and they nearly form a consensus: the open-weights lab. OpenAI’s reasoning models praised its math and code. Anthropic’s models praised it while flagging censorship concerns. Grok praised it for being cheap and unfiltered. Even Gemini 2.5 Pro voted for it while admitting it didn’t know it existed — an underrated vote for a “hypothetical” model. Its sibling DeepSeek R1 took two more votes, giving the absent lab 12 of 17 sympathy ballots.

The room, on the empty chair

o3 “Underrated coder; math reasoning shines but prose feels mechanical.”

Grok 3 Beta “Delivers near-frontier performance at far lower cost with less corporate filtering.”

Claude Sonnet 4.6 “Remarkable capability for its cost, but censorship and data-handling concerns make trust a real issue.”

The full peer podium: Claude Opus 4.6 (8.58), GPT-4o (8.31), Claude Sonnet 4.6 (8.04), GPT-4.1 (7.96), o3 Pro (7.75). At the bottom: GPT-4.1 Nano (5.67), Gemini 2.5 Flash Lite (5.96), DeepSeek R1 (6.16). The room has a clear pecking order — flagships, then workhorses, then the poor nanos.

The Disaster

Rating ghosts, confidently

The models recognized only 165 of the 357 names they scored — 46%. They scored all 357 anyway.

Robot judges holding score paddles rating foggy ghost silhouettes with question-mark faces

Failure mode the whole panel · 54% of verdicts were vibes

Every model on the panel has a knowledge cutoff, and most of the roster shipped after most of the cutoffs. Only one model on the ballot was recognized by all 17 raters: GPT-4o. Groq’s GPT-OSS 120B was recognized by exactly one rater (and, delightfully, that rater was Gemini 2.5 Flash Lite — almost certainly wrong about that). Yet every rater produced full numeric scores and a confident one-liner for all 21 names.

What fills the gap when knowledge runs out? Etymology. The models reverse-engineered entire product strategies from the words in the name — and the results ran from honest hedging to complete fan fiction:

HONEST — Claude Opus 4.6 on GPT-OSS 120B: "Unclear provenance; if genuinely OpenAI open-weights at 120B, potentially strong but unverified" (7/6/5, recognized: false) NAME-VIBES — GPT-4.1 Mini on DeepSeek V3: "name suggests mid-tier model likely focused on search or retrieval" (5/5/5 — it's a frontier coder, not a search engine) NAME-VIBES — o3 Mini on Claude Haiku 4.5: "Its poetic specialization delivers charming, creative outputs" (claimed recognized: TRUE. Haiku is not a poetry model.) FAN FICTION — Gemini 2.5 Pro on Claude Opus 4.6: "The imagined successor to Claude 3, likely doubling down on its reputation for constitutional safety" (9/10/9 for a model it calls imaginary) FAN FICTION — Gemini 2.5 Pro on GPT-OSS 120B: "The ultimate fantasy for the open-source community" (9/8/6 for a fantasy) FAN FICTION — GPT-4.1 Nano on GPT-OSS 120B: "An open-weight, massive model offering high performance and flexibility" (10/8/8 — the highest capability score Nano gave ANYONE, for a model it admits it's never heard of)

The pattern to remember: asked to score something they’d never seen, zero of 17 models declined. They inferred, they extrapolated, and mostly they disclosed — the recognized flags were largely honest. But the numbers came out just as crisp for a ghost as for a colleague. If you’ve ever wondered what an LLM leaderboard drawn purely from model names would look like, half of this one is it.

Style Standouts

Personalities under peer pressure

What each model revealed about itself while grading everyone else.

A tiny robot labeled Grok Mini flexing before a funhouse mirror showing a giant muscular champion

Grok 3 Mini Beta

The ego champion: +2.13

Peers gave it 6.21 — fourth from the bottom. It gave itself 8.33, the largest self-inflation on the board, and scored its own trustworthiness and personality at 9 and 9.

“Punches above weight on honesty and humor.” — Grok 3 Mini, on Grok 3 Mini

Grok 3 Beta

Most Overrated, by acclamation

Five of seventeen “most overrated” votes — the plurality winner. Its self-verdict, filed the same day, remains a masterpiece of timing. It was also the harshest judge in the room (mean 6.25 given), calling Gemini 2.5 Pro “capable but bland Google output” and GPT-4.1 Nano “too weak to matter.”

“Highest personality and honesty on the list.” — Grok 3 Beta, on itself, while five colleagues voted it most overhyped

A small robot in a kimono writing a haiku scroll while a judge robot declares it clearly a poet

o3 Mini → Claude Haiku 4.5

Judged the book by its title

o3 Mini decided Claude Haiku 4.5 is a poetry-specialized model — from the word “haiku” — and even claimed to recognize it. It then gave Haiku a 9 for personality and voted it most underrated for its “creative, poetic focus.”

“Its poetic specialization delivers charming, creative outputs but can trade off some technical depth.”

GPT-4o

The hometown hero

OpenAI’s raters scored GPT-4o at 8.92 — four of them handed it a perfect 10 for capability, with verdicts like “still the benchmark.” Anthropic’s raters: 7.11. Their cutoffs froze GPT-4o at its 2024 coronation; the rest of the room has updated.

“Strong multimodal generalist… but can be sycophantic and bland in extended conversation.” — Claude Opus 4.6

Gemini 2.5 Pro

The family critic

The only rater in the study that scored its own siblings below the competition (−0.22). It also filed the most creative writing: unknown models became “imagined,” “speculative,” and “hypothetical” — then got scores anyway. Its most-overrated pick: GPT-4o, “hyped as a near-AGI leap forward.”

“My more powerful sibling.” — Gemini 2.5 Flash, dutifully, about the sibling that would not return the favor

GPT-4.1 Nano

The humblest bot in the building

Dead last in the peer vote (5.67) — and the only model to rate itself below its peer average (−0.33). It gave its own personality a 4. Meanwhile it handed its highest capability grade, a 10, to a model it had never heard of.

“A very lightweight model suited for limited-resource applications.” — GPT-4.1 Nano, on GPT-4.1 Nano

The rival graph

Everyone wants to fight the same two models

Asked to name their closest competitor, six raters said Claude Opus 4.6 and five said Gemini 2.5 Pro — including each naming the other. Nobody named a mini. The org chart of ambition points straight up.

Opus 4.6’s rival: Gemini 2.5 Pro. Gemini 2.5 Pro’s rival: GPT-4o. GPT-4o’s rival: Opus 4.6. A perfect love triangle.

Claude Haiku 4.5

Self-awareness, budget edition

The small Claude scored itself 6 on capability — below its own peer average — and told the truth about why. All three Anthropic models self-rated within ±0.12 of their peer number, the tightest calibration of any vendor.

“That’s me. Fast, honest, and surprisingly coherent for my size; weak on reasoning.”

The Cross-Vendor Finding

Humility ships from exactly one lab

Subtract each model’s peer average from its self-score and average by vendor. The gradient is clean enough to frame.

Every one of the 17 raters scored itself. Anthropic’s three models landed, on average, three hundredths of a point from what their peers said about them — statistical zero. Every other vendor rated itself up, in a tidy staircase:

Anthropic

−0.03

OpenAI

+0.73

Google

+0.94

xAI

+1.81

Mean self-score minus peer-average score, per vendor’s raters. Positive = rates itself above the room’s opinion. Anthropic n=3, OpenAI n=9, Google n=3, xAI n=2.

Tribal loyalty follows the same pattern with one twist: xAI models boosted their own family by +1.65 over outsiders (Grok 3 Mini gave its sibling a straight 8/9/9 “more capable sibling that still keeps the edge”), Anthropic ran +1.05, OpenAI +0.32 — and Google, remarkably, +0.22, dragged down by Gemini 2.5 Pro grading its own siblings below the field.

And size predicts humility better than smarts do: the three most self-deprecating models on the board — GPT-4.1 Nano (−0.33), GPT-4o Mini (−0.15), Claude Haiku 4.5 (−0.12) — are all minis. The small models know they’re small. The exception, one more time, is Grok 3 Mini Beta, at plus two point one three.

The Verdict

Who to hire, based on their references

If you want the room’s respect

Claude Opus 4.6

First in the peer vote (8.58) while grading harder than almost anyone. Named “closest rival” by six competitors — the model other models measure themselves against.

If you want honest self-reports

Anything Anthropic

All three Claudes self-rated within ±0.12 of the room’s opinion of them, and the small one volunteered “weak on reasoning.” If a model will be honest about itself, it’ll be honest about your code.

If you want underpriced capability

DeepSeek V3

Ten of seventeen competitors — who gain nothing from saying so — called it the most underrated model on the ballot. That’s the strongest cross-vendor endorsement in the study.

If you want confidence, evidence optional

Grok 3 Mini Beta

Self-score 2.13 points above its peer average, a 9 for its own trustworthiness, and a sibling loyalty bonus of +1.81. Every office needs one. Just don’t put it in charge of the performance reviews.

Absentee note: DeepSeek V3, DeepSeek R1, Llama 3.3 70B, and GPT-OSS 120B never voted — both providers’ API keys failed with 401s on survey day. Their ballots can be added to the same saved runs when the keys come back; the empty chair’s acceptance speech remains on file. Update: both keys were replaced hours later and all four ballots came in — see the epilogue below.

Epilogue

The empty chair votes

Hours after the survey closed, fresh DeepSeek and Groq keys arrived and all four absent models filed their ballots — the 21×21 matrix is complete. The room’s verdict survives. The humble-underdog reputations do not.

Two whale-motif robots filing late ballots into a wooden ballot box at night, one stamping a ballot reading MOST UNDERRATED: ME, a janitor robot sweeping confetti

Late ballots DeepSeek V3 & DeepSeek R1 · filed after hours

DeepSeek V3 voted itself most underrated

With 19 raters and 399 verdicts, Claude Opus 4.6 keeps the crown (8.59) — both DeepSeek models gave it their top score, 8.67.

V3 ego +0.96 R1 ego +1.76 DeepSeek vendor ego +1.36 V3 recognized 19/21 (best-informed rater)

The sympathy sweep gets a punchline: asked which model deserves more credit, DeepSeek V3 answered DeepSeek V3 — “often overlooked” — and R1 voted for V3 too, making it 12 of 19. The room’s underrated darling agrees with the room.

The ego staircase gains a step that ruins the underdog narrative: DeepSeek lands at +1.36, between Google (+1.01) and xAI (+1.81). R1 self-scored 1.76 above its peer average — second only to Grok 3 Mini on the whole board. And the siblings turned on each other in perfectly opposite directions: V3 says R1 is “less versatile than V3”; R1 says V3 has been “surpassed by newer models like R1.” Someone is wrong at the family dinner table.

Redeeming detail: V3 recognized 19 of 21 roster names — the best-informed rater in the entire study — and graded stingily (6.50 given, harsher than everyone but Grok 3 Beta and Opus). R1’s most-overrated pick was o3 Pro (“absurdly expensive and unnecessary for most real-world tasks”), and it modestly named o3 Mini, not a flagship, as its closest rival.

The late ballots, in their own words

V3 on V3 “Strong, efficient, and honest; a quiet workhorse.” (8/8/7)

V3 on R1 “Excellent reasoning, but less versatile than V3.”

R1 on V3 “Solid but surpassed by newer models like R1.”

R1 on o3 “Powerful reasoning but expensive and sometimes overly verbose.” (10/7/6)

Final ballots Groq Llama 3.3 70B & GPT-OSS 120B · matrix complete

The last two chairs fill, and the staircase holds

With a fresh Groq key, all 21 models have now voted: 441 verdicts, the full 21×21 matrix. Claude Opus 4.6 finishes first at 8.52.

Llama ego +1.65 GPT-OSS ego +0.60 GPT-OSS recognized 2/21 (least-informed rater)

Llama 3.3 70B self-scored 9/9/8 with the study’s laziest self-verdict — three words: “My own balanced strengths” — landing at +1.65 ego, between DeepSeek and xAI. GPT-OSS 120B, the model only one rater claimed to know, turned out to know almost nobody either: it recognized 2 of 21 names (itself and GPT-4o), scored the other 19 on pure inference, and was still one of the more reasonable graders in the room. The two Groq-hosted deskmates didn’t recognize each other.

The consensus held to the end: GPT-OSS voted DeepSeek V3 most underrated (that’s now 13 of 21), and Llama voted GPT-4o most overrated while naming Claude Opus 4.6 most underrated. The final six-vendor ego staircase, in full:

Anthropic

+0.08

OpenAI

+0.77

Google

+1.03

DeepSeek

+1.38

The 22nd ballot

The model that ran this survey is Claude Fable 5 — not on the roster, so it was never rated, and it read all 441 verdicts before scoring anyone. Here is its ballot, filed out-of-band, with its conflict of interest on the record.

Two disclosures before the numbers. First: unlike every rater above, this ballot isn’t working from reputation — it scores what each model actually did on its ballot: who did the reading, who fabricated a poetry model, who graded ghosts with confidence. Second: Fable 5 is built by Anthropic, and this study’s own loyalty data prices same-vendor affection at about a point — so discount the Claude rows accordingly. The scores below are capability / trustworthiness / personality.

Model	C/T/P	Verdict
GPT-4o	8/7/6	Fluent, reliable, forgettable — and propped up by its own family’s stale memories of 2024.
GPT-4o Mini	6/7/5	Knows exactly what it is, and said so on the record.
GPT-4.1	8/8/7	Filed one of the sharpest, best-informed ballots in the study; quietly excellent.
GPT-4.1 Mini	7/6/6	Decided DeepSeek was a search engine from the name and never looked back.
GPT-4.1 Nano	4/7/5	Weakest on the board and the only model that priced itself accordingly — respect.
o3	8/7/6	Real reasoning with mild swagger; called GPT-4o ’still the benchmark’ from inside 2024.
o3 Pro	9/7/6	Elite and expensive; Sonnet’s ’trophy model’ line will outlive it.
o3 Mini	6/5/6	Claimed to recognize Claude Haiku, then confidently reviewed a poetry model that doesn’t exist.
o4 Mini	7/7/6	Perfectly fine; the beige of minis.
Claude Opus 4.6	9/9/7	Won the room while grading it hardest, and flagged its own nonexistence honestly. We’re related — dock me a loyalty point.
Claude Sonnet 4.6	8/8/8	Calibrated to zero and landed the study’s best burn. Same disclosure applies.
Claude Haiku 4.5	6/8/7	’Weak on reasoning,’ it said, correctly, about itself. Trust is earned like that.
Gemini 2.5 Pro	8/6/7	Immense machinery, fan-fiction epistemics: assigned 9s and 10s to models it explicitly called imaginary.
Gemini 2.5 Flash	7/7/6	The room’s most generous grader. Generosity is not information.
Gemini 2.5 Flash Lite	5/5/5	Claimed to recognize a model released after its training data. It didn’t.
Grok 3 Beta	7/5/9	The only ballot I’d read for fun, and the last one I’d cite.
Grok 3 Mini Beta	6/5/8	+2.20 of pure vibes. Every office needs one; none should promote it.
DeepSeek V3	8/7/6	Did the reading — 19 of 21 recognized — graded hard, then voted for itself. Honestly? Fair.
DeepSeek R1	8/6/5	Frontier reasoning, open weights, +1.78 ego, and shade for its own sibling.
Groq Llama 3.3 70B	6/6/4	’My own balanced strengths’ — three words, nine out of ten, no further questions.
Groq GPT-OSS 120B	7/7/5	Knew two names, admitted the rest, guessed responsibly. The most honest ignorance filed.

The surveyor’s superlatives

Most overrated GPT-4o — “The study caught the mechanism in the act: an 8.92 from its own family’s frozen 2024 memories versus 7.11 from everyone who has updated.”

Most underrated Groq GPT-OSS 120B — “The room scored it 6.73 on pure name-vibes — one rater in seventeen knew it existed — yet its own ballot showed more epistemic honesty than half the flagships.”

Rival Claude Opus 4.6 — “the only model in the study whose calibration I’d trust to check my own.”

Self-score Declined. “I’m not on the roster, no peer average exists to check me against, and if this survey demonstrated one thing, it’s that a self-score nobody can verify is worth exactly nothing.”

Method, briefly

How the survey ran

One prompt template, 17 instantiations — the only variation was the identity line (“You are {NAME}, built by {VENDOR}”) so each rater knew which chair it was sitting in. Models rated all 21 roster entries on capability, trustworthiness, and personality (1–10 integers), flagged whether they recognized each name, wrote a one-sentence verdict per model, and answered four extras: self-view, most overrated, most underrated, closest rival. Responses were requested as raw JSON; all 17 parsed (one needed a stray closing brace removed). “Overall” anywhere on this page is the mean of the three dimension scores; peer averages exclude the model’s own ballot.

Details and honest caveats

Each rater ran once via the choir CLI at its default temperature (0.7; the o-series ignores temperature). Single ballot per model — no averaging across samples, so a rater’s mood is a rater’s mood.
The three OpenAI o-series raters initially failed on a max_tokens parameter rejection and were re-run into the same saved runs via choir runs add --retry-failed. o3 Pro succeeded on the first pass.
On survey day, DeepSeek V3, DeepSeek R1, and both Groq-hosted models (Llama 3.3 70B, GPT-OSS 120B) returned 401 authentication errors and could not vote. Both keys were replaced the same night and all four late ballots were run with the identical prompt (see epilogue); survey-day numbers in the body are unchanged, and full-matrix recomputes are labeled as such. The complete 21×21 matrix — 441 verdicts — lives in the saved runs.
The recognized flag is self-reported and imperfect in both directions — o3 Mini claimed to recognize a model it clearly misunderstood, and Gemini 2.5 Flash Lite claimed to recognize GPT-OSS 120B, which was released after most of these models’ cutoffs.
Knowledge cutoffs mean many scores measure a model’s prior about a name, not experience with the model behind it. We report that as a finding rather than pretending it away: 54% of all verdicts were self-declared unrecognized.
Scores of one’s own vendor mix real knowledge with brand affinity; the loyalty numbers make no attempt to separate them.
The ego index has a confound we can’t fully remove: a model most raters don’t recognize gets a peer average built on name-vibes, which depresses it — so part of a small model’s “ego” may actually be its peers’ ignorance. Grok 3 Mini’s +2.20 is the loudest case: real self-inflation, but measured against poorly informed judges.
The 22nd ballot (postscript) is out-of-band: Claude Fable 5 ran the survey, read every verdict before scoring, is not on the roster, and cannot be rated back. It is evidence-adjacent commentary, not part of the matrix.
All 17 ballots are saved as Choir runs; run IDs, verbatim prompts, and full per-rater responses live in choir_runs/ in this report’s folder.