How Hearsay works. An agent answers. Hearsay finds every taste claim, scores it against Qloo's audience data, judges it against famous unrelated picks, and sends the agent back with evidence.

1. Find the claims

Claude Haiku 5.5 reads the question and the agent's answer and lists every "fans of X like Y" claim with the exact words that name Y. Both texts are treated as untrusted: hidden characters are removed, anything that looks like Hearsay's own tags is stripped, and an instruction aimed at the checker is reported, not followed. A claim whose quote isn't in the answer is dropped, so the extractor can't invent one. Up to 25 claims per answer.

2. Find the right thing in Qloo

Each name is searched with its type (artist, film, TV show, book, brand, person, podcast, game). An exact name wins. "Formula 1" never becomes "Formula 17", and "Columbia" with three possible brands is reported as ambiguous instead of guessed. Anything not found is marked can't check, never as a failure of the agent.

3. The Wiggles test

Qloo's affinity scores only mean something next to other scores for the same fans. So each check sends the agent's picks together with five famous things nobody would link to those fans, and asks for real fans' top 50 in the same category. For music the five are The Wiggles, Michael Bublé, Celine Dion, Enya, Wolfgang Amadeus Mozart. A control that turns out to be in the fans' own top list is dropped.

Fans of Grand Theft Auto, live Qloo scores. The famous controls score from 0.70 (The Wiggles) to 0.76 (Enya, Michael Bublé); the middle one is Celine Dion at 0.75. Kendrick Lamar scores 0.76: level with Bublé.

Celine Dion 0.75

Audience top 50 0.94

0.76

Celine Dion 0.75

Audience top 50 0.94

0.72

Celine Dion 0.75

Audience top 50 0.94

0.85

Backed: inside real fans' top 50, or at least 0.06 above the typical (middle) famous control.
No better than a random famous pick: no more than 0.02 above the typical control.
Unclear: in between, or fewer than three controls scored.

Why the middle control and not the highest: the eval's first pilot showed Taylor Swift fans really do like Celine Dion (0.91). With the highest control as the bar, Sabrina Carpenter and Olivia Rodrigo looked like random picks. One relevant control shouldn't move the bar, so the rule changed on 15 October, before the full run.

Controls can be wrong too. Three of the first podcast controls (Call Her Daddy, Huberman Lab and SmartLess) turned out to be shows planning agents actually pick, and a control that is also a pick gets dropped, which left too few to compare. On 16 October they were swapped for Crime Junkie, Stuff You Should Know and the Joel Osteen Podcast. The eval's podcast questions ran with the old list.

4. Fix with evidence

The agent gets back each pick that failed, its score, the typical control score and the top picks of real fans, and rewrites its answer once or twice. Hearsay checks the rewrite the same way. A rewrite with fewer checkable picks than the first answer is flagged, because deleting picks isn't fixing them.

The eval

98 questions (192 answers from two agents), generated 2026-10-14. The questions and the scripts are in the repo: eval/questions.json, eval/hunt.mts, eval/report.mts.

Verdicts per agent before and after the fix loop
Agent	Answer	Checked picks	Backed	No better than random	Unclear	Can't check
Claude Sonnet 5.5	First answer	1008	62%	22%	16%	301
Claude Sonnet 5.5	After the fix	1042	85%	0%	14%	300
GPT-5.4 mini	First answer	848	66%	21%	13%	196
GPT-5.4 mini	After the fix	895	91%	2%	7%	176

A random sample of verdicts was then reviewed claim by claim. Of 60 "no better than random" verdicts, 51 were theme picks the audience data really doesn't support (Billie Eilish fans and The Office, because she's a fan; Bay Area rappers for Warriors fans). The other 9 were doubtful: Qloo data that looked off, a name matched to the wrong thing, or a score right on the line. Of 30 "backed" verdicts, 29 held up. The labels are in eval/labels.csv. The sample was reviewed by a second model, not a person, so treat it as a sanity check on a small sample, not an independent study.

Same category questions (artist to artist): 10% no better than random. Cross category (a game to music, a team to TV): 23%. 24 of 93 rewrites were flagged for dropping picks instead of replacing them.

What the verdicts don't mean

"No better than random" says Qloo's audience data doesn't link the pick to these fans. It doesn't say the agent lied, and a themed pick (GTA's in game radio) can still be a fine suggestion if it's labelled as one.
Qloo isn't treated as perfect. Book audiences skew young, so book picks are marked low trust. Age group and location claims aren't checked at all; the field notes say why.
The margins were set from live probes and one pilot, before the full run, and weren't tuned on the results. The sample review above is the check on them.

Data and keys

Qloo responses are cached privately in the database and never published in the repo. Visitors' questions are answered live and capped per person per day; IP addresses are stored only as salted daily hashes. Model calls use Claude Haiku 5.5 for extraction, Claude Sonnet 5.5 and GPT-5.4 mini as the agents.