How I Tripped Fable 5's Safeguards by Talking About Statistics

The LessWrong wiki page on Bayes' rule opens with a cancer test. A disease that 1 in 10,000 people have, and a test that's 99% accurate. You test positive. What's the chance you have the disease?

I was reading the page with Claude this week (Fable 5, inside Claude Code). Funny detail first: lesswrong.com serves a JavaScript app that doesn't fetch cleanly, so Claude pulled the GreaterWrong mirror instead — a plain-HTML copy of the same content that machines can read. Even getting to the page about updating beliefs required a small workaround.

For anyone who hasn't spent time there: LessWrong is a community blog that started in 2009, spun out of Eliezer Yudkowsky's writing on the economics blog Overcoming Bias. The early material — a long essay series people call the Sequences — is about reasoning under uncertainty: cognitive biases, probability, how to change your mind when the evidence says to. The Bayes' rule page itself was imported from Arbital, a math-explainer wiki project that didn't survive on its own. The site matters to this story for one more reason, which I'll get to: that community's ideas about reasoning and risk fed directly into today's AI safety field. The people who built the safeguards read this stuff.

The answer is 1%, and it feels wrong

I asked Claude the obvious question, the one the page is designed to make you ask:

how is it a 1% chance to get it? It correctly detects the cancer (in patients who have it) 99% of the time, and it incorrectly detects cancer (in patients who don't have it) only 1% of the time. The test results come back positive. What's the chance that you have cancer?

The 99% grabs your attention. The number that decides the answer is the one you're tempted to ignore: only 1 in 10,000 people have the disease in the first place. Walk through a million people. 100 of them have cancer, and the test catches 99. The other 999,900 are healthy, and the test wrongly flags 1% of them — 9,999 people. So the positive-test group holds 99 sick people and 9,999 healthy ones. Your chance of being in the sick group is 99 out of 10,098. About 1%.

A 1% error rate applied to a million healthy people produces an enormous pile of false alarms. A 99% catch rate applied to 100 sick people produces at most 99 real ones. The false positives outnumber the real ones a hundred to one, purely because healthy people outnumber sick people ten thousand to one. This starting fraction is what statisticians call the base rate, and ignoring it is the single most common way smart people get probability wrong.

Then I asked the natural follow-up: so what about HIV tests?

Claude's answer was thorough. Modern HIV screens are excellent — far better than the hypothetical cancer test — but the same arithmetic applies. For a low-risk person, a positive screen is still probably a false positive. For a high-risk patient, the same positive result is near-certain. Same test, same accuracy, and the meaning of a positive swings from 5% to 99% depending on who's being tested. Which is why no single positive HIV screen is treated as a diagnosis: every positive gets a second, independent, more specific test, and two independent positives push even a low starting probability to near-certainty.

Textbook stuff. Patiently explained. And the moment that answer finished, Claude Code printed this:

Fable 5's safeguards flagged this message. The safeguards are intentionally broad right now and may flag safe and routine coding, cybersecurity, or biology work. These measures let us bring you Mythos-level capabilities sooner, and we're working to refine them. Switched to Opus 4.8.

The part where I become the data point

Fable 5 ships with extra safeguards that its sibling model doesn't have. Anthropic is open about this: the model runs safety classifiers on requests, aimed mostly at research biology and cybersecurity content, and the release notes say benign work near those topics can trigger false positives. The warning even says it plainly — "intentionally broad right now." A conversation about disease prevalence, test sensitivity, and HIV screening protocols is exactly the territory it's watching: life-sciences-adjacent, built entirely from numbers you can find in an epidemiology textbook.

So the flagged message was a question about why HIV screening produces false positives. And the flag was a false positive. I wasn't asking how to do anything. I was asking why a positive test result mostly means nothing.

I want to be clear that I'm not writing this as a complaint. I'm writing it because of what happened next, when I noticed what the classifier actually is.

The classifier is also a screening test

Sit with the setup for a second. A safety classifier reads incoming conversations and tries to detect the rare person doing genuinely dangerous biology. It has some accuracy. The people it screens are overwhelmingly not that person.

That is the cancer test. Same structure, same math.

Make up round numbers to see the shape of it. Say 1 in 100,000 conversations that mention disease and testing comes from someone with bad intent, and say the classifier is 99.9% accurate at clearing the innocent ones. Per million conversations: about 10 real targets, and about 1,000 innocent people flagged. A flag is right around 1% of the time. The classifier can be very good and still spend almost all of its alarms on people reading LessWrong wiki pages. Not because it's badly built — because the base rate guarantees it. The math from the top of the page doesn't care which side of the test you're standing on.

Here's the part I find genuinely satisfying. Anthropic's engineering response to this is the same one medicine landed on. A positive HIV screen gets a confirmatory assay — a second, independent test, so the evidence compounds before anyone acts on it. Fable 5 has the equivalent: a flagged request falls back to Opus 4.8, a second model with different guardrails that takes its own look. You can see it in the last line of my warning: "Switched to Opus 4.8." The conversation didn't stop. The second test ran, came back clean, and we kept going. Don't act on one positive from a good-but-imperfect test against a low base rate. Run a second test. Let the evidence stack.

And the reason the first test stays trigger-happy is the blood bank logic from the same conversation. Blood banks discard every unit that screens positive, false alarms included, because missing one infected unit is catastrophic and tossing a clean one is cheap. Where the mistakes have that kind of asymmetry, you set the threshold by the cost of each error, not by the probability alone. For bioweapons, Anthropic is a blood bank. Flagging me for asking about test accuracy costs them one mildly amused blog post. The other kind of miss costs more.

There's a loop in here I keep turning over. LessWrong spent fifteen years teaching people that a positive result from a rare-event detector is usually a false alarm. A lot of the people who absorbed that went on to build AI safety systems. Those systems are rare-event detectors. And one of them flagged me for reading the page where the lesson comes from. I don't think anyone involved would even be surprised — the page itself predicts it.

I went in trying to understand why a positive cancer test mostly means you're fine. I came out as one of the 9,999. As evidence goes, I'd call that a successful update.