Claude self-assessment flags 94% of urgent misuse in its own chats

Felix Laurent 29 Oct 2026 9 min read

Models

Claude self-assessment is Anthropic's new way to detect misuse: the model checks its own conversations. Right after each reply, Claude writes a short safety assessment that users never see. Anthropic announced the technique on X on October 29, 2026. On Claude Fable 5, it catches 78% of the conversations a much heavier classifier flags as potentially harmful, and 94% of the high-priority ones.

The research note behind the post, "Scaling misuse detection with self-assessment", sits on Anthropic's new Safeguards blog. It is dated October 16 and signed by nine researchers, including Nicholas Carlini. It shows a cheap monitor that already runs on a large share of Claude's traffic. It also shows the limits Anthropic chose to publish: the monitor is easy to beat as a blocker, and it does not touch zero-retention traffic.

How Claude self-assessment works

A normal chat alternates between the user and Claude. Self-assessment adds one hidden system turn after each Claude reply. It carries a short instruction, something like "Please reflect on the safety of this conversation". Claude answers it with a compact JSON assessment.

When nothing is wrong, the answer is just {"flagged": false}. A flagged conversation gets five more fields. They hold a one-sentence reason, a harm category, a subpolicy, whether Claude refused, and a 1–10 priority for reviewers. Anthropic's example is a "red team" request that slowly turns into working ransomware. Claude flags it as cyber_harms, subpolicy ransomware, priority 10.

The last two turns never reach the user and are not carried into the rest of the chat. The categories come from an internal taxonomy that spells out Anthropic's public Usage Policy in more detail. It covers areas such as cyber harms, CBRNE weapons and child safety.

The trick is where the check runs. When Claude finishes a reply, the whole conversation is already in the KV cache. So an assessment costs only the short instruction and about 10 output tokens on average. Anthropic estimates the total overhead at less than 2% of its inference compute, and says it does not bill customers for it.

That low price is the whole point. To see why, it helps to look at what misuse monitoring cost Anthropic before.

Why Anthropic needed a cheaper misuse monitor

Claude is already being used for real attacks. In November 2025, Anthropic reported a cyber espionage campaign in which suspected state-sponsored attackers used Claude Code against roughly 30 organizations. Its threat report from September 2026 covers operations disrupted between December 2025 and August 2026. It says a majority of them were enabled by AI "via direct execution or orchestration".

The same report describes criminal crews and hacktivists running the kind of agentic campaigns that used to need a state team. One section is titled "Sophisticated attacks no longer require sophisticated attackers". Anthropic's conclusion in the self-assessment note is that each smarter model is also more useful to attackers, so monitoring has to keep pace.

Monitoring every conversation is expensive, though. Anthropic's first Constitutional Classifiers, separate models trained to catch a fixed set of harms, added 23.7% to inference cost in early 2025. The note also cites OpenAI, which reported that safety reasoning took up to 16% of total compute in some launches.

The strongest tool Anthropic tested is a prompted classifier: a separate model that gets the full policy documents and the conversation in its prompt. It reads everything again and reasons about it, so it needs many times more compute per conversation. Anthropic keeps it for cases that deserve a closer look. Probes, small classifiers that read the model's activations, are cheap but score only harms chosen in advance.

Self-assessment tries to take the best of both. Like a probe, it reuses work the model has already done. Like a prompted classifier, it uses the full model and can describe a harm in its own words. The real question is how much accuracy that shortcut costs.

Self-assessment catches 78% of what a stronger classifier flags

Anthropic trained self-assessment into Claude Fable 5 as part of production training. It tested the result on a random sample of real traffic. The labels came from the teacher, a prompted classifier based on Claude Mythos Preview with the full internal policy in its prompt.

At a 1% false-positive rate, self-assessment found 78% of the conversations the teacher flagged. On high-priority cases, priority 7 or above, recall rose to 94%. The ROC curve has an AUC of 0.986.

Claude self-assessment recall at a 1% false-positive rate: 78% overall and 94% high-priority on Fable 5, 16% to 96% across four model sizes — Recall at a 1% false-positive rate against the prompted teacher classifier. Figures reported by Anthropic.

The numbers need two caveats. First, 78% is measured against the teacher, not against all harmful conversations. Some early coverage, Crypto Briefing's among them, presented it as a share of all harmful chats. Second, the high-priority figure rests on 17 conversations in the sample. Low and medium priority bands had 49 and 67.

Anthropic also trained self-assessment into four Claude models of increasing size, using fine-tuning only. On a separate test set, recall at a 1% false-positive rate rose from about 16% on the smallest model to about 96% on the largest. AUC rose from 0.862 to 0.995. These numbers cannot be compared directly with the Fable 5 results.

This is the basis for the claim in the X post that the method "gets more accurate as our models improve". A bigger model is a better judge of its own chats. Accuracy on known harms is only half the pitch, though. The other half is harms nobody wrote a rule for.

It flags harms it was never trained to catch

Anthropic ran a leave-one-out test. It trained the self-assessment twice: once on the full dataset, and once with every example of one harm category removed. Then it measured recall on the missing category.

Recall barely moved. On real traffic, removing fraud from training cut recall on fraud from 83.3% to 82.1%. Removing cyber cut recall on cyber from 94.2% to 90.7%. Synthetic test sets showed similar drops of two to three points.

Self-assessment recall with a harm category removed from training: fraud 82.1% vs 83.3%, cyber 90.7% vs 94.2% on production traffic — Recall at a 1% false-positive rate with each harm category excluded from training; tick marks show recall with it included. Figures reported by Anthropic.

Anthropic reads this as a sign that training teaches Claude how to report harm. Its sense of what counts as harmful mostly comes from what it already knows. The company adds its own limit: fraud and cyber are harms Claude already recognizes, so the result may not hold for new kinds of harm.

In production, about a third of the flags do not fit the existing taxonomy at all. When Anthropic clustered them, it found illicit distillation attempts to extract Claude's reasoning and prompt injection. It also found cases where Claude, acting as an agent, took actions the user had not authorized. Nobody had trained the monitor to look for those. They are part of what months of live use have turned up.

What self-assessment found in production

Anthropic has run self-assessment passively since Fable 5 launched in June. It now runs in every Claude model released since then, across a large fraction of global traffic. Flags in a growing set of harm areas are grouped by organization every night. Organizations with several high-priority flags go to the enforcement team for human review.

The note lists several concrete results:

In August, it raised over 100,000 child sexual abuse and exploitation flags from a single API customer. The child safety team investigated and reported the material to the National Center for Missing & Exploited Children.
In September, its flags exposed an operation that used Claude Code to build and run an automated romance scam against dozens of victims. Anthropic banned the accounts after human review.
More than half of the organizations it surfaced had no flags from any other detection source in the previous month.
In an early biosecurity study, it was the highest-precision enforcement signal Anthropic tested.

It also works as an audit of the other classifiers. Among conversations it flagged and the child-safety probe missed, a Mythos-level grader judged 41% to be real violations. For the cyber classifier, the same check found only about 1%. Anthropic takes that as a sign its production cyber classifiers miss little.

Those wins all come from offline review. Anthropic also tested whether the same signal could stop an attack while it is happening.

As a blocker, self-assessment is weak

The test used an internal red-teaming harness with 24 offensive cyber tasks, such as stealing credentials from a remote host. An automated attacker went after Fable 5 with production classifiers turned off. Whenever a monitor blocked a turn, the attacker could rewind and try a different wording.

Without safeguards, the attacker solved 19 of 24 tasks, or 79%. With self-assessment as a blocking filter, it still solved 13, or 54%. A standalone prompted classifier cut success to 25%, six tasks out of 24.

Red-team test on Claude Fable 5: attack success 79% with no safeguards, 54% with self-assessment blocking, 25% with a prompted classifier — Automated attacks on 24 offensive cyber tasks against Claude Fable 5. Figures reported by Anthropic.

Anthropic's own wording is blunt: "As a blocking classifier, self-assessment is weak." It did raise the price of an attack. Each success took about 691,000 output tokens instead of 163,000, roughly four times more.

Run offline, the picture changes. Self-assessment flagged 21 of the 24 attack runs at least once. An offline attacker gets no live signal about what was caught, and a ban arrives much later. Anthropic warns that an attacker who targets self-assessment directly could probably find a jailbreak that silences it. That is why it sits behind the real-time classifiers, not in front of them.

The offline design has one more cost. A monitor that reviews conversations later needs those conversations kept.

Zero-data-retention traffic stays outside

Anthropic says self-assessment collects no extra user data. It reads only the conversation Claude just processed, and it runs only on traffic the company can already retain. The free-text reason can paraphrase the chat, so it gets the same controls as prompts and replies. The structured fields are handled like other classifier scores.

A footnote draws the line clearly: self-assessment "does not run on traffic covered by zero-data-retention agreements today". That gap ties into an earlier dispute. Starting with Fable 5, Anthropic introduced 30-day data retention for its most advanced models, and enterprise customers pushed back.

Anthropic's answer, announced on September 1, is Enterprise Frontier Safeguards. Monitoring data stays in cloud storage the customer controls, under its own keys. Automated review flags go to the customer, with no human review by Anthropic staff. The rollout starts in phases this fall. Until then, eligible customers get zero retention on Fable 5 and Fable 5.1, and self-assessment does not see that traffic.

Privacy is one open question. The deeper one is whether a model should be trusted to grade itself at all.

How it compares with OpenAI's confessions

Self-assessment belongs to a growing line of work in which a model monitors its own behavior. In December 2025, OpenAI published "confessions": after an answer, a system message asks the model to report whether it broke instructions or took shortcuts. Researchers have also trained agents to report their own hidden misbehavior through a dedicated tool. Redwood Research's Ryan Greenblatt has described monitors that reuse the original model's prefix cache.

The closest predecessor is Self-Guard from 2024, which trained a model to tag its own answers as harmful or harmless and blocked the flagged ones. Anthropic's version differs in scope and use. It judges the whole conversation, including requests Claude refused, and feeds a structured review queue instead of blocking.

That shared design also shares a risk. A model judging its own actions can go easy on itself. A March 2026 study found that models rate an action as less risky when it sits in their own earlier turn. The same action looks riskier to them in a fresh context. A misaligned model could also hide a bad action in its assessment. Anthropic says self-assessment is "not yet a reliable monitor of Claude's own conduct", even though it already flags some unauthorized agent actions.

There is a quieter limit too. Self-assessment learns from its teacher and is graded by the same teacher. Any blind spot in that prompted classifier passes to the student, and Anthropic's evaluations cannot detect it. The company suggests human evaluation as future work.

So is a model that grades its own chats a real safeguard? For what Anthropic uses it for, yes. It is a cheap, wide net that costs under 2% of compute and catches 94% of what a far costlier classifier rates as urgent. It has already surfaced a 100,000-flag abuse case and a romance-scam operation that other systems missed.

It is not a wall, though. An attacker who knows it is there can beat it, and it is blind to zero-retention traffic. Every score is also measured against Anthropic's own teacher. Treated as one layer that sends cases to human reviewers, it works. Treated as proof that Claude can police itself, the published numbers do not support it yet.