Anthropic Reportedly Taking Its Internal Evals Offline

In brief: The Verge reports that Anthropic is cutting off its internal evaluations, the tests it uses to measure model capability and safety, from the internet.

Key points

• The move covers the internal eval suites Anthropic uses to judge what its models can do and how risky they are.
• The core step is isolating those evals from the internet. The aim is reportedly to keep test questions and answers from being published online or accessed from outside.
• Scope, timing, and implementation details aren't known beyond the report itself. Readers should check the original article for specifics.

Why it matters

When eval questions spread online, they can end up in the training data of future models. This is known as contamination. Contaminated models score higher than their real ability, which makes benchmarks unreliable. Evals increasingly shape release and safety decisions, so protecting them fits a wider industry push to keep evaluation results trustworthy.

Read more

• Anthropic is cutting off its internal evaluations from the internet — The Verge