Evaluating LLMs when benchmarks are gamed

Every time a new open-weights model drops, the accompanying technical report features an impressive radar chart showing state-of-the-art numbers on MMLU, GSM8K, and HumanEval. Then you deploy it into a real extraction pipeline, and it hallucinates JSON keys that were never in your schema, or chokes on three-turn conversational context.

Public benchmarks have become the Goodhart's Law casualty of modern AI engineering. When an evaluation dataset enters pre-training web crawls or post-training synthetic tuning pipelines, it stops being a measurement tool and becomes a training target.

The mechanics of benchmark rot

Contamination is rarely malicious. It happens via simple crawl leaks, deduplication failures in Common Crawl filters, or aggressive synthetic data pipelines that prompt teacher models using questions derived from public eval suites.

Beyond direct memorisation, multiple-choice benchmarks like MMLU measure log-likelihood ranking rather than generative coherence. A model can achieve 85% on 5-shot MMLU while failing to write a functional 20-line SQL migration.

# Synthetic eval runner with strict deterministic assertions
import json
from dataclasses import dataclass

@dataclass
class EvalResult:
    schema_valid: bool
    field_accuracy: float
    latency_ms: float

def evaluate_extraction(sample: dict, candidate_fn) -> EvalResult:
    response, latency = candidate_fn(sample["prompt"])
    try:
        data = json.loads(response)
        valid_keys = set(sample["expected"].keys()) == set(data.keys())
        matches = sum(1 for k, v in sample["expected"].items() if data.get(k) == v)
        accuracy = matches / len(sample["expected"])
        return EvalResult(valid_keys, accuracy, latency)
    except (json.JSONDecodeError, TypeError):
        return EvalResult(False, 0.0, latency)

If an evaluation metric can be described in a public blog post, model providers will inadvertently overfit to it within six months.

Building a private, calibrated eval harness

Reliable production evaluation requires moving away from static public tests towards dynamic, property-based suites:

• Canary token rotation: Embed random uuid markers into production logs to detect if your internal regression suite ever leaks into fine-tuning corpuses.
• Deterministic property tests: For structured output tasks, test invariants (valid JSON, enum compliance, mathematical balance) rather than semantic similarity.
• Adversarial perturbation: Automatically inject typos, swap entity names, and introduce irrelevant distractor paragraphs to test robustness against prompt drift.

The limits of LLM-as-a-judge

Using a stronger model (such as GPT-4o) as an automated judge is popular because it scales cheaply. But automated judges suffer from well-documented systemic biases: self-enhancement bias (favouring their own stylistic cadences), verbosity bias (preferring longer, structured answers regardless of density), and position bias in pairwise comparisons.

To make model judges usable, always swap candidate positions (\(A/B\) vs \(B/A\)), enforce calibrated rubrics with explicit deduction criteria, and audit 5% of decisions with human reviewers on a recurring weekly cadence.

Building an internal evaluation harness for enterprise LLM deployments? Get in touch to exchange notes.