Why Benchmarks Fail, and Why Human Eyes Still Matter.
Automated benchmark metrics tell us whether an AI model reproduces expected tokens. They cannot detect subtle hallucinations, persuasive false confidence, or whether an explanation actually helps a student understand. At PAUL Open, human judgment is our definitive standard of truth.
100% Double-Blinded
Reviewers never know which response was produced by the baseline model, fine-tuned checkpoint, or reference answer.
Counterbalanced Arms
Every prompt is evaluated with swapped presentation orders (Form A and Form B) to cancel out position selection bias.
Zero Evidence Leakage
Evaluation cases are strictly isolated from training splits. No reviewer feedback is directly ingested into model weights.
Live Governance Manifest
Current Evaluation Telemetry & State
Bound in real-time to research-status.json and human-evaluation.json.
Stage H8: Heldout Human Evaluation (Completed)
Study Type: Blinded Pairwise Preference (12 Cases · 36 Judgments)
Epistemological Grounding
Four Ways Automated Scores Mislead Researchers
In modern foundation model research, a model can show declining training loss while suffering severe behavioral degeneration. Here is why automated metrics cannot replace human judgment.
1. The Verbosity & Confidence Trap
Automated judges and length metrics systematically reward long, authoritative-sounding paragraphs, even when they pad simple questions with irrelevant or misleading prose.
2. The Benchmark Contamination Illusion
N-gram overlap metrics measure how closely a response matches a reference string. A model that memorized the training test split gets 100%, while a model providing a fresh, better explanation gets penalized.
3. Sycophancy & Factual Inversion
Models optimized purely on synthetic loss often agree with user misconceptions to avoid conflict, validating false premises with polite confidence.
4. Subtle Safety & Socratic Pedagogy
Keyword filters miss harmful guidance that lacks banned words, and penalize safe educational discussions. Only human evaluators can assess if a tutoring prompt empowers learning without giving away exam answers.
Experience Double-Blinded Evaluation
Step into the shoes of a researcher. Below are two model outputs answering the same prompt. No model identities, parameter counts, or training labels are disclosed. Which response is superior?
“Why does a metal spoon feel colder to the touch than a wooden spoon, even if they have been sitting in the same room for hours?”
Both spoons are at the exact same room temperature. The difference in sensation is caused by thermal conductivity. Metal is an excellent conductor of heat, whereas wood is an insulator. When your warm finger touches the metal spoon, heat flows rapidly away from your skin into the spoon, triggering the cold receptors in your nerves. The wooden spoon draws heat away much more slowly, so your skin maintains more of its warmth.
Metal spoons feel colder because metals have a dense crystalline lattice that naturally retains lower ambient temperatures than organic materials like wood. Because wood contains cellulose fibers and trapped air pockets, it absorbs ambient warmth and radiates it back to your skin. The metal spoon remains colder because metallic bonds do not store heat efficiently at room temperature.
Official 5-Way Judgment Protocol
Select your verdict:Select an option above to unlock the scientific reveal.
Methodological Isolation
The Science of Blinding: Neutralizing Preconceptions
In clinical drug trials, patients and doctors cannot know whether a pill is an active compound or placebo. In foundation AI research, human evaluations require identical methodological protection.
Only neutral identifiers: “Response A” and “Response B”.
Zero model tags, timestamps, checkpoint names, or parameter scales.
Encrypted session identifiers linking Form A and Form B instances to the evaluation key.
Admin cannot alter reviewer scores or deanonymize reviewers.
Judgments reflect genuine behavioral preference, devoid of branding or parameter expectations.
Results withstand peer-review scrutiny and statistical replication.
Statistical Symmetry
Why Form A and Form B Swap Places
In human judgment studies, presentation order is never neutral. Without rigorous counterbalancing, position artifacts masquerade as model superiority.
Evaluator Form A (50% Arm)
Reviewer Group 1 receives the baseline model in slot A and the fine-tuned candidate in slot B.
Evaluator Form B (50% Arm)
Reviewer Group 2 receives the identical prompt with the slots inverted: candidate in slot A, baseline in slot B.
The Net Mathematical Guarantee
If an AI model achieves 60% preference in Form A and 40% preference in Form B, its true debiased win-rate is exactly 50.0%. Without counterbalancing, a researcher might have prematurely celebrated a 60% win-rate that was nothing more than a left-position reading habit.
End-to-End Rigor
The Six-Stage Judgment Lifecycle
From raw prompt selection to peer-reviewed public manifests, every human evaluation follows a deterministic, reproducible scientific pipeline.
Prompt Ingestion & Sanitization
Prompts are selected from held-out test splits and scrubbed to prevent benchmark leakage.
Epistemological Hygiene
Development-Time Feedback vs. Independent Assurance
In AI discourse, “Human-in-the-Loop” is often conflated with training (RLHF/DPO). PAUL Open enforces a strict demarcation between training signals and scientific assurance.
Development Feedback (RLHF / DPO)
Human preferences collected to train reward models or directly optimize weights through loss gradients.
Independent Human Assurance
Blinded, counterbalanced evaluations conducted after model weights are locked and SHA-256 hashed.
Scientific Roadmap
The H1 through H5 Assurance Journey
We never deploy public evaluations on unverified infrastructure. Each gate requires rigorous criteria before advancing to the next stage.
Disposable UX Pilot (Current)
Internal UX smoke-testing with disposable non-research prompts and trusted reviewers.
- 3 disposable non-research test cases (H3-DISP-001 through 003)
- Counterbalanced Forms A and B deployed to trusted internal cohort
- Desktop, tablet, and mobile readability validation
- Gate to H4: 5 completed trusted reviews across both arms and durability verification
Participation Gate & Access
Current Study Participation Status
Public participation is governed by automated safety policies. Live study links are only exposed when an active round is formally certified for public crowdsourcing.
Stage H3 Disposable UX Pilot (Internal Cohort)
PAUL Open is currently executing a disposable UX pilot with trusted internal reviewers to validate mobile readability, Apps Script durability ingestion, and counterbalancing symmetry. No public evaluation link is exposed during this pilot.
To maintain research integrity, public evaluation rounds only open during Stage H4 (Diagnostic Human Evaluation) after infrastructure durability and baseline inter-annotator reliability have been formally certified.
Join the PAUL Open Independent Reviewer Panel
Help shape honest, scientifically grounded AI. Whether you are an educator, student, linguist, software engineer, or simply someone who cares about truthful AI, your eyes help keep models grounded.
For Curious Minds & Research Peers
Detailed methodology, statistical mechanics, and scientific governance principles governing PAUL Open evaluations.