LLM Evaluatology
This leaderboard instantiates LLM Evaluatology in practice, jointly modeling ten indispensable evaluation components and systematically sampling their configurations to yield interpretable, reproducible, and causally trustworthy rankings of intrinsic LLM capability.
Why this site
Underspecified setups cause huge accuracy swings
Small, undocumented choices in prompts, decoding, and other settings can change a model's accuracy by up to 70% and even flip rankings.
Memorization hides brittle reasoning
Models may succeed on problems they have effectively memorized, yet fail when the same reasoning needs be applied to slightly altered but equivalent conditions.
Fragmented studies miss critical interactions
Existing work typically varies one component at a time, overlooking how multiple evaluation factors interact to shape outcomes and confound attribution.
Why Evaluatology
Evaluation Is Everywhere, Yet Still Ad‑Hoc
Evaluation underpins computer benchmarks, rankings, financial indices, clinical trials, psychological scales, and social policy assessments, but most practices are still isolated, experience-driven, and lack a shared foundation. Evaluatology is needed to move beyond one-off, ad‑hoc solutions toward a coherent discipline.
Vague Concepts and Weak Theory Limit Trustworthy Evaluation
Across domains, "evaluation" has multiple, inconsistent definitions and is often mixed up with measurement and testing, making results hard to interpret and trust. Evaluatology seeks rigorous concepts and models so that evaluation outcomes are clearly defined, explainable, and reliable for high-stakes decisions.
Evaluatology: A Unified Science and Engineering of Evaluation
Evaluatology provides axioms and methods that clarify true value, traceability, comparability, and consistency of evaluation results, turning benchmarks and indices into well-founded tools rather than arbitrary artifacts. By formalizing evaluation systems and conditions, it enables consistent, comparable, and theoretically grounded evaluation across domains.