Evaluating LLMs with option-free, deterministic scoring.
A benchmark score measures not only the model, but also the instrument used to evaluate it.
Open response + judge
Expressive answers, but the result can depend on who evaluates them.
Option-free + deterministic
Construct a canonical subset, ordering, or matching, then verify it exactly.
Visible final options
Easy to score, but answer choices can turn recall into recognition.
The evaluation problem
MCQs expose candidate answers. CRQs remove those candidates, but their scores depend on an evaluator. CSRQs examine whether these useful functions can be recomposed.
Select from visible final answers.
Construct a canonical structured answer.
Generate an open natural-language response.
How CSRQ works
CSRQs use familiar multi-select, ordering, and matching structures as canonical answer spaces for LLM evaluation, not as new human-test item archetypes.
Return every statement that satisfies the question.
Canonicalize → compare with key → exact score
𝟙 [ canonicalize(prediction) = canonicalize(answer) ]No judge requiredThe cultural testbed
A Malaysia-centered benchmark in Bahasa Malaysia, built to hold cultural knowledge constant while changing how that knowledge is elicited and scored.
Empirical validation
Across benchmark families and model scales, removing final-answer combinations changes performance, even when scoring remains deterministic.
Across MalayMMLU and CS-EN, models consistently score lower under CSRQ evaluation.
Every bar shows its exact score. Hover, focus, or click a model to update the breakdown above; use the dataset controls to simplify the chart.
Each bar shows how far one model's score moves when only the question format changes. It tells you how sensitive the score is to the format, not how capable the model is.
Toggle either gap to isolate option-removal sensitivity or the shift to judge-based scoring.
Judge-dependent scoring
The same CRQ outputs were evaluated under the same rubric by different LLM judges. Pairwise ranking agreement ranged from 0.40 to 1.00.
GPT-5 evaluated by Qwen3-30B Inst. This evaluated model varies by σ = 12.80 points across judges.
Hover, focus, or click a cell for context. Highlight a row to compare how judges score one model.
Benchmark conversion
The documented workflow converts suitable MCQ or CRQ items into CSRQs through knowledge acquisition, structured generation, and independent verification.
Extract the factual, relational, or procedural knowledge point.
Choose multi-select, ordering, or matching for the target knowledge.
Construct the question, statements, and canonical answer.
Check structure, answerability, and source-supported evidence.
Practical selection
For broad, low-cost screening and reproducible leaderboard comparison.
When answer-option scaffolding may distort absolute scores and judge-free scoring matters.
When expressive responses are essential and reliable human or multi-judge adjudication is available.
Responsible interpretation
01CSRQs do not prove genuine understanding or higher-order comprehension.
02They work best when an answer can be decomposed into valid response units.
03The answer space remains finite and does not fully replicate open generation.
04Construction adds costs for generation, distractor design, and verification.
Resources
The paper, MyCulture dataset, and evaluation toolkit are available now.
@inproceedings{hew2026myculture,
title={Between Multiple Choice and Open Response:
Evaluating LLMs with Option-Free Deterministic Scoring},
author={Hew, Zhong Ken and Yang, Sze Jue and Chan, Chee Seng},
booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
year={2026}
}