Can a rule score detect arithmetic errors?
FL-EXP-002 · 2026-10-15 · 160 executed production-function evaluations.
Observed results
All 80 true/false arithmetic pairs received equal scores and issue labels. Plain wording scored 52; procedural framing scored 82 in English and 91 in Chinese, including incorrect answers.
The current rule score cannot verify arithmetic truth. It inspects visible writing features; a high score is not factual correctness and must not be an automatic acceptance threshold.
Method, scope and reproduction
Additions from 11 to 20, true/false pairs, two languages and four wording conditions. All 160 evaluations retained; no model calls. No human participants, paper reproduction or independent dual review. Other error-detection rates cannot be inferred.
All inputs, outputs and source hashes · Protocol and reproduction instructions · Runner (requires the repository)
From the repository root:
node docs/research-experiments/2026-10-15-rule-score-audit/run.mjs