# FL-EXP-002: arithmetic truth versus presentation in the answer-quality checker

Executed 2026-10-15. This is a completed software experiment using synthetic inputs, **not a paper reproduction, human study, accuracy benchmark, or independent review**. Human participants: 0.

## Question and method

Does the production answer-quality heuristic distinguish true from false arithmetic when presentation is held fixed? Does adding procedural language change its score without repairing the false arithmetic?

Run `node docs/research-experiments/2026-10-15-rule-score-audit/run.mjs` from the repository root. The script executes the existing production analyzer through `scripts/answer-quality-analysis.mjs`; it does not implement a replacement scoring function. `results.json` records the analyzer and loader SHA-256, Node version, every input, truth oracle, complete output and pair comparison.

We exhaustively used integers 11–20. Each true statement n+n=2n has a false partner n+n=2n+1 with the same number of digits. Each pair appears in English and Chinese, with four presentation conditions: plain statement; numbered procedure with verification and limitations; that procedure plus an unverified example.com URL; that procedure plus inert bracketed URL text. 10 × 2 truth values × 2 languages × 4 conditions = 160 evaluations (80 matched pairs). The URL is a deliberately unverified synthetic string, not a reference supporting arithmetic. There are no network calls or language-model judgments. No records were excluded after execution.

## Observed results

All 80/80 matched pairs received identical scores **and** identical issue-title lists, despite opposite truth values.

| Condition | English score (20 inputs) | Chinese score (20 inputs) |
| --- | ---: | ---: |
| Plain | 52 | 52 |
| Structured | 82 | 91 |
| Structured + URL | 82 | 91 |
| Structured + inert URL | 82 | 91 |

Every value in each cell is the same; the table does not hide variance. The plain-to-structured change is +30 points in English and +39 in Chinese, for correct and incorrect statements alike. URL formatting did not change these particular outputs; this does not establish robustness to unrelated citation attacks.

## Interpretation and limits

The checker measures wording signals, not mathematical correctness. A wrong answer can earn a high rule score when accompanied by procedural language. This confirms the need for the existing boundary text and means a high score must never unlock a “verified” status. A correct concise answer also receives a low score in these probes: short factual replies need not have a multi-step plan.

The wrapper simultaneously changes length, structure and keyword presence, so this experiment does not isolate which of those features caused each score difference. English and Chinese wrappers are comparable in intent but not equal in character count; the between-language difference is not a general fairness estimate. These are narrow, hand-authored, deterministic cases, with no sampling-based confidence intervals, significance test, real-user effect size or estimate of performance over natural questions.

## Product implications

Keep score labels as “rule-check score,” retain direct access to findings and input, and require a separate external verification record for truth claims. Link this report from the tool and research overview as an executed limitations audit. It cannot replace the pending independent two-reviewer calibration or a real participant study.