Model Behavior Regression Explorer

How it works

Paste eval results for two model versions on the same items (CSV or JSONL with id, input, expected, output_a, output_b and optional category). Each output is scored with your metric – exact, normalised exact, contains, numeric tolerance or regex – then accuracy is compared per version and category, regressions and improvements are listed, an exact McNemar test tells whether the difference is more than noise, and flaky (inconsistent repeated runs) or ambiguous items are flagged.

• Metrics: exact (trimmed), normalised exact (Unicode NFKC, lower case, punctuation/symbols and the articles a/an/the removed, whitespace collapsed), contains (normalised expected appears as whole words in the normalised output), numeric (last number in the output vs the expected number, absolute tolerance; thousands separators removed), regex (expected is a case-insensitive regular expression).
• Repeated ids are repeated runs: an item counts as correct for a version when the majority of its runs are correct; items whose runs disagree are flaky and excluded from the McNemar test.
• Accuracy = correct / scored items per version. Regression = A correct, B wrong; improvement = A wrong, B correct.
• Exact McNemar test: two-sided binomial test of the discordant pairs (b = only A correct, c = only B correct) against p = 0.5: p = min(1, 2·Σ_{i≤min(b,c)} C(b+c, i)/2^(b+c)). Significant at α = 0.05.
• Ambiguous: empty expected value, unparsable regex/number, or a near miss that only fails because of formatting (e.g. would pass under the normalised or contains metric).