# Model Behavior Regression Explorer

> Compare eval results of two model versions: accuracy, regressions, McNemar p-value, flaky items. Paste eval results for two model versions on the same items (CSV or JSONL with id, input, expected, output_a, output_b and optional category). Each output is scored with your metric – exact, normalised exact, contains, numeric tolerance or regex – then accuracy is compared per version and category, regressions and improvements are listed, an exact McNemar test tells whether the difference is more than noise, and flaky (inconsistent repeated runs) or ambiguous items are flagged.

Part of usefulweb.site (https://usefulweb.site/). Results are deterministic calculations from the inputs and the listed assumptions; the API returns exactly what the web page shows.

## Use it
- Human interface: https://modelbehaviorregressionexplorer.usefulweb.site/ (interactive tool at https://modelbehaviorregressionexplorer.usefulweb.site/tool)
- JSON API: POST https://modelbehaviorregressionexplorer.usefulweb.site/api/v1/calculate with a JSON object of inputs (schema: https://modelbehaviorregressionexplorer.usefulweb.site/api/v1/tool)
- OpenAPI: https://modelbehaviorregressionexplorer.usefulweb.site/openapi.json
- MCP (Streamable HTTP): https://modelbehaviorregressionexplorer.usefulweb.site/mcp - tool `model_behavior_regression_explorer`

## Inputs
- data (textarea, required): Eval results (CSV or JSONL) - Columns: id, input, expected, output_a, output_b, category (optional). Repeated ids = repeated runs.
- metric (select): Scoring metric
- tolerance (number): Numeric tolerance (absolute)
- labelA (text): Name of version A
- labelB (text): Name of version B
- maxListed (integer): Maximum items listed per table

## Method
- Metrics: exact (trimmed), normalised exact (Unicode NFKC, lower case, punctuation/symbols and the articles a/an/the removed, whitespace collapsed), contains (normalised expected appears as whole words in the normalised output), numeric (last number in the output vs the expected number, absolute tolerance; thousands separators removed), regex (expected is a case-insensitive regular expression).
- Repeated ids are repeated runs: an item counts as correct for a version when the majority of its runs are correct; items whose runs disagree are flaky and excluded from the McNemar test.
- Accuracy = correct / scored items per version. Regression = A correct, B wrong; improvement = A wrong, B correct.
- Exact McNemar test: two-sided binomial test of the discordant pairs (b = only A correct, c = only B correct) against p = 0.5: p = min(1, 2·Σ_{i≤min(b,c)} C(b+c, i)/2^(b+c)). Significant at α = 0.05.
- Ambiguous: empty expected value, unparsable regex/number, or a near miss that only fails because of formatting (e.g. would pass under the normalised or contains metric).