Model Regression Gate
A reproducible model promotion gate with versioned tasks, paired accuracy comparisons, slice tolerances and a separate review-required outcome.
Repository · Experiment metadata and decisions · Release v0.1.0
Problem and relevance to agents
An aggregate accuracy score can hide a regression in a customer segment. Invalid outputs, uncertain comparisons and pending human review also need different release decisions.
An explicit contract binds candidate and reference predictions to the same tasks and evaluates each required slice before permitting promotion.
Architecture
- Validate identities and outputs. Task versions, content hashes, unique IDs, labels and exact prediction coverage are checked before comparison.
- Compare paired outcomes. Per-slice gains and losses produce approximate Wilson difference intervals with a Bonferroni adjustment and frozen tolerances.
- Keep three decisions. Supported regressions and contract violations fail. Uncertainty, insufficient samples or a human-review flag require review. Failure takes precedence; no human approval is invented.
What this demonstrates
- Real local training and inference of a small multinomial naive Bayes support-ticket classifier.
- Frozen train, development and evaluation partitions with preserved improvement, deliberate regression and review-required outcomes.
- Offline replay of 31 decisions across 90 verified file hashes, plus 14 regression and contract tests.
Published results
The single-process CPU experiment used Python 3.15.0, an Intel Xeon W-3223, 48 GiB RAM and macOS 26.7.1.
Baseline training has 60 examples and the improved model has 120. The negative model rotates training labels. Evaluation contains 600 synthetic texts across six slices; each slice uses tolerance 0.10 and minimum count 50. Ten controls shuffle training order with seeds 0 through 9.
Held-out promotion decisions
| Evaluation candidate | Correct / 600 | Decision |
|---|---|---|
| Baseline against itself | 400 | PASS |
| Improved vocabulary | 600 | PASS |
| Rotated training labels | 0 | FAIL |
| Ten shuffled-order controls | 400 each | PASS, all ten |
The fixtures share twelve intent words and correlated templates. Numeric text variants are not independent semantic examples. Perfect fixture accuracy and passing controls do not establish production quality or a production false-positive rate.
At 50 rows per development slice, the baseline and improved model require review. All ten development controls also require review without failing. Missing or invalid outputs fail; uncertainty and explicit human-review cases remain separate.
Wall time was 0.4965 s, process CPU time 0.4056 s and peak RSS 25,161,728 bytes. These include training, inference, report writing and hardware queries, not isolated inference latency. Cost and energy were not measured.
Reproduce
Use uv 0.8.23 and Python 3.15.0. The setup command creates a repository-local environment; uv must be able to locate the matching interpreter. Install that Python version first if it is unavailable in uv's download catalog. Initial setup and test dependencies may need Internet access. Replay and unittest tests use only the standard library, without model downloads, credentials, GPUs or hosted inference.
After environment setup, saved-evidence replay verifies hashes and recomputes decisions offline without training models. The second block runs small regression tests, including training tiny fixture models. For a full experiment, follow the README in a clean checkout and serialize it with the shared benchmark lock on the original host. Preserve the published evidence because the full experiment overwrites its output files.
Verify saved evidence
git clone https://github.com/lucioduran/model-regression-gate.git
cd model-regression-gate
git checkout 1631dc59ad5037b8cf6c2dac5613af6d3255da4a
UV_CACHE_DIR="$PWD/.tools/uv-cache" UV_PYTHON_INSTALL_DIR="$PWD/.tools/python" uv venv --python 3.15.0 .venv
PYTHONPATH=src .venv/bin/python scripts/replay.pyRun regression tests
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -vScope and limitations
- The classifier learns lexical routing from owned synthetic examples; it is not an LLM evaluation or a test of natural support traffic, distribution shift or adversarial robustness.
- Order-invariance controls represent the same effective model and are dependent. Their counts are consistency checks, not independent trials estimating false-positive probability.
- Wilson intervals are approximate, including after the slice adjustment. A poor baseline can pass unchanged; applications need separate absolute quality requirements.
- Caller-provided model hashes identify claims but cannot authenticate inference. Hash manifests establish saved consistency, not signatures, independent measurement attestation or human approval.
Sources and revision
Inputs and policy were frozen at commit c2fe489. The recorded experiment used 2937f6b7efb3ec07f12ab2253d56a5610e3573c8 and began at the timestamp below. The recorded finish time marks completed cases before hardware queries; timer readings were taken later, before manifest writing. The final release retains trained artifacts, predictions, decisions and source hashes.
- Reviewed revision:
1631dc59ad5037b8cf6c2dac5613af6d3255da4a - Revision recorded by the experiment:
2937f6b7efb3ec07f12ab2253d56a5610e3573c8 - Report timestamp (UTC):
- Release published: v0.1.0 ·
- Experiment metadata and decisions SHA-256:
c0625013f7950f3f5821cf6f2afaf0287d18944099d9f9601230cd1c51aee32d