{"schemaVersion":1,"id":"https://lucioduran.com/projects/quantization-quality-lab","url":"https://lucioduran.com/projects/quantization-quality-lab","title":"Quantization Quality Lab","description":"A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.","language":"en","datePublished":"2026-11-10","dateModified":"2026-11-10","contentFormat":"text/markdown","content":"[Projects](<https://lucioduran.com/projects>) / Quantization Quality Lab\n\n# Quantization Quality Lab\n\nA real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 and PyTorch 2.2.2 on CPU · License: [MIT](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE>)\n\nPage published: 2026-11-10 · Page updated: 2026-11-10\n\n[Repository](<https://github.com/lucioduran/quantization-quality-lab>) · [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>) · [Release v1.0.0](<https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0>)\n\n## Problem and relevance to agents\n\nSmaller quantized weights do not necessarily reduce process memory or improve every latency percentile. Aggregate quality scores can also hide changed retrieval ranks.\n\nA fixed model, corpus and protocol allow baseline and dynamic int8 inference to be compared on the same machine without treating file size as a memory or quality guarantee.\n\n## Architecture\n\n1. **Pin the model and recipe.** The Apache-2.0 paraphrase-MiniLM-L3-v2 encoder is fixed at revision 4ca70771034acceecb2e72475f72050fcdde4ddc. PyTorch dynamic qint8 quantizes its 19 Linear modules with fbgemm; embeddings remain float32.\n2. **Run separate processes.** Three fresh workers per mode alternate FP32 and int8 execution under the shared benchmark lock. Each loads its selected artifact and records loaded and peak process RSS separately.\n3. **Retain and replay observations.** Raw embeddings, rankings, token counts and timings support offline recalculation. Manifests bind evidence, source, model artifacts and frozen fixtures without redistributing weights.\n\n- [Design and provenance](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md>)\n- [Frozen measurement protocol](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md>)\n\n## What this demonstrates\n\n- Real inference at 32, 64 and 128 tokens within the model configuration limit, with explicit truncation counts.\n- A 23.50% reduction in serialized weight bytes alongside higher observed loaded and peak RSS on this loading path.\n- Thirty-one regression tests and dependency-free evidence verification, including observed ranking losses, slower tail latencies and retained outliers.\n\n## Published results\n\nThe experiment ran on an Intel Xeon W-3223, 48 GiB RAM and macOS 26.7.1 with two intra-op threads and one inter-op thread. It used 12 documents, four development queries and 24 held-out queries.\n\nThe frozen protocol uses 45 timed calls per mode and length, pooled across three workers after three warmups each. Forward timing includes pooling and normalization but excludes tokenization and loading. Retrieval uses the same texts, labels and token caps in both modes; no calibration is required for this dynamic recipe. MRR averages the reciprocal rank of the relevant document.\n\n### Storage and process memory\n\n| Measurement | FP32 | Dynamic int8 |\n| --- | --- | --- |\n| Serialized weight bytes | 69,576,640 | 53,223,452 |\n| Median RSS after load, MiB | 355.22 | 404.26 |\n| Median process peak RSS, MiB | 429.11 | 468.55 |\n\n### Warm inference latency\n\n| Token count | FP32 median / p95, ms | Int8 median / p95, ms |\n| --- | --- | --- |\n| 32 | 8.495 / 12.692 | 5.536 / 8.277 |\n| 64 | 7.674 / 9.367 | 7.343 / 10.498 |\n| 128 | 12.549 / 13.716 | 11.188 / 15.176 |\n\n### Held-out retrieval quality\n\n| Token cap | Top-1 correct, both modes | FP32 MRR | Int8 MRR |\n| --- | --- | --- | --- |\n| 32 | 19 / 24 | 0.850116 | 0.849124 |\n| 64 | 19 / 24 | 0.849124 | 0.849124 |\n| 128 | 21 / 24 | 0.930556 | 0.930556 |\n\nMedian inference improved by 1.05x to 1.53x, while int8 p95 worsened at 64 and 128 tokens. Loaded RSS and process peak RSS are distinct observations; neither is pure weight memory. The measured path supports a storage reduction, not a deployment-memory saving claim.\n\nTop-1 predictions did not change under quantization. Query long4 fell from rank 6 to 8 at 32 tokens and from 7 to 8 at 64 tokens, while long6 improved from 8 to 7. At 128 tokens there were no rank changes, although embeddings still drifted. [Measured results and negative outcomes](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md>).\n\nFour long texts were truncated at 32 and 64 tokens, sometimes removing the actual question. All 40 texts fit at 128 tokens. The four development queries reached top-1 accuracy and MRR of 1.0 in both modes at every cap.\n\nOnly the initial 32-token call in each worker was process-cold. The 3.14-second FP32 first call and 107.60-ms int8 shape-first call at 64 tokens remain in the evidence. Later lengths were already warm, and disk caches were not flushed.\n\n## Reproduce\n\nUse uv 0.8.23 and Python 3.11.13. The setup command creates a repository-local environment; uv must be able to locate the matching interpreter. Install that Python version first if it is unavailable in uv's download catalog. Initial setup and test dependencies may need Internet access. Replay uses only the standard library and does not load model weights. The separate test block installs the pinned development requirements and wheel 0.45.1. Full inference needs the larger benchmark environment documented in the repository.\n\nAfter environment setup, the replay command recomputes saved results offline without loading a model and writes replay.json. A successful report says status consistent and authenticity\\_verified false. The second block installs pinned test tooling and the small package, then runs regression tests. Full model measurements are documented in the README and require the original shared lock infrastructure; do not bypass its check.\n\n### Verify saved evidence\n\n```\ngit clone https://github.com/lucioduran/quantization-quality-lab.git\ncd quantization-quality-lab\ngit checkout b122cfbf3d81ae4b01810dc57dc9fb526f89c05b\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" UV_PYTHON_INSTALL_DIR=\"$PWD/.tools/python\" uv venv --python 3.11.13 .venv\nPYTHONPATH=src .venv/bin/python -m quantization_quality_lab.cli verify > replay.json\n```\n\n### Run regression tests\n\n```\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" uv pip install --python .venv/bin/python -r requirements-dev.txt wheel==0.45.1\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" uv pip install --python .venv/bin/python --no-build-isolation .\n.venv/bin/python -m pytest -q\n```\n\n## Scope and limitations\n\n- One encoder, one machine and a small synthetic corpus do not establish general quality preservation, production performance or behavior of other quantization methods.\n- Only Linear layers are quantized. Float embeddings, temporary construction, allocator retention, packed weights and runtime libraries contribute to the observed process memory.\n- Fixed length order, unflushed caches and other machine activity remain confounders. Reciprocal-median embeddings per second is a batch-one derivation, not measured concurrent capacity.\n- The full benchmark harness uses macOS hardware collection and a host-specific shared lock path. Other benchmark platforms are untested, although offline verification runs in Linux CI.\n- The legacy pinned PyTorch path is not a deployment recommendation. No generation, perplexity, GPU, energy or monetary operating-cost measurement is reported. Replay establishes consistency, not authenticity.\n\n- [Related quantization article](<https://lucioduran.com/blog/llm-quantization-gptq-awq-gguf-compared>)\n\n## Sources and revision\n\nThe measured source was commit 93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba. The run lasted from 2026-11-09T20:28:41.396081+00:00 to 2026-11-09T20:30:17.590156+00:00. The release was published later, at the separate release timestamp below, and adds documentation and regression tests without changing measured source. Model, artifact, fixture and source hashes are retained.\n\n- Reviewed revision: [` b122cfbf3d81ae4b01810dc57dc9fb526f89c05b `](<https://github.com/lucioduran/quantization-quality-lab/commit/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b>)\n- Revision recorded by the experiment: [` 93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba `](<https://github.com/lucioduran/quantization-quality-lab/commit/93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba>)\n- Report timestamp (UTC): 2026-11-09T20:28:41.396081+00:00\n- Release published: [v1.0.0](<https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0>) · 2026-11-09T20:35:17Z\n- [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>) SHA-256: ` 8cd6da46740ed8e8783513158f6f321e3705dba080212b9294ee4baf4b9d6870 `\n- [Execution environment and timestamps](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json>) SHA-256: ` 10b145e1ffad3e95a7166e91ebfc38e9527c4d0d62c08f881fe43c3a9314e890 `\n- [Pinned model and artifact hashes](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json>) SHA-256: ` e6390cfcecc950909ac74c7ed3fb993420fefa4253cf1c0a21827548ac8ab5ef `\n- [Frozen documents and query partitions](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json>) SHA-256: ` 35fb5acba663653900b6a77d3d032477a59d3d39630239bda35fdca16dbcccfe `\n\n- [README and full reproduction](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/README.md>)\n- [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>)\n- [Execution environment and timestamps](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json>)\n- [Pinned model and artifact hashes](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json>)\n- [Measured results and negative outcomes](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md>)\n- [MIT license](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE>)\n- [Third-party model license](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/THIRD_PARTY.md>)\n- [Design and provenance](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md>)\n- [Frozen measurement protocol](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md>)\n- [Frozen documents and query partitions](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json>)\n- [Subprocess exits and warnings](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/execution.json>)\n- [Evidence hash manifest](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/manifest.json>)\n- [First FP32 worker observations](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/fp32-0.json>)\n- [First int8 worker observations](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/int8-0.json>)\n- [Release validation scope](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/validation.md>)","headings":[{"level":1,"text":"Quantization Quality Lab","url":"https://lucioduran.com/projects/quantization-quality-lab#title"},{"level":2,"text":"Problem and relevance to agents","url":"https://lucioduran.com/projects/quantization-quality-lab#problem"},{"level":2,"text":"Architecture","url":"https://lucioduran.com/projects/quantization-quality-lab#architecture"},{"level":2,"text":"What this demonstrates","url":"https://lucioduran.com/projects/quantization-quality-lab#demonstrates"},{"level":2,"text":"Published results","url":"https://lucioduran.com/projects/quantization-quality-lab#results"},{"level":3,"text":"Storage and process memory","url":"https://lucioduran.com/projects/quantization-quality-lab#results-storage-memory"},{"level":3,"text":"Warm inference latency","url":"https://lucioduran.com/projects/quantization-quality-lab#results-warm-latency"},{"level":3,"text":"Held-out retrieval quality","url":"https://lucioduran.com/projects/quantization-quality-lab#results-retrieval-quality"},{"level":2,"text":"Reproduce","url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce"},{"level":3,"text":"Verify saved evidence","url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce-replay"},{"level":3,"text":"Run regression tests","url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce-checks"},{"level":2,"text":"Scope and limitations","url":"https://lucioduran.com/projects/quantization-quality-lab#scope"},{"level":2,"text":"Sources and revision","url":"https://lucioduran.com/projects/quantization-quality-lab#sources"}],"sections":[{"heading":"Quantization Quality Lab","level":1,"url":"https://lucioduran.com/projects/quantization-quality-lab","content":"[Projects](<https://lucioduran.com/projects>) / Quantization Quality Lab"},{"heading":"Quantization Quality Lab","level":1,"url":"https://lucioduran.com/projects/quantization-quality-lab#title","content":"A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.\n\nAuthor: [Lucio Durán](<https://lucioduran.com/>) · Python 3.11.13 and PyTorch 2.2.2 on CPU · License: [MIT](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE>)\n\nPage published: 2026-11-10 · Page updated: 2026-11-10\n\n[Repository](<https://github.com/lucioduran/quantization-quality-lab>) · [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>) · [Release v1.0.0](<https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0>)"},{"heading":"Problem and relevance to agents","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#problem","content":"Smaller quantized weights do not necessarily reduce process memory or improve every latency percentile. Aggregate quality scores can also hide changed retrieval ranks.\n\nA fixed model, corpus and protocol allow baseline and dynamic int8 inference to be compared on the same machine without treating file size as a memory or quality guarantee."},{"heading":"Architecture","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#architecture","content":"1. **Pin the model and recipe.** The Apache-2.0 paraphrase-MiniLM-L3-v2 encoder is fixed at revision 4ca70771034acceecb2e72475f72050fcdde4ddc. PyTorch dynamic qint8 quantizes its 19 Linear modules with fbgemm; embeddings remain float32.\n2. **Run separate processes.** Three fresh workers per mode alternate FP32 and int8 execution under the shared benchmark lock. Each loads its selected artifact and records loaded and peak process RSS separately.\n3. **Retain and replay observations.** Raw embeddings, rankings, token counts and timings support offline recalculation. Manifests bind evidence, source, model artifacts and frozen fixtures without redistributing weights.\n\n- [Design and provenance](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md>)\n- [Frozen measurement protocol](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md>)"},{"heading":"What this demonstrates","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#demonstrates","content":"- Real inference at 32, 64 and 128 tokens within the model configuration limit, with explicit truncation counts.\n- A 23.50% reduction in serialized weight bytes alongside higher observed loaded and peak RSS on this loading path.\n- Thirty-one regression tests and dependency-free evidence verification, including observed ranking losses, slower tail latencies and retained outliers."},{"heading":"Published results","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#results","content":"The experiment ran on an Intel Xeon W-3223, 48 GiB RAM and macOS 26.7.1 with two intra-op threads and one inter-op thread. It used 12 documents, four development queries and 24 held-out queries.\n\nThe frozen protocol uses 45 timed calls per mode and length, pooled across three workers after three warmups each. Forward timing includes pooling and normalization but excludes tokenization and loading. Retrieval uses the same texts, labels and token caps in both modes; no calibration is required for this dynamic recipe. MRR averages the reciprocal rank of the relevant document."},{"heading":"Storage and process memory","level":3,"url":"https://lucioduran.com/projects/quantization-quality-lab#results-storage-memory","content":"| Measurement | FP32 | Dynamic int8 |\n| --- | --- | --- |\n| Serialized weight bytes | 69,576,640 | 53,223,452 |\n| Median RSS after load, MiB | 355.22 | 404.26 |\n| Median process peak RSS, MiB | 429.11 | 468.55 |"},{"heading":"Warm inference latency","level":3,"url":"https://lucioduran.com/projects/quantization-quality-lab#results-warm-latency","content":"| Token count | FP32 median / p95, ms | Int8 median / p95, ms |\n| --- | --- | --- |\n| 32 | 8.495 / 12.692 | 5.536 / 8.277 |\n| 64 | 7.674 / 9.367 | 7.343 / 10.498 |\n| 128 | 12.549 / 13.716 | 11.188 / 15.176 |"},{"heading":"Held-out retrieval quality","level":3,"url":"https://lucioduran.com/projects/quantization-quality-lab#results-retrieval-quality","content":"| Token cap | Top-1 correct, both modes | FP32 MRR | Int8 MRR |\n| --- | --- | --- | --- |\n| 32 | 19 / 24 | 0.850116 | 0.849124 |\n| 64 | 19 / 24 | 0.849124 | 0.849124 |\n| 128 | 21 / 24 | 0.930556 | 0.930556 |\n\nMedian inference improved by 1.05x to 1.53x, while int8 p95 worsened at 64 and 128 tokens. Loaded RSS and process peak RSS are distinct observations; neither is pure weight memory. The measured path supports a storage reduction, not a deployment-memory saving claim.\n\nTop-1 predictions did not change under quantization. Query long4 fell from rank 6 to 8 at 32 tokens and from 7 to 8 at 64 tokens, while long6 improved from 8 to 7. At 128 tokens there were no rank changes, although embeddings still drifted. [Measured results and negative outcomes](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md>).\n\nFour long texts were truncated at 32 and 64 tokens, sometimes removing the actual question. All 40 texts fit at 128 tokens. The four development queries reached top-1 accuracy and MRR of 1.0 in both modes at every cap.\n\nOnly the initial 32-token call in each worker was process-cold. The 3.14-second FP32 first call and 107.60-ms int8 shape-first call at 64 tokens remain in the evidence. Later lengths were already warm, and disk caches were not flushed."},{"heading":"Reproduce","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce","content":"Use uv 0.8.23 and Python 3.11.13. The setup command creates a repository-local environment; uv must be able to locate the matching interpreter. Install that Python version first if it is unavailable in uv's download catalog. Initial setup and test dependencies may need Internet access. Replay uses only the standard library and does not load model weights. The separate test block installs the pinned development requirements and wheel 0.45.1. Full inference needs the larger benchmark environment documented in the repository.\n\nAfter environment setup, the replay command recomputes saved results offline without loading a model and writes replay.json. A successful report says status consistent and authenticity\\_verified false. The second block installs pinned test tooling and the small package, then runs regression tests. Full model measurements are documented in the README and require the original shared lock infrastructure; do not bypass its check."},{"heading":"Verify saved evidence","level":3,"url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce-replay","content":"```\ngit clone https://github.com/lucioduran/quantization-quality-lab.git\ncd quantization-quality-lab\ngit checkout b122cfbf3d81ae4b01810dc57dc9fb526f89c05b\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" UV_PYTHON_INSTALL_DIR=\"$PWD/.tools/python\" uv venv --python 3.11.13 .venv\nPYTHONPATH=src .venv/bin/python -m quantization_quality_lab.cli verify > replay.json\n```"},{"heading":"Run regression tests","level":3,"url":"https://lucioduran.com/projects/quantization-quality-lab#reproduce-checks","content":"```\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" uv pip install --python .venv/bin/python -r requirements-dev.txt wheel==0.45.1\nUV_CACHE_DIR=\"$PWD/.tools/uv-cache\" uv pip install --python .venv/bin/python --no-build-isolation .\n.venv/bin/python -m pytest -q\n```"},{"heading":"Scope and limitations","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#scope","content":"- One encoder, one machine and a small synthetic corpus do not establish general quality preservation, production performance or behavior of other quantization methods.\n- Only Linear layers are quantized. Float embeddings, temporary construction, allocator retention, packed weights and runtime libraries contribute to the observed process memory.\n- Fixed length order, unflushed caches and other machine activity remain confounders. Reciprocal-median embeddings per second is a batch-one derivation, not measured concurrent capacity.\n- The full benchmark harness uses macOS hardware collection and a host-specific shared lock path. Other benchmark platforms are untested, although offline verification runs in Linux CI.\n- The legacy pinned PyTorch path is not a deployment recommendation. No generation, perplexity, GPU, energy or monetary operating-cost measurement is reported. Replay establishes consistency, not authenticity.\n\n- [Related quantization article](<https://lucioduran.com/blog/llm-quantization-gptq-awq-gguf-compared>)"},{"heading":"Sources and revision","level":2,"url":"https://lucioduran.com/projects/quantization-quality-lab#sources","content":"The measured source was commit 93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba. The run lasted from 2026-11-09T20:28:41.396081+00:00 to 2026-11-09T20:30:17.590156+00:00. The release was published later, at the separate release timestamp below, and adds documentation and regression tests without changing measured source. Model, artifact, fixture and source hashes are retained.\n\n- Reviewed revision: [` b122cfbf3d81ae4b01810dc57dc9fb526f89c05b `](<https://github.com/lucioduran/quantization-quality-lab/commit/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b>)\n- Revision recorded by the experiment: [` 93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba `](<https://github.com/lucioduran/quantization-quality-lab/commit/93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba>)\n- Report timestamp (UTC): 2026-11-09T20:28:41.396081+00:00\n- Release published: [v1.0.0](<https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0>) · 2026-11-09T20:35:17Z\n- [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>) SHA-256: ` 8cd6da46740ed8e8783513158f6f321e3705dba080212b9294ee4baf4b9d6870 `\n- [Execution environment and timestamps](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json>) SHA-256: ` 10b145e1ffad3e95a7166e91ebfc38e9527c4d0d62c08f881fe43c3a9314e890 `\n- [Pinned model and artifact hashes](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json>) SHA-256: ` e6390cfcecc950909ac74c7ed3fb993420fefa4253cf1c0a21827548ac8ab5ef `\n- [Frozen documents and query partitions](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json>) SHA-256: ` 35fb5acba663653900b6a77d3d032477a59d3d39630239bda35fdca16dbcccfe `\n\n- [README and full reproduction](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/README.md>)\n- [Recomputable measurement summary](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json>)\n- [Execution environment and timestamps](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json>)\n- [Pinned model and artifact hashes](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json>)\n- [Measured results and negative outcomes](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md>)\n- [MIT license](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE>)\n- [Third-party model license](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/THIRD_PARTY.md>)\n- [Design and provenance](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md>)\n- [Frozen measurement protocol](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md>)\n- [Frozen documents and query partitions](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json>)\n- [Subprocess exits and warnings](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/execution.json>)\n- [Evidence hash manifest](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/manifest.json>)\n- [First FP32 worker observations](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/fp32-0.json>)\n- [First int8 worker observations](<https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/int8-0.json>)\n- [Release validation scope](<https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/validation.md>)"}],"links":[{"text":"Projects","url":"https://lucioduran.com/projects"},{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"MIT","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE"},{"text":"Repository","url":"https://github.com/lucioduran/quantization-quality-lab"},{"text":"Recomputable measurement summary","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json"},{"text":"Release v1.0.0","url":"https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0"},{"text":"Design and provenance","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md"},{"text":"Frozen measurement protocol","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md"},{"text":"Measured results and negative outcomes","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md"},{"text":"Related quantization article","url":"https://lucioduran.com/blog/llm-quantization-gptq-awq-gguf-compared"},{"text":"b122cfbf3d81ae4b01810dc57dc9fb526f89c05b","url":"https://github.com/lucioduran/quantization-quality-lab/commit/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b"},{"text":"93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba","url":"https://github.com/lucioduran/quantization-quality-lab/commit/93eaf528dd32dfb5b4abf59a2e5c7d1c9b8f67ba"},{"text":"v1.0.0","url":"https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0"},{"text":"Execution environment and timestamps","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json"},{"text":"Pinned model and artifact hashes","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json"},{"text":"Frozen documents and query partitions","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json"},{"text":"README and full reproduction","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/README.md"},{"text":"MIT license","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE"},{"text":"Third-party model license","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/THIRD_PARTY.md"},{"text":"Subprocess exits and warnings","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/execution.json"},{"text":"Evidence hash manifest","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/manifest.json"},{"text":"First FP32 worker observations","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/fp32-0.json"},{"text":"First int8 worker observations","url":"https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/int8-0.json"},{"text":"Release validation scope","url":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/validation.md"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"WebPage","@id":"https://lucioduran.com/projects/quantization-quality-lab#page","url":"https://lucioduran.com/projects/quantization-quality-lab","name":"Quantization Quality Lab","description":"A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"mainEntity":{"@id":"https://lucioduran.com/projects/quantization-quality-lab#software"},"datePublished":"2026-11-10","dateModified":"2026-11-10","breadcrumb":{"@id":"https://lucioduran.com/projects/quantization-quality-lab#breadcrumb"}},{"@type":"SoftwareSourceCode","@id":"https://lucioduran.com/projects/quantization-quality-lab#software","name":"Quantization Quality Lab","description":"A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.","inLanguage":"en","url":"https://lucioduran.com/projects/quantization-quality-lab","codeRepository":"https://github.com/lucioduran/quantization-quality-lab","programmingLanguage":"Python","runtimePlatform":"Python 3.11.13 and PyTorch 2.2.2 on CPU","license":"https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/LICENSE","version":"v1.0.0","identifier":{"@type":"PropertyValue","propertyID":"git-commit","value":"b122cfbf3d81ae4b01810dc57dc9fb526f89c05b"},"author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"isAccessibleForFree":true,"citation":["https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/README.md","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/summary.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/environment.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/model.json","https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/results.md","https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/THIRD_PARTY.md","https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/design.md","https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/protocol.md","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/fixtures/tasks.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/execution.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/manifest.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/fp32-0.json","https://raw.githubusercontent.com/lucioduran/quantization-quality-lab/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/evidence/int8-0.json","https://github.com/lucioduran/quantization-quality-lab/blob/b122cfbf3d81ae4b01810dc57dc9fb526f89c05b/docs/validation.md","https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0"],"publisher":{"@id":"https://lucioduran.com/#person"},"mainEntityOfPage":{"@id":"https://lucioduran.com/projects/quantization-quality-lab#page"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Projects","item":"https://lucioduran.com/projects"},{"@type":"ListItem","position":2,"name":"Quantization Quality Lab","item":"https://lucioduran.com/projects/quantization-quality-lab"}],"@id":"https://lucioduran.com/projects/quantization-quality-lab#breadcrumb"}]}],"contentHash":"2fb4e565eaaad442e5339ecf917e29fd58de16a7a04229416288c55f227bfa29","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/projects/quantization-quality-lab","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":58546,"contentBytes":12197}}