AI/ML # MoE expert pruning: active parameters, memory budgets, and HOPE 2026-11-097 min read [Lucio Durán]() Engineering Manager & AI Solutions Architect Also available in: [Español]() ## A timely reason to revisit MoE memory Amazon published its [HOPE engineering explanation]() on November 9, 2026. The [paper]() first appeared in September. Today's publication makes expert pruning a timely deployment topic, without turning an earlier research result into a new release. The useful question is concrete: when a model activates only a fraction of its parameters for each token, why can it still need so much memory? In a mixture-of-experts (MoE) layer, a router selects several expert networks and combines their outputs. Active parameters describe computation along a selected path. Capacity planning must also account for parameters available to other paths, request state, and temporary allocations. An expert that is idle for one token can still occupy memory. This article develops a memory example and a small selection experiment, followed by a proposed evaluation protocol. It does not report a local reproduction of HOPE or a GPU benchmark. ## Separate four memory budgets Start with an inventory for a fixed model revision, numerical format, serving engine, and workload: | Budget | What belongs there | What to inspect | | --- | --- | --- | | Checkpoint storage | Serialized weights and quantization metadata | Actual files, rather than the model's parameter label | | Resident weights | Expert and non-expert tensors placed on each device | Placement, replication, precision, and offloading | | Request state | KV cache and other persistent per-request state | Context length, simultaneous requests, and cache policy | | Runtime headroom | Temporary tensors, kernels, communication buffers, allocator slack | Peak allocation during the real workload | These categories overlap across implementations, so define the accounting boundary before adding measurements. Do not add a framework's total reserved memory to its allocated memory: those are overlapping measurements. For a deliberately invented capacity example, assume 80 GiB of resident expert weights and 16 GiB of everything else at a fixed request profile. Removing half of those expert bytes leaves: ```text before = 80 + 16 = 96 GiB after = 40 + 16 = 56 GiB saved = 40 / 96 = approximately 41.7% ``` The reduction is 50% of expert storage and 41.7% of this hypothetical total. Neither number predicts tokens per second. The remaining 16 GiB was held constant only to make the arithmetic explicit; real runtime allocations can change after pruning. Offloading and expert parallelism change where weights live and when they move. They do not make the storage obligation disappear. Likewise, reclaimed capacity can support longer requests only if the serving system can use it without hitting another constraint. The existing articles on [quantization]() and [KV cache and batching]() cover adjacent mechanisms. A deployment can combine them, but their benefits must be measured together. ## Why independent scores can choose the wrong set HOPE selects a removal set through a quadratic objective, ` p^T F p `, with a fixed removal count. Pairwise terms influence the choice. Its error analysis separates substitution from renormalization; the surrogate is not a direct guarantee on total error or task quality. [Paper, section 3 and appendix A]() Here is an independent arithmetic example. Suppose three synthetic contribution magnitudes, A, B, and C, are observed on two samples: | Sample | A | B | C | | --- | --- | --- | --- | | 1 | 1 | 1 | 0 | | 2 | 0 | 0 | 1.5 | Define the cost of a removal set as the average squared sum of its contributions. This is an illustrative objective, not measured model damage. Scored alone, A and B each cost 0.5; C costs 1.125. Removing the two lowest individual scores selects A and B. Together, however, their contributions coincide: their joint cost is 2. Removing A and C costs 1.625. The disagreement comes from the cross term in ` (a + b)^2 `. Ranking individual squares cannot represent it. It does not establish that A and B understand the same subject, cooperate causally, or should always survive together. ## A runnable selection example The following uses only Python's standard library. It enumerates every pair for this tiny example and asserts the expected results. ```python from itertools import combinations names = ("A", "B", "C") samples = ((1.0, 1.0, 0.0), (0.0, 0.0, 1.5)) def removal_cost(indices): return sum( sum(sample[index] for index in indices) ** 2 for sample in samples ) / len(samples) individual = sorted(range(len(names)), key=lambda i: removal_cost((i,))) independent = tuple(sorted(individual[:2])) joint = min(combinations(range(len(names)), 2), key=removal_cost) assert independent == (0, 1) assert joint == (0, 2) assert removal_cost(independent) == 2.0 assert removal_cost(joint) == 1.625 for selected in (independent, joint): print(tuple(names[i] for i in selected), removal_cost(selected)) ``` Expected output, verified locally: ```text ('A', 'B') 2.0 ('A', 'C') 1.625 ``` B and C tie with A and C. Iteration order picks A and C; there is no unique winner. Exhaustive enumeration is suitable for three candidates, not a proposed large-model optimizer. The example omits routing replacements, renormalization, and interactions between layers. The [published solver]() instead uses continuous relaxation and rounding, without certifying a global optimum. It also normalizes pair statistics by co-selection counts. Our example averages over all samples. Those choices produce different objectives; this example is not an implementation of the released algorithm. ## Turn compression into a controlled comparison Before collecting calibration data, write down the decision the experiment must support. For example: fit a model on a smaller device configuration while retaining acceptable correctness on a particular workload. Keep calibration, configuration selection, and final evaluation separate. A coding-heavy calibration set should not silently become the definition of acceptable behavior for a multilingual support workload. Preserve rare tasks as named evaluation slices instead of allowing their failures to disappear inside an average. I would compare the original checkpoint, a simple independent-score pruning baseline, and the pairwise method at identical removal counts. If quantization is part of deployment, evaluate each candidate at that same precision as well. Keep the tokenizer, prompts, decoding settings, serving engine, and request schedule fixed wherever the comparison permits. | Measurement | Proposed acceptance question | | --- | --- | | Task correctness by slice | Which required behaviors regress, and by how much? | | Failed tool calls and invalid outputs | Does compression break execution contracts? | | Resident and peak device memory | Does the candidate fit with realistic headroom? | | Time to first token and time per output token | Which phase changes under the same load? | | Completed requests per unit time | Does the same service objective support more work? | | Calibration and preparation cost | Is preparing the checkpoint worthwhile at the expected usage? | For noisy measurements, retain per-request observations and repeat runs. For stochastic answers, compare the same tasks across multiple samples. Report the distribution and uncertainty rather than choosing the best run. The released [pruning code]() preserves the original top-k routing count. Fewer stored experts therefore do not imply proportionally fewer expert evaluations per token. That is a specific reason to separate memory savings from speed claims. ## Read benchmark gains with their denominator attached For Qwen3.5-35B-A3B at 50% pruning, the authors report SWE-Bench Pro scores of 42.3 for HOPE, 41.8 unpruned, and 36.2 for REAP. These are their reported results, not measurements from this article. [Published comparison]() A difference of 6.1 score points against another compressed model answers a different question from improvement against the original checkpoint. Neither comparison establishes lower latency, a universal quality improvement, or the outcome of a different calibration distribution. The practical release artifact should include the exact source model and code revisions, calibration identity, removal set, checkpoint hashes, evaluation configuration, and rejected cases. Keep the original checkpoint available for rollback and test restoration before depending on the compressed artifact. Choose a candidate only after it satisfies the memory target and the workload's quality limits. If it saves memory but misses an essential task slice, the deployment constraint has not been solved. ## Frequently asked questions ### Does removing half the experts halve inference latency? No. Expert count alone does not determine the work per token, attention cost, communication, or scheduling. Measure the compressed checkpoint on the intended serving stack. ### Is the Python example a reproduction of HOPE? No. It exhaustively scores three synthetic contributions to illustrate why independent rankings and joint selection can disagree. It does not load a model or reproduce the published solver. ### Were the reported model benchmarks independently reproduced for this article? No. Model benchmark figures are attributed to their authors. Only the small, dependency-free arithmetic example was executed locally. ## Tips 1. Separate checkpoint bytes, resident weights, KV cache, and peak runtime memory. A percentage without a denominator is a poor capacity plan. 2. Freeze calibration and evaluation sets before selecting experts. Rare production tasks deserve their own acceptance checks. 3. Measure latency at the same workload and concurrency. Extra memory capacity and faster requests are different outcomes. - mixture-of-experts - llm-inference - model-compression - expert-pruning - hope - evaluation [Back to Blog]()