Higher-Order Pruning of Experts (HOPE) for Mixture-of-Experts Models

Understanding the MoE Memory Bottleneck

Mixture-of-Experts (MoE) models, such as Qwen3.5-122B, often store hundreds of experts in memory even though only a small subset is activated per token. In many architectures, expert MLPs account for up to 95% of the total parameter memory footprint. While expert pruning is an effective strategy to reduce this footprint without requiring retraining or dynamic off-loading, existing first-order methods often degrade performance by treating experts as independent entities. These methods fail to account for the fact that experts frequently form stable, cooperative groups that must be preserved together to maintain model capability.

The HOPE Approach: Accounting for Expert Interaction

HOPE (Higher-Order Pruning of Experts) improves upon first-order pruning by modeling pairwise expert interactions. Rather than assigning a single scalar importance score to each expert, HOPE constructs an interaction matrix $F$ that captures the co-activation patterns and joint contributions of expert pairs.

By minimizing a surrogate objective function that upper-bounds the pruning error, HOPE solves a quadratic program (QP) to identify the optimal set of experts to remove. The diagonal of the $F$-matrix captures individual importance (similar to existing methods like REAP), while the off-diagonal entries quantify the strength of cooperative relationships. This allows the algorithm to sacrifice individually high-scoring experts if their removal would disproportionately damage a critical cooperative cluster.

Performance and Benchmarking

HOPE demonstrates superior performance compared to first-order baselines—including REAP, MAN, and EAN—across various models and calibration datasets. Key findings include:

  • Efficiency at Scale: HOPE's advantage becomes more pronounced at higher pruning rates (40% to 50%), where the preservation of cooperative structures is critical.
  • Agentic Task Superiority: On the SWE-Bench Pro benchmark, which requires complex, multi-turn reasoning and tool use, HOPE-pruned models significantly outperform those pruned via first-order methods. For example, a Qwen3.5-35B model pruned by 50% using HOPE achieved a score of 42.3, effectively matching the performance of the unpruned model.
  • Negligible Overhead: The method requires only a single calibration pass over the data, followed by a standard QP solver execution that takes 1–2 seconds per layer, making it highly practical for deployment.

Implementation and Workflow

The HOPE pipeline consists of three distinct steps, supported by the official HOPE GitHub repository:

  1. Calibration: A forward pass over calibration data records expert selection, gate weights, and output norms to generate the $F$-matrix.
  2. Solving: The system solves the binary quadratic program to produce a pruneset.json file defining which experts to remove.
  3. Pruning: The model is pruned based on the generated set, resulting in a standard Hugging Face checkpoint that is smaller in memory and on disk.

Limitations and Future Outlook

While HOPE is highly effective at 40-50% pruning, its performance gains are more modest at lower pruning rates (10-20%). Additionally, the current implementation prunes layers independently, ignoring potential cross-layer interactions. While a theoretical framework for cross-layer pruning exists, it remains in the preliminary stages. Future research may explore incorporating higher-order interactions (triplets and beyond) to further refine the pruning process, though these present significant combinatorial challenges.

Sources