MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints

Mixture of Experts (MoE) LLMs, with sparse activation patterns, offer a promising approach to scaling language models while avoiding proportional cost increases. However, their large parameter sizes pose deployment challenges in resource-constrained environments with limited GPU memory. Typical deployments use CPU-GPU hybrid execution, where the GPU handles compute-intensive GEMM operations and the CPU processes the attention mechanism. This setup introduces a challenge: optimizing resource utilization across CPU and GPU. Prior work designs system optimizations based on performance models with a limited scope that don't capture complex hardware-system interactions. Therefore, they neither identify nor achieve hardware limits. This paper presents MoE-Lens, a high-throughput MoE LLM inference system designed for resource-constrained environments. MoE-Lens utilizes a novel performance model that holistically analyzes CPU memory capacity, GPU compute power, and workload characteristics to understand the theoretical performance upper bound of MoE inference. It also captures system execution mechanisms, including workload scheduling and the effects of paged KV cache, to identify key hardware bottlenecks and predict achievable throughput. Guided by such a performance model, MoE-Lens introduces a resource-aware scheduler, an execution engine that overlaps computation, a data transfer mechanism for model weights, and an optimized CPU-based attention implementation. Evaluated on diverse MoE models and datasets, MoE-Lens outperforms the state-of-the-art MoE-Lightening by 4.6 × (up to 25.5 ×), with our theoretical model predicting performance with an average 94% accuracy.