Optimizing Large Language Model Performance on Apple Silicon: A Kimi K3 Benchmark Update
In the ever-evolving landscape of artificial intelligence, efficiency and performance optimization for large language models (LLMs) are paramount—especially when running on consumer-grade hardware such as Apple’s MacBooks. Recent experiments have demonstrated promising progress in harnessing the power of an M1 Max MacBook Pro for hosting substantial models like Kimi K3 (with 2.8 trillion parameters).
Progress Overview
Initially, executing the full Kimi K3 model on an M1 MacBook yielded throughput rates of approximately one token per minute. After comprehensive profiling, iterative experimentation, and targeted optimizations, recent benchmarks reveal a substantial improvement, achieving a median rate of roughly 4.1 tokens per minute. This corresponds to about 14.6 seconds per token and a throughput of 0.069 tokens per second—a notable fourfold enhancement for such a sizable model.
Key Optimization Techniques
The advancements stem from several strategic modifications:
- Selective Loading of Experts
The model employs a mixture-of-experts (MoE) architecture. By selectively loading only the 16 routed experts required per layer, and leveraging parallel raw-span reads, memory utilization became more efficient, reducing I/O bottlenecks.
- Quantization and Kernel Fusion
The resident model’s core spine was quantized to 8-bit integers (int8), significantly decreasing memory footprint. A fused Metal dequantization and copy kernel was utilized to streamline data processing, resulting in faster inference.
- Enhanced Output Projection Processing
Running the extensive output projection via Apple’s Metal Performance Shaders (MPS) with packed int8 matrix multiplication further optimized performance. This approach decreased residency from approximately 4.7 GB to 1.17 GB and increased median decode throughput by around 17%.
Hardware Context
All measurements were conducted on a single 64 GB M1 Max MacBook Pro—an early-generation Apple Silicon device. While capable, it is not as advanced as the latest M3, M4, or M5 chips, nor those with larger memory configurations (e.g., 128 GB). It is reasonable to expect that newer hardware, especially with higher RAM capacity, will deliver even better performance margins.
Future Outlook
These developments illustrate that with targeted software optimizations, it is feasible to run and experiment with large-scale models on relatively modest hardware. The community’s insights and benchmarking on newer Apple Silicon models would be highly valuable, and further performance improvements are anticipated as hardware continues to evolve.
Resources
For those interested in exploring these optimizations further, the complete implementation details and code are available in the Deltafin repository: https://github.com/gavamedia/deltafin.
Conclusion
While still in the realm of experimental optimization, achieving an average of over four tokens per minute on a single MacBook marks a significant milestone. This progress opens avenues for more accessible AI research and deployment on consumer hardware, bringing powerful language models closer to practical, everyday use.
Author’s note: Benchmarking on newer hardware and sharing results would be highly appreciated to further inform the community.
