Kimi K3 Model Runs on 8GB Memory Devices, Developers Open Source Lightweight C Inference Engine

Developers have open-sourced the project kimi-k3-in-c, aimed at enabling the Kimi K3 model, which has 2.78 trillion parameters, to run on devices with only 8GB of memory. The project is 176KB in size and is written entirely in C99, without relying on GPU, CUDA, PyTorch, or BLAS, completing model inference solely through the CPU. This solution leverages the MoE (Mixture of Experts) architecture feature of Kimi K3, where although the total parameter scale of the model reaches 2.78T, only 16 out of 896 experts in each layer are activated. Therefore, the complete model weights of approximately 1.56TB are not loaded into memory; instead, most expert weights are stored on NVMe drives and read in real-time according to inference needs. Additionally, some dense layers are also loaded in a layer-by-layer streaming manner. Under the 8GB memory mode, the model generates one token in about 32.7 seconds and requires nearly 1.7TB of high-speed storage space to support it. The developers state that this solution is currently more of an experimental exploration of optimizing the infrastructure for large model inference and does not possess practical production value, but it provides a new idea for running ultra-large-scale models at low cost in the future.

This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.