A new C-based engine called Colibri is breaking hardware barriers by running the massive 744-billion-parameter GLM 5.2 model on a standard consumer laptop with only 25 GB of RAM. Normally requiring tens of thousands of dollars in GPU hardware or high-end Mac Studios, this breakthrough leverages a clever software loophole to stream weights directly from solid-state drives (SSDs) rather than relying solely on system memory.
How Colibri Leverages Mixture of Experts (MoE)
The secret behind Colibri’s feasibility lies in the architecture of Mixture of Experts (MoE) models. In GLM 5.2, layers contain 256 smaller neural networks (experts) managed by a router. For any given token, the router selects only the eight most relevant experts, leaving the remaining 248 inactive. Consequently, only about 5% (40 billion parameters) of the model’s total size is active at any moment. Colibri pins the essential, permanently active parts of the model (attention layers, embeddings, and shared experts, totaling about 10 GB) into RAM. The remaining 21,500 experts sit on the SSD and are fetched dynamically as needed.
Smart Prefetching vs. Standard OS Memory Mapping
While tools like llama.cpp can memory-map models larger than system RAM, they rely on the operating system’s blind page faulting, which pauses processing. Colibri improves on this by utilizing the model’s internal router. Because the router determines which experts are needed ahead of time, Colibri issues asynchronous, parallel requests to fetch the exact experts from the SSD while the CPU continues processing the current layer. It even attempts to predict the next layer’s required experts early, achieving 72% prediction accuracy to hide disk latency.
Current Trade-Offs and Real-World Performance
The primary compromise of this approach is processing speed. Depending on the hardware, token generation speeds range from 0.1 tokens per second (t/s) on budget SSDs to 1.0 t/s on top-tier M5 Max internal SSDs. While too slow for interactive chat, this equates to roughly 8,000 high-quality tokens per day. Furthermore, because streaming consists entirely of read operations, it does not degrade SSD lifespan (which is only worn down by writes), though cheap drives may encounter thermal throttling under sustained loads.
A New Tier for Local AI Inference
Colibri is not an isolated experiment. Prominent developers are building similar SSD-streaming runtimes, such as antirez’s DS4, and mainstream projects like llama.cpp and vLLM are implementing expert caching. As predictive routing algorithms improve beyond the current 72% accuracy mark, background loading will eliminate more disk latency. This shift transforms the SSD from a basic filing cabinet into an active tier of system memory, allowing developers to run frontier-class models locally and securely without data-center budgets.
Mentoring question
If execution speed is no longer the primary bottleneck, how could your team utilize slow, cost-effective, but highly capable frontier-class models locally for tasks like offline code refactoring or sensitive document analysis?
Source: https://youtube.com/watch?v=19xCOJxWU0A&is=i86BA7alg-PlmHGl