AI & ComputeLOT 261122 · 3 min read
Two RTX 3090s Plus a 414 GB RAM Drive Lift Optane Kimi Build to 3.11 Tokens/s
A homelabber's second-gen Optane build pairs two RTX 3090s with a 414 GB tmpfs RAM drive to run the 2.8T-parameter Kimi K3 model at 3.11 tokens/s — 41% faster generation than Optane alone.
Key takeaways
- Two RTX 3090s plus a 414 GB tmpfs RAM drive run the 2.8T-parameter Kimi K3 Q2_K model at 3.11 tokens/s, up 41% from 2.20 on Optane alone.
- Prompt evaluation rose 26%, from 13.3 to 16.8 tokens/s.
- 45 of 94 model shards (414 GB of 865 GB) moved into DRAM; the rest stay on Optane DIMMs cooled by two 12W blower fans.
- The 48 GB of total VRAM covers only ~5.5% of the model; MoE routing (16 of 896 experts per token, 104B active parameters) makes the split feasible.
- The builder predicts further gains from 'customized quantization,' per-tensor or per-layer.
By Rebecca Stone
A homelabber's second-generation Optane rig now runs the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second — a 41% jump in generation speed earned by parking 414 GB of the 865 GB model in a RAM drive, flanked by two RTX 3090s. Prompt evaluation climbed 26%, from 13.3 to 16.8 tokens per second, according to the builder's own measurements posted in a follow-up on Reddit, run through llama.cpp with DAX enabled on the Optane portion.
The numbers are modest by datacenter standards. But they're honest, and they chart a workable path for anyone determined to serve a frontier-class model from consumer silicon.
What changed in the second-generation build?
Two things. First, the GPUs: a pair of RTX 3090s, 24 GB of VRAM each. Second, the memory split: 45 of the model's 94 shards moved off Intel Optane DIMMs and into a tmpfs RAM drive, the Linux file system that, per the kernel documentation, "keeps all of its files in virtual memory."
| Configuration | Model storage | Generation | Prompt evaluation |
|---|---|---|---|
| Optane only | All 94 shards | 2.20 tok/s | 13.3 tok/s |
| RAM drive + Optane | 45 of 94 shards (414 GB of 865 GB) | 3.11 tok/s | 16.8 tok/s |
The gain is roughly 41% on generation, 26% on prompt evaluation.
The GPU install didn't come free. The builder had to cut metal out of the case and relocate the PSU before the two 350W-class cards and their cooling would fit. And the thermals haven't improved: "the improved throughput [did not change] the original thermal warning," the builder noted, so the two 12W blower fans blowing on the Optane DIMMs stay in place.
Why don't the GPUs just hold the model?
They can't. Two RTX 3090s carry 48 GB of VRAM between them — about 5.5% of the 865 GB Q2_K model file. The arithmetic doesn't work, even before you ask what a 2.8T-parameter model weighs at higher precision: roughly 5.6 TB at 16-bit, or about 1.56 TB in Moonshot's own 4-bit MXFP4 release. The Q2_K quantization used here averages under 3 bits per parameter.
The trick is architecture. Kimi K3 is a mixture-of-experts model with 896 experts, of which 16 are routed to each token plus two shared ones, leaving 104B parameters active out of the full 2.8T. That split dictates the hardware layout:
- GPUs handle the parts that touch every token — attention, the dense feed-forward layer, shared experts, compute buffers, the KV cache and stored conversation context.
- CPU-side memory — Optane plus, now, the tmpfs DRAM drive — holds the routed-expert tensors, the bulk of the weights.
Because only a few experts fire per token, slow tiers only need to serve a fraction of the weights at any moment. That's the whole gambit.
Does DRAM undercut the Optane point of the project?
Somewhat, and the builder admits it. The original build existed to avoid buying enormous DRAM capacity by leaning on cheap persistent Optane instead. Moving 45 shards into tmpfs reintroduces the very cost the first design dodged — though DRAM is faster than Optane, and the distributed workload now reads those shards without touching the slower tier. Anyone who's set aside system memory as a pseudo-non-volatile disk knows the pattern; not everyone has that much RAM lying spare these days.
The results landed where the builder predicted. In the original build thread, the guess was "somewhere between 2 and 4 tokens per second" after the GPU upgrade. 3.11 sits comfortably inside that band.
What comes next?
More optimization, potentially "customized quantization" — per-tensor or per-layer choices that could trim bandwidth and capacity pressure on the slow tiers, in the vein of techniques used by lawyer Nick Mermiges. The setup "might be faster yet," the builder wrote, while cautioning that raw speed isn't the whole story: output quality has to figure into any future plan.
Tom's Hardware (Source)
More from Rebecca Stone
Show full bio
Correspondent covering technology and product news in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.
1,245 articles