AI & ComputeLOT 261122 · 4 min read

12 Used Nvidia V100s Run All DeepSeek Experts at NVFP4 for $9,450

A $9,450 server of 12 used Nvidia V100s now runs all 15,360 DeepSeek V4.1 Flash experts at NVFP4, decoding at up to 184 tokens/s and beating DGX Spark on single requests.

By Tom Whitfield716 words

Key takeaways

  1. Nick Mermiges expanded his server to 12 used V100s for $9,450 total, running all 15,360 DeepSeek V4.1 Flash experts at NVFP4.
  2. Decode reaches up to 184 tokens/s on code; first token at 256K context fell from 108.4 to 58 seconds.
  3. The 12 cards provide enough memory for the 278.9 GiB of NVFP4 experts that eight cards (254 GiB) could not hold.
  4. Three requests at 128K context total about 360 tokens/s on code, but the setup is not a batched-decode engine.
  5. Batched decoding is planned next — 'the one thing the Sparks have that I want.'
Lawyer’s $9,450 private AI server of 12 used Nvidia V100s now runs every DeepSeek V4.1 Flash expert at 4-bit NVFP4
Lawyer’s $9,450 private AI server of 12 used Nvidia V100s now runs every DeepSeek V4.1 Flash expert at 4-bit NVFP4 — LauraTara / pixabay

A lawyer named Nick Mermiges has expanded his private AI server to 12 used Nvidia V100s for a total spend of $9,450, and the machine now runs every one of DeepSeek V4.1 Flash's 15,360 experts at full 4-bit NVFP4 precision. He reports decode speeds up to 184 tokens per second on code, and claims the system decodes a single request faster than published DGX Spark stacks — at half the price of three of them.

The upgrade adds a third board of four V100s to a setup that previously held eight of the 32GB cards. The numbers explain why. Eight cards provide 254 GiB of memory, but the model's experts in Nvidia's NVFP4 format need 278.9 GiB. His earlier workaround kept every expert at 2-bit precision, then promoted the 5,949 most-used experts — about 38.7% — to NVFP4 until the cards filled.

What changed with the third board?

All 15,360 experts now sit on the cards in NVFP4, with "none compressed further," according to his GitHub repository. In a longer write-up on his site, he states that the expert values "equal the ones in DeepSeek's own release," although other parts of the model use compact forms.

One hardware limitation remains, and he spells it out plainly: the V100, a card from 2018, has no FP4 or FP8 tensor cores. Every NVFP4 number must be unpacked in software into half precision first. The compact experts still pay off "because the cards read fewer bytes, not because they compute faster." That memory-bandwidth argument explains how a Blackwell-era format runs on seven-year-old silicon.

How is the system wired?

The model's 40 layers divide across three boards: layers 0–13, 14–27, and 28–39, with each board's section split across its four cards. Cards on a board link through NVLink, while hand-offs between boards cross the host's PCIe through two separate switches. The first two boards share a PEX88096 switch, so their traffic stays inside it. The third board sits behind a separate PLX 8749, which means the second hand-off must also cross the host's PCIe bridges and adds latency.

Mermiges also retuned each card: the power cap rose from 150 W to 175 W, and the clock cap dropped from 1,522 MHz to 1,452 MHz. The trade favors throughput over peak frequency, and the measurements back it up.

What are the measured gains?

  • First code token at 256K context: 58 seconds, nearly twice as fast as the 108.4 seconds after the previous upgrade.
  • Prompt reading: "about 4,500 tokens a second" past 32K, more than double the roughly 2,200 tokens a second of the first build, though the higher power cap likely accounts for some of that.
  • Code decode: 23% faster at 1K context, but no real gains at 32K and up — the practical win is the shorter wait for the first token.

Not every experiment worked. A day of pushing each card to 200 W ended when "the host tripped once under four long chats at once," suggesting the draw was too much. Four further optimizations — a conversation store that saves to disk, a leaner hand-off between sections, tighter expert packing, and exact four-card-sum arithmetic — missed their targets and had to be turned off.

What about multiple requests?

Running several requests at once usually nets a speed improvement. Here, three requests at 128K context achieved about 360 tokens per second in total on code and 213 on legal text. But he is candid about the ceiling: "this is not a batched-decode engine; more requests than sections gain little."

That may be the next project. He plans to study batched decoding, calling it "the one thing the Sparks have that I want" on Reddit, and to enable the conversation store so a legal matter's documents survive a restart. He concedes the numbers do not establish "answer quality in use," yet the server has been his default DeepSeek machine since Monday, and he calls the model the first one "that works like Opus 4.6" for him. His stated roadmap now points at batching as the last gap between a $9,450 V100 cluster and current-generation hardware.

Tom's Hardware (Source)

Filed under

Share this article:

More from Tom Whitfield

Tom Whitfield

Show full bio

Staff writer covering companies, deals and strategy in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.

1,200 articles

Read also

« Previous article