Eight-V100 Legal AI Server Moves DeepSeek to Near-NVFP4 Precision

A lawyer in Columbia, South Carolina has moved roughly 80% of the work on his private AI server from 2-bit to 4-bit NVFP4 precision, at the cost of a 3% to 30% slowdown in code output. The rig runs DeepSeek V4.1 Flash on eight used Nvidia V100 GPUs, each with 32GB of memory, for his trial practice. The builder is Nick Mermiges, founder of PLI Labs and proprietarylegal.ai, identified through his own comments on the earlier coverage. His other firm, PLI, is a technology consultancy that now lists private, on-prem LLM serving as a service.

What did the upgrade change?

The build, renamed from "Volta sPLIt" to "Volta Max uPLIft," keeps every expert resident at 2-bit and promotes the most-used ones to NVFP4 until the cards fill. That came out to 38.7%, or 5,949 experts, at exact NVFP4 precision. The model has 384 routed experts across 40 layers, or 15,360 in total, which at official precision weighs about 269 GiB by his estimate — more than the 256 GiB the eight V100s hold before anything else loads. The old 2-bit build held weights of 248.8GB; the new Hugging Face package is about 556GB.

"Most-used" comes from a count over four equally weighted text types, one of which is his own legal harness traffic. In single-request tests, 80% to 86% of decode routes land on the NVFP4 experts — lowest for general text, highest for source code. For his two-request setup, he predicts 78% to 85%. The remainder still hits 2-bit experts.

How close is it to the reference?

Mermiges measured against Nvidia's own NVFP4 build of the same model, which the project README says matches the official weights at 4.25 bits per weight. Same next token rose to 95.1% from 92.2%. KL divergence fell to 0.079 nats from 0.196. Across 513 questions from a standard multiple-choice knowledge test, changed answers dropped from 79 to 8.

This measures fidelity, not accuracy, and accuracy was about the same across all three cases. The tests ran on the first release of the build, not on the two-request setup he now runs — a gap he does not claim to have closed.

What did it cost in speed?

Code output is 3% to 30% slower depending on prompt length; legal writing is 9% to 23% slower. At 1K-token prompts, code fell from 215.0 to 150.1 tokens per second. At 256K tokens, code went from 160.0 to 154.6. Legal writing at 1K dropped from 117.6 to 105.5 tokens per second.

Concurrency partly offsets the loss. Two requests at once raise total output to 159.0 tokens per second, against 118.9 for a single request. First tokens also arrive sooner at every prompt length: 108.4 seconds versus 118.7 seconds at 256K tokens. But each slot keeps its own state on the cards, leaving room for fewer NVFP4 experts — 6,215, 5,949, and 5,663 for one, two, and three slots. Each added slot may push one to two points of calls back to 2-bit.

What else changed in the hardware?

The system grew from 256GB to 1TB of RAM this week, using eight 128GB modules at 2,933 MT/s; the old 32GB modules had to run at DDR4-2666 due to ECC errors. Two boots hung at the first transfer between boards, so he reordered the cards in software. The GPUs remain capped at 1,522 MHz. The crashes first blamed on PCIe were, he concluded, "probably one bad PSU"; after the power fix, nine of nine long chats read clean. He also added an RTX 3090, though moving DeepSeek's draft model onto it has not happened yet.

Mermiges reaffirmed that "the documents never leave the office." He built the software side with AI: Claude Code wrote the kernels, and the Opus 5.5 and Fable 5.2 models handled optimizations and re-quantizing. The V100s remain on "borrowed time" — R580 is the last Nvidia driver branch to support them. Two DGX Sparks already run the same model at 1M context for agent work, and a second server with eight more V100s and 512GB of 3,200 MT/s ECC RAM waits in the wings, which suggests the fidelity-versus-throughput trade may get another round of tuning before the Volta cards age out.