Eight Used Tesla V100 GPUs Deliver 215 Tokens per Second

By Tom Whitfield

Eight used 2018-era Nvidia Tesla V100-SXM2-32GB GPUs deliver up to 215 tokens per second on code, according to measurements the builder published Nov. 5. The lawyer, who posted on Reddit, now uses the two-board system for private first-pass work on legal case files, keeping client material inside the office.

Each four-GPU board cost about $2,400 when the builder bought it. Current eBay listings put used V100 SXM2 32GB cards at about $900, more than double the roughly $400 per card paid in the original post, and the listings suggest prices are still climbing.

What hardware sits inside the two-board server?

Each of the eight cards provides 32GB of HBM2 memory with 900GB/s bandwidth. SXM2 serves as Nvidia's mezzanine socket for data-center GPUs rather than a conventional PCIe slot, and two four-card baseboards divide the system.

Within each board, NVLink provides direct GPU-to-GPU links and pools 128GB of memory. The two boards communicate only through PCIe, a slower route than the NVLink links inside each board.

Each board reaches the host through two PEX, or PLX, switch cards connected with SlimSAS cables. Four switches handle two GPUs apiece at PCIe 3.0 x8 speeds, with each switch offering its own x16 uplink to the host.

The host uses a Gigabyte Threadripper Pro motherboard, a 12-core 3945WX CPU, eight 32GB DDR4 RDIMMs and one NUMA node. Two PSUs provide 2,800W in total, and the builder places the server on two ordinary office circuits. A 240V circuit may replace that arrangement later.

The machine gains a large pool of fast memory by accepting a slower link between its two boards.

How does DeepSeek fit across eight GPUs?

The builder compresses DeepSeek V4.1 Flash mostly to 2 bits here, and its weights still total 248.8GB. The model's experts, the parts that do most of the work for each word, fit entirely in GPU memory.

About 110GB of Engram lookup tables remain in system RAM; each fetch takes roughly 0.12 milliseconds per step, which the builder says barely affects speed.

Layers 0 through 19 run on the first board, while layers 20 through 39 run on the second. Each half stays behind its own NVLink fabric, so only one handoff per step crosses PCIe. An earlier arrangement split every layer across all eight cards and made speeds collapse because of that PCIe hop.

In plain terms, each board runs half of the model, so the system makes one slow transfer per step.

What speed does the builder measure?

• 1K tokens: 1.2-second first-token wait; 215 tokens/s for code; 118 tokens/s for legal writing.
• 32K tokens: 17.2-second wait; 200 tokens/s for code; 117 tokens/s for legal writing.
• 128K tokens: 59.2-second wait; 183 tokens/s for code; 111 tokens/s for legal writing.
• 256K tokens: 118.7-second wait; 160 tokens/s for code; 103 tokens/s for legal writing.

The builder recorded prompt ingestion at about 2,200 tokens per second. Even so, a 128K-token prompt waits 59.2 seconds for its first token, while the system accepts contexts up to 524K tokens.

Three days before the post, the same model on four cards managed only 36–38 tokens per second. The builder said the second board could roughly double performance under ideal conditions, then credited custom kernels and prefill work for the rest of the jump.

Code reaches its top speed because a small draft model built into DeepSeek guesses five tokens ahead, and code is easy to predict. Legal writing peaks at 118 tokens per second. These figures come from the builder's Nov. 5 tests, which used greedy decoding, generated 1,024 tokens and averaged two runs.

The server generates quickly, but very long documents still take time before the answer begins.

Why did the server reset under load?

With uncapped GPU clocks and a higher power limit, the host reset twice with a fatal PCIe error on the link to one GPU board. The cards drew 192–232W, above the 180W limit. The builder's best guess, although unproven, was that a board stops responding during load spikes.

The resets stopped after the builder capped the clocks and removed the board with the burn-marked power connector. The practical lesson was to check power connections.

Each card now runs at a 150W cap, half its 300W rating, and the graphics clock is capped at 1522MHz. Tests showed that a 200W cap brought no appreciable performance gains.

System memory also produced uncorrected ECC errors until the builder reduced the RAM from DDR4-3200 to DDR4-2666. One GPU was connected at x4 instead of x8; the builder plans to reseat it, and a commenter traced a similar problem on another V100 to a bad cable.

Stable power settings and sound connections mattered more than additional processing power.

What changes for the lawyer's daily work?

In daily use, the server reads a folder of records, extracts facts, dates and numbers, summarizes the material and attempts routine drafts. It supports retrieval-augmented generation databases and tool calls, controlled through a Discord-style interface over SSH from a Windows PC. The system currently handles one request at a time.

The lawyer said: “Nothing goes out the door without a real review.” That rule supports cases where client files cannot be sent to cloud or frontier models. The builder estimates that 50% to 60% of existing LLM workflows can now run on the local box instead of frontier models.

A second DeepSeek server on two DGX Spark systems hands work back and forth with the V100 box. The builder says the setup appears to save money compared with API pricing, although it is not faster. The stated objective is simple: “once it is self contained, which is the goal.”

Support also has a deadline because Nvidia's R580 driver branch is the last to support Volta. The lawyer describes the project as mostly a hobby, with a possible business later. The builder has tested concurrent requests on another system but has not deployed that design here.

If the machine handles two or three requests successfully, it could become a small private legal-AI service, though rising used-card prices will test that business case.