AI & ComputeLOT 261122 · 3 min read

Tufa Labs squeezes 1GB of Wikipedia into 105.39MB with a lone neural net

tufazip compressed 1GB of Wikipedia to 105.39MB using only a 163M-parameter transformer — no hand-built model — but needs two B200 GPUs and 7.3 days to do it.

By Sophie Lindqvist643 words

Key takeaways

  1. tufazip compressed 1GB of Wikipedia text to 105.39MB, lossless, using only a 163M-parameter Transformer predictor.
  2. Compression took 631,548 seconds (~7.3 days) on two Nvidia B200 GPUs; decompression took ~7.5 days.
  3. It ranks 13th on the Large Text Compression Benchmark as of Nov. 19; all 12 entries above it are hybrids.
  4. The archive ships only a 4.33MB quantized starter model; the full 12-layer model is rebuilt deterministically during unpacking.
  5. A smaller 4K-word tokenizer beat 8K and 16K variants; a 267M-parameter model did slightly worse than a 188M one.
One gigabyte of Wikipedia losslessly compressed to 105.39MB with self-learning AI transformer model
One gigabyte of Wikipedia losslessly compressed to 105.39MB with self-learning AI transformer model — Tho-Ge / pixabay

A 163-million-parameter Transformer has compressed 1GB of Wikipedia text into 105.39MB, losslessly, with no hand-built model in the pipeline. Tufa Labs' tufazip, built by researcher Tommy He, is the smallest result on the Large Text Compression Benchmark from any entry that uses a neural network as its only predictor. The claim comes from a research post, not a shipped product.

The cost of that record is steep. Compression of the 105.39MB version took 631,548 seconds, about 7.3 days, on two Nvidia B200 GPUs paired with two Intel Xeon Platinum 8570 CPUs. Decompression took 645,748 seconds, about 7.5 days. Those figures exclude the offline training of the starter model. Peak GPU memory reached 43.37 GiB across both cards, with total memory use peaking near 68,525 MiB during compression and 73,748 MiB during decompression.

How does the model rebuild itself on unpacking?

The system carries no full trained model in the archive. The 105.39MB package includes the compressed text, the unpacking program and a 4.33MB pre-trained starter model, stored quantized to save space. The full model is a 12-layer Transformer. The starter holds just two trained layers, copied repeatedly to fill all 12 stages once the model loads in memory; each copy then continues learning independently.

Training runs in two phases, according to the research post. The 34-million-parameter starter trains offline before compression begins. The full model then learns online, predicting each token and training on it, while the decompression side repeats every step identically. That is only possible because predictor updates are, as the researchers put it, "fully deterministic." Nothing is lost. The archive also excludes the full trained model, since the receiving machine rebuilds it from text alone.

Where does it rank, and against what?

Tufa Labs says no traditional hand-built compression model assists tufazip, which separates it from all 12 entries currently ranked above it on the benchmark; every one of those mixes a Transformer with hand-built components. The first, online-only version of tufazip ranked second when submitted in early September at 105.98MB. The 105.39MB version sat 13th as of the benchmark's Nov. 19 update. The benchmark covers text only; media files are out of scope.

For comparison, cmix-neif 2.0 reached 89.35MB in 49.2 hours on four cores of an AMD Epyc 7B13. DNA-4 hit 105.55MB on an Nvidia RTX 4090 in 19.8 hours including training. NNCP v3.2 managed 107.26MB in 67.2 hours on an RTX 3090. A dozen hybrid compressors beat tufazip's score. None relies on a neural network alone.

What did the experiments actually show?

Several findings cut against the bigger-is-better instinct. A 4K-word tokenizer beat both 8K and 16K variants. Giving Wikipedia's XML markup dedicated tokens made compression worse. A 267-million-parameter model performed slightly worse than a 188-million-parameter model on this data set. On a smaller 100MB file, gains leveled off around 77 million parameters.

Improvement can also be bought at a poor exchange rate. Widening the training window of already-processed text from 15 to 35 million tokens cut the payload by only 1.21% while demanding 40% more compute. One run started from a larger six-layer offline model predicted better, with a 99.17MB payload, but its heavier weights made the total archive larger than the online-only run.

He frames that six-layer trade-off as the next research question: how much predictive quality a smaller model can retain. Tufa Labs stresses the tool is for research, not a product; nothing here threatens WinRAR today. The open question is whether model-size efficiency can close the 16MB gap to the hybrid leaders before the week-long runtime closes the market for it.

Tom's Hardware (Source)

Filed under

Share this article:

More from Sophie Lindqvist

Sophie Lindqvist

Show full bio

News editor covering money, funding and market shifts around Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.

1,238 articles

Read also

« Previous article