UltraBERT Fits 32K Tokens in One Vector. RIP Chunking?

Every retrieval pipeline starts with the same call: how big is a chunk? Too small, and a chunk loses its context. Too big, and the embedding model cuts it off.

DMIS Lab’s UltraBERT-Embed-150M moves that cutoff to 32K tokens. It’s a 151M-parameter dense embedding model built on UltraBERT, an English encoder trained for 32K-token inputs. Most documents fit in one pass, and you get back one 768-dimension vector.

So is chunking dead? We tested it in Qdrant: one vector per document against 512-token chunks, the two together, and both compressed.

How We Tested

We used five tasks from LongEmbed, a long-document benchmark. Its documents average 7,600 to 13,800 tokens. Each task asks a different kind of question:

• SummScreenFD: the query summarizes a TV episode, and the right document is its transcript.
• 2WikiMultihopQA: multi-hop questions over Wikipedia passages.
• QMSum: the query summarizes one part of a meeting transcript.
• Passkey: each document hides a pass key in filler text, and the query asks for that key.
• Needle: each document is an essay with one unrelated sentence added, and the query asks about that sentence.

To test at scale, we also used MLDR: 200,000 English Wikipedia articles and 800 questions, each about one article.

We embedded every document in two ways:

• Whole document: one vector for the full text, up to 32K tokens.
• 512-token chunks: one vector per chunk. A document scores by its best chunk.

The full results are at the end.

Chunks Still Find the Details

The whole-document vector really reads all 32K tokens. On Passkey, it ranked the right document first 99% of the time across all lengths, and every time at 32K tokens. But on most other tasks, chunks scored higher.

Chunks won when the answer is in a few passages: on 2WikiMultihopQA, QMSum, and MLDR. The whole-document vector won only on SummScreenFD, where the query describes the whole episode.

Needle shows why chunks win. One vector can’t hold every detail of a long document. At 256 tokens, the whole-document vector ranked the right essay first 84% of the time. At 32K tokens, it did so only 30% of the time. At the same length, 512-token chunks got 56%.

We also tried using both. In Qdrant, one point can hold two named vectors: the whole-document vector, and the chunk vectors as a multivector. A hybrid query searches both and fuses the results. This helped a little on SummScreenFD and Needle, by up to 0.02 and 0.03 over chunks alone. On 2WikiMultihopQA, QMSum, and MLDR, it did worse than chunks alone, most of all on MLDR.

Chunk Vectors for 200,000 Articles in About 140 MB

Chunks mean more vectors. MLDR’s articles split into 7 chunks each on average, 1.43 million vectors in total. In float32, that’s about 4 GB.

1-bit TurboQuant shrinks each vector from 3,072 bytes to 104. That’s 30x smaller, and about 140 MB for all 200,000 articles. We stored each chunk as its own Qdrant point, and a grouped search returns each article once.

Against exact search, the compressed vectors kept 95% of the quality, and 98% with rescoring. The uncompressed vectors in the same index kept 99%. Rescoring has a cost. In our setup, the original vectors were on disk. Reading them added 4 to 11 ms per query, or over 100 ms while the disk cache was cold.

A whole-document vector compresses less well than chunk vectors. On MLDR, it kept 89% of its quality at 1 bit. At 2 bits, it scored the same as the uncompressed vector in the same index, so start it there, or keep rescoring on.

So Is Chunking Dead?

Not just yet. In these tests, chunks still found details better. We’d keep them as the starting point for detail search, even when the whole document fits.

UltraBERT gives us room to try retrieval at different scales. We could search whole documents to find the right topic, then search their chunks for the answer. That’s worth testing, especially since combining both searches didn’t consistently help here.

Whole-document vectors still have a cost. Embedding SummScreenFD’s transcripts whole took about 3x as long as embedding 512-token chunks on our NVIDIA L4.

For detail search, we’d start with 512-token chunks and test 256 for single facts. Compression makes the extra vectors more affordable, though quality and rescoring latency still need testing on your own queries.

The 32K-token window gives us more freedom over what each vector represents. The next question is how to use that freedom to find better answers.

Full Results

Thanks to DMIS Lab for UltraBERT-Embed, and for giving us a good reason to revisit chunking.