Embedding a large corpus is usually the slowest part of building a vector search pipeline: transformer-based models capture semantics very well, but they are heavy on CPU, so you either wait days for the embeddings or pay for a costly GPU.
Static embedding models avoid that cost by skipping the neural network altogether. A static model is a lookup table that associates each token in the model’s vocabulary with an embedding: to embed a text, you tokenize it, look up the vector of each token, and average them. With potion-base-8M, this let us embed 32.9 million passages on a laptop CPU in 35 minutes, where a transformer-based model would have needed about 53 days. The tradeoff is retrieval quality, because without layers the tokens never influence each other.
The potion models series and the model2vec library behind it come from Minish, and we built on their work to add static embedding support to FastEmbed. In this article, we cover how static embeddings work, how fast we can make them run, and which tradeoffs come with them.
So: What Is a Static Embedding Model?
A static embedding model gives every token in its vocabulary one fixed vector. To embed a text, you split it into tokens, look up the vector of each token, and average those vectors (mean pooling) into a single embedding. Nothing else happens between the text and the vector: there are no layers in which tokens influence each other, which is what makes a static model so cheap to run, and also why it is less capable than a transformer.
Static model weights are saved like those of any other model, often in the safetensors format, but they hold the token embeddings rather than the parameters of a network. To see how this works, start with datatypes: like every other model, a static embedding model stores its weights with a given datatype (f32, f16, bf16, all the way down to u8), which determines how many bits (the precision) represent each number. In our example, we use the mainstream choice: f32, a full-precision floating point number that takes 32 bits (4 bytes). When we read the model tensor from the safetensors file, we decode each weight by taking 4 contiguous bytes and reading them as an f32 (in numpy, you would use np.frombuffer with dtype="<f4").
Decoding a single weight is not enough, though: embeddings usually have hundreds of dimensions, so we need to read as many weights as the model’s output dimension (let’s assume 256). At this point we know how to read weights into a vector, but we still need to associate vectors with tokens and produce a final 256-dimensional vector for the whole text.
To do that, we use a tokenizer, which transforms text into token IDs; we then use each ID to locate the embedding of that token within the model bytes. The position of the embedding is DATATYPE_BYTES x DIMENSIONS x TOKEN_ID, so a token with ID 73 starts at byte 4 x 256 x 73 = 74,752 of the tensor. Given a token ID, we can therefore decode its embedding by reading DIMENSIONS x DATATYPE_BYTES bytes into f32 values.
A text can contain hundreds of tokens: if each of them gets its own embedding, how do we get to a single vector? You guessed it: mean pooling, which means averaging the token vectors position by position. We can then apply an optional L2 normalization step, which scales the vector to unit length, and leads to better performance in practice. This is all it takes to produce the final embedding.
Making Static Models Go Brr
As the previous section suggests, static embedding models are much faster than transformer-based ones: locating a token, decoding its weights into an f32 vector, and mean pooling the results are all cheap operations, to the point where tokenization itself is often the bottleneck.
In our previous FastEmbed implementation, we ran static models through ONNX Runtime, the same way we run transformer-based models. ONNX is an excellent runtime for neural networks, but the graph optimizations it applies under the hood add substantial overhead to an inference step that only performs a table lookup.
Inspired by model2vec and by some experiments with Rust, we decided to port the inference to pure numpy operations. A few optimizations are worth calling out:
- Custom tensor reading, with no
safetensorsdependency: we read the header size from the first 8 bytes of the file, parse the header as JSON to find the region that holds the weights and their datatype, and then load that region, optionally throughmmap, all withnumpy.
- No padding: ONNX models generally pad every batch to its longest text, while here we keep the token IDs in one flat array plus a lengths array, built from numpy views over the tokenizer output, which avoids that extra work.
- Pooling as a single matrix product: we build a
documents x distinct tokenscount matrix and multiply it with the embedding rows of those tokens (counts @ embeddings[distinct_rows]), which means one matrix multiplication per batch instead of a loop over documents; each distinct token row is fetched once per batch, and the temporary memory is capped to avoid out-of-memory errors.
But the main speedup did not come from numpy itself. To see where the time goes, we profiled the three stages of the static path (tokenization, lookup, and pooling) on 2,048 documents of about 150 tokens each. Tokenization dominated, taking about 61% of the wall time for potion-base-8M and 54% for potion-retrieval-32M: as we said above, static embedding inference is so cheap that tokenization becomes the bottleneck, so the fix was to speed up the tokenizer.
In an experimental branch, we migrated from tokenizers v0.x to the v1 pre-release, which Hugging Face’s own benchmarks show to be substantially faster. With v1, tokenization drops to about 14% and 12% of the wall time, the total time per batch roughly halves, and pooling becomes the largest stage. End to end on FiQA with potion-base-8M, throughput more than doubled, from 18K to 40K docs/s, and the p50 latency of embedding a single query fell by more than half, from 39 µs to 14 µs. The PR has the full details.
The FiQA throughput numbers are nice, but FiQA is a relatively small dataset (57K documents), and we promised at the beginning that static embeddings could help with massive datasets, so we tested that too.
32.9M Embeddings in 35 Minutes
We chose the English corpus of MIRACL, which contains 32,893,221 passages. To give a sense of how long a transformer-based model would need, we tested stella_en_400M_v5 through its ONNX export in FastEmbed: it delivered 7 docs/s on a MacBook Pro with an M4 Max, 14 cores, and 48 GiB of unified memory, so it would have needed about 53 days to embed the entire dataset.
To load the entire dataset into Qdrant with static embeddings on the same hardware, we needed 9 to 11 hours in total, but the share of that time spent on embedding was remarkably low: 35 minutes for potion-base-8M, 38 for potion-retrieval-32M, and 47 for potion-multilingual-128M, which corresponds to roughly 11K to 15K docs/s. The rest was mostly network transfer and ingestion: the data went to a Qdrant Cloud cluster over the network, so the share spent on transfer and ingestion would likely be smaller on a local instance.
These numbers are fantastic but, before you add a static embedding model to your search pipeline, stay with us for one more section: as always in computer science, there are tradeoffs.
Tradeoffs, and Maybe a Solution?
The recall@10 numbers for static embedding models are not particularly high:
Two things stand out when choosing a model. The first is that size does not decide the ranking: potion-retrieval-32M, which was tuned for retrieval, is the strongest dense model, while potion-multilingual-128M, which is the largest of the three, is built to cover many languages, and on this English corpus it scores below even potion-base-8M. How well a model’s training fits your language and task, in this case, matters more than its parameter count.
The second is that hybrid search is worth testing. In our hybrid setup, Qdrant retrieves the top 50 results with the static model’s dense vectors and the top 50 with a BM25 sparse vector for the same query, then fuses the two lists with reciprocal rank fusion (RRF) and returns the top 10. BM25 roughly doubled recall for potion-base-8M and added a few points to potion-retrieval-32M, so the weaker the dense model, the more it helped.
For reference, the published MTEB results for transformer-based dense models on the English portion of MIRACL sit between 53.7% (stella_en_400M_v5) and 63.5% (snowflake-arctic-embed-m). The tradeoff is concrete: the best static model gives up roughly 10 to 24 points of recall@10, depending on the transformer and on whether BM25 is added, in exchange for embedding speed. If recall is more important for your pipeline, a transformer is the safer choice, while, if you’re working with constrained resources, a static model with hybrid search makes a cheap first-stage retriever.
While static embedding models will likely improve, we have been working on an interesting alternative at Qdrant: constella, a family of models derived from the stella_en_400M_v5 encoder mentioned above, which needs no neural network at query time. A larger model like stella creates the document embeddings, and a much smaller one, constella-zero, produces query embeddings from a lookup table, which is essentially the same mechanism as static embeddings. This does not solve the problem of re-embedding massive datasets, but it significantly speeds up query serving, so you can think of it as a middle ground. You can read more about constella here.
Should You Go Static?
Static embedding models turn the embedding step from the slowest part of an indexing pipeline into a rounding error: we embedded 32.9 million passages in 35 to 47 minutes on a laptop CPU, against the roughly 53 days a transformer-based model would have needed, and most of the time that remained went into moving vectors over the network. That speed comes at a price in retrieval quality, so they are not a drop-in replacement for a good transformer-based model; they shine when embedding volume or hardware is the real constraint.
They are a good fit if you have to embed tens of millions of documents, if you re-embed often (to compare models, for example), or if you run on CPU-only machines, ideally as a cheap first-stage retriever paired with hybrid search. If quality comes first, a transformer on the document side is still the safer choice, and constella-zero can keep your query path cheap while you pay the embedding cost once, offline.
If you want to try this on your own data, static embedding models are available in FastEmbed: start with potion-base-8M for the fastest run, move to potion-retrieval-32M if retrieval quality matters more, and add BM25 for hybrid search. All the code and raw results behind the numbers in this article are in the miracl-static-embeddings repository, and we would love to hear how static embeddings work (or don’t) for your use case.