BM25 with Qdrant Edge

BM25 (Best Matching 25) is a popular ranking algorithm for keyword search. Qdrant Edge computes BM25 scores from a full-text index on a payload field, so you can run keyword search on-device without an embedding model or an internet connection. The index holds everything BM25 needs, including the length of each document, so Qdrant Edge derives all statistics from your data.

Qdrant Edge also includes a BM25 embedder that generates sparse vectors, compatible with sparse vector BM25 on Qdrant Server. In Python, use the Bm25 class. In Rust, use EdgeBm25 from the qdrant_edge::bm25_embed module. Refer to Sparse Vector Search.

Create a Full-Text Index with Scoring

An Edge Shard requires at least one vector. This example configures a small dense vector, which you can also use for hybrid search:

from qdrant_edge import Distance, EdgeConfig, EdgeShard, EdgeVectorParams

config = EdgeConfig(
    vectors={"embedding": EdgeVectorParams(size=4, distance=Distance.Cosine)},
)

shard = EdgeShard.create(SHARD_DIRECTORY, config)

const SHARD_DIRECTORY: &str = "./qdrant-edge-bm25";

use std::path::*;
use qdrant_edge::*;

let config = EdgeConfigBuilder::new()
    .vector("embedding", EdgeVectorParamsBuilder::new(4, Distance::Cosine).build())
    .build();

let shard = EdgeShard::new(Path::new(SHARD_DIRECTORY), config)?;

To rank points with BM25, create a full-text index on the field you want to search and enable scoring. For example, to search the titles of books:

from qdrant_edge import (
    Language,
    SnowballLanguage,
    SnowballParams,
    TextIndexParams,
    UpdateOperation,
)

shard.update(UpdateOperation.create_field_index(
    "title",
    TextIndexParams(
        scoring=True,
        stemmer=SnowballParams(language=SnowballLanguage.English),
        stopwords=Language.English,
    ),
))

use qdrant_edge::*;

shard.update(UpdateOperation::FieldIndexOperation(
    FieldIndexOperations::CreateIndex(CreateIndex {
        field_name: "title".try_into().unwrap(),
        field_schema: Some(PayloadFieldSchema::FieldParams(PayloadSchemaParams::Text(
            TextIndexParamsBuilder::new()
                .scoring(TextScoringParams::default())
                .stemmer(StemmingAlgorithm::Snowball(SnowballParams {
                    r#type: Snowball::Snowball,
                    language: SnowballLanguage::English,
                }))
                .stopwords(StopwordsInterface::Language(Language::English))
                .build(),
        ))),
    }),
))?;

This index also applies English stemming and stopword removal, which improves BM25 results for English text. Refer to Text Processing.

Add Points

Store the text in the indexed payload field:

from qdrant_edge import Point, UpdateOperation

shard.update(UpdateOperation.upsert_points([
    Point(1, {"embedding": [0.05, 0.61, 0.76, 0.74]}, {"title": "The Time Machine"}),
    Point(2, {"embedding": [0.19, 0.81, 0.75, 0.11]}, {"title": "Time Travel in Einstein's Universe"}),
    Point(3, {"embedding": [0.36, 0.55, 0.47, 0.94]}, {"title": "The Travels of Marco Polo"}),
]))

use qdrant_edge::external::serde_json::json;
use qdrant_edge::*;

let books: [(u64, [f32; 4], &str); 3] = [
    (1, [0.05, 0.61, 0.76, 0.74], "The Time Machine"),
    (2, [0.19, 0.81, 0.75, 0.11], "Time Travel in Einstein's Universe"),
    (3, [0.36, 0.55, 0.47, 0.94], "The Travels of Marco Polo"),
];

let points = books
    .iter()
    .map(|(id, embedding, title)| {
        PointStruct::new(
            *id,
            Vectors::new_named([("embedding", embedding.to_vec())]),
            json!({ "title": title }),
        )
        .into()
    })
    .collect();

shard.update(UpdateOperation::PointOperation(
    PointOperations::UpsertPoints(PointInsertOperations::PointsList(points)),
))?;

Query a Text Field

To run a BM25 query, use a TextQuery with the name of the indexed payload field and the query text. The following example searches for books with “time travel” in the title:

from qdrant_edge import QueryRequest, TextQuery

results = shard.query(QueryRequest(
    query=TextQuery("title", "time travel"),
    limit=3,
    with_payload=True,
))

use qdrant_edge::*;

let results = shard.query(
    QueryRequestBuilder::new(3)
        .query(TextQueryBuilder::new("title".try_into().unwrap(), "time travel").build())
        .with_payload(WithPayloadInterface::Bool(true))
        .build(),
)?;

A point matches the query if its text contains at least one of the query terms. Points that contain none of the terms are not returned. To require or exclude specific terms, add a full-text filter to the query.

A TextQuery can also be used in a prefetch, for example, to combine BM25 with dense vector search in hybrid search.

Configure BM25 Parameters

The BM25 ranking function has two parameters that you can set per query:

k. Controls term frequency saturation. Higher values increase the influence of term frequency. Must be 0 or greater. Defaults to 1.2.
b. Controls document length normalization. Ranges from 0 (no normalization) to 1 (full normalization). A higher value means longer documents have less impact. Defaults to 0.75.

Because these parameters are set per query, you can change them without rebuilding the index:

from qdrant_edge import Bm25Params, QueryRequest, TextQuery

results = shard.query(QueryRequest(
    query=TextQuery("title", "time travel", scoring=Bm25Params(k=1.5, b=0.5)),
    limit=3,
    with_payload=True,
))

use qdrant_edge::*;

let results = shard.query(
    QueryRequestBuilder::new(3)
        .query(
            TextQueryBuilder::new("title".try_into().unwrap(), "time travel")
                .scoring(TextQueryScoring::Bm25(
                    Bm25ParamsBuilder::new().k(1.5).b(0.5).build(),
                ))
                .build(),
        )
        .with_payload(WithPayloadInterface::Bool(true))
        .build(),
)?;

Was this page useful?

On this page:

Ready to try Qdrant?