Full-Text Search

Full-text search is similar to full-text filtering, with the key difference being that full-text queries are used for ranking. For each document that matches the search terms, Qdrant calculates a relevance score based on how well the document matches the search terms. That score is used to rank the results. To rank text with learned sparse models like SPLADE++ and miniCOIL instead, see Sparse Vector Search.

BM25

BM25 (Best Matching 25) is a popular ranking algorithm that takes a probabilistic approach to score calculation. For each search term, BM25 considers several statistics about the term and the document to calculate a relevance score:

Term frequency (TF): the more often a term appears in a document, the more relevant that document is likely to be.
Inverse document frequency (IDF): the rarer a term is across all documents, the higher the weight of that term.
Document length: a term appearing in a shorter document is more relevant than the same term appearing in a longer document.

Qdrant computes BM25 scores from a full-text index on a payload field. The index holds everything BM25 needs, including the length of each document, so Qdrant derives all statistics from your data.

Create a Full-Text Index with Scoring

To rank points with BM25, create a full-text index on the field you want to search and set scoring to true. For example, to search the titles of books:

PUT /collections/books/index?wait=true
{
  "field_name": "title",
  "field_schema": {
    "type": "text",
    "scoring": true
  }
}

from qdrant_client import QdrantClient, models

client.create_payload_index(
    collection_name="books",
    field_name="title",
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        scoring=True,
    ),
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.createPayloadIndex("books", {
  field_name: "title",
  field_schema: {
    type: "text",
    scoring: true,
  },
});

use qdrant_client::qdrant::{
    CreateFieldIndexCollectionBuilder, FieldType, TextIndexParamsBuilder, TokenizerType,
};
use qdrant_client::Qdrant;

client
    .create_field_index(
        CreateFieldIndexCollectionBuilder::new("books", "title", FieldType::Text)
            .field_index_params(
                TextIndexParamsBuilder::new(TokenizerType::Word)
                    .bm25_scoring()
                    .build(),
            ),
    )
    .await?;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Collections.PayloadIndexParams;
import io.qdrant.client.grpc.Collections.PayloadSchemaType;
import io.qdrant.client.grpc.Collections.TextIndexParams;
import io.qdrant.client.grpc.Collections.TextScoringParams;
import io.qdrant.client.grpc.Collections.TextScoringType;
import io.qdrant.client.grpc.Collections.TokenizerType;

QdrantClient client =

client
    .createPayloadIndexAsync(
        "books",
        "title",
        PayloadSchemaType.Text,
        PayloadIndexParams.newBuilder()
            .setTextIndexParams(
                TextIndexParams.newBuilder()
                    .setTokenizer(TokenizerType.Word)
                    .setScoring(
                        TextScoringParams.newBuilder().setType(TextScoringType.Bm25).build())
                    .build())
            .build(),
        null,
        null,
        null)
    .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.CreatePayloadIndexAsync(
    collectionName: "books",
    fieldName: "title",
    schemaType: PayloadSchemaType.Text,
    indexParams: new PayloadIndexParams
    {
        TextIndexParams = new TextIndexParams
        {
            Tokenizer = TokenizerType.Word,
            Scoring = new TextScoringParams { Type = TextScoringType.Bm25 }
        }
    }
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.CreateFieldIndex(context.Background(), &qdrant.CreateFieldIndexCollection{
	CollectionName: "books",
	FieldName:      "title",
	FieldType:      qdrant.FieldType_FieldTypeText.Enum(),
	FieldIndexParams: qdrant.NewPayloadIndexParamsText(
		&qdrant.TextIndexParams{
			Tokenizer: qdrant.TokenizerType_Word,
			Scoring: &qdrant.TextScoringParams{
				Type: qdrant.TextScoringType_Bm25,
			},
		}),
})

If the collection already contains data, Qdrant builds the index from the text stored in the payload, so you don’t need to upload your data again.

Query a Text Field

To run a BM25 query, use a text query and set using to the name of the indexed payload field. The following example searches for books with “time travel” in the title:

POST /collections/books/points/query
{
  "query": {
    "text": "time travel"
  },
  "using": "title",
  "limit": 10,
  "with_payload": true
}

from qdrant_client import QdrantClient, models

client.query_points(
    collection_name="books",
    query=models.TextQuery(text="time travel"),
    using="title",
    limit=10,
    with_payload=True,
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.query("books", {
  query: {
    text: "time travel",
  },
  using: "title",
  limit: 10,
  with_payload: true,
});

use qdrant_client::Qdrant;
use qdrant_client::qdrant::{Query, QueryPointsBuilder};

client
    .query(
        QueryPointsBuilder::new("books")
            .query(Query::new_text("time travel"))
            .using("title")
            .limit(10)
            .with_payload(true)
            .build(),
    )
    .await?;

import static io.qdrant.client.WithPayloadSelectorFactory.enable;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Points.Query;
import io.qdrant.client.grpc.Points.QueryPoints;
import io.qdrant.client.grpc.Points.TextQuery;

QdrantClient client =

client
    .queryAsync(
        QueryPoints.newBuilder()
            .setCollectionName("books")
            .setQuery(
                Query.newBuilder()
                    .setText(TextQuery.newBuilder().setQuery("time travel").build())
                    .build())
            .setUsing("title")
            .setLimit(10)
            .setWithPayload(enable(true))
            .build())
    .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.QueryAsync(
    collectionName: "books",
    query: new Query { Text = new TextQuery { Query = "time travel" } },
    usingVector: "title",
    payloadSelector: true,
    limit: 10
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.Query(context.Background(), &qdrant.QueryPoints{
	CollectionName: "books",
	Query: qdrant.NewQueryText(&qdrant.TextQuery{Query: "time travel"}),
	Using:       qdrant.PtrOf("title"),
	WithPayload: qdrant.NewWithPayload(true),
	Limit:       qdrant.PtrOf(uint64(10)),
})

A point matches the query if its text contains at least one of the query terms. Points that contain none of the terms are not returned. To require or exclude specific terms, add a full-text filter to the query. Query terms that appear more than once count only once.

A text query can also be used as a prefetch, for example, to combine BM25 with dense vector search in hybrid search.

Configuring BM25 Parameters

The BM25 ranking function has two parameters that you can set per query:

k. Controls term frequency saturation. Higher values increase the influence of term frequency. Must be 0 or greater. Defaults to 1.2.
b. Controls document length normalization. Ranges from 0 (no normalization) to 1 (full normalization). A higher value means longer documents have less impact. Defaults to 0.75.

Because these parameters are set per query, you can change them without rebuilding the index:

POST /collections/books/points/query
{
  "query": {
    "text": {
      "query": "time travel",
      "scoring": {
        "bm25": {
          "k": 1.5,
          "b": 0.5
        }
      }
    }
  },
  "using": "title",
  "limit": 10,
  "with_payload": true
}

from qdrant_client import QdrantClient, models

client.query_points(
    collection_name="books",
    query=models.TextQuery(
        text=models.TextQueryInput(
            query="time travel",
            scoring=models.Bm25Scoring(
                bm25=models.Bm25Params(k=1.5, b=0.5),
            ),
        ),
    ),
    using="title",
    limit=10,
    with_payload=True,
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.query("books", {
  query: {
    text: {
      query: "time travel",
      scoring: {
        bm25: {
          k: 1.5,
          b: 0.5,
        },
      },
    },
  },
  using: "title",
  limit: 10,
  with_payload: true,
});

use qdrant_client::Qdrant;
use qdrant_client::qdrant::{Query, QueryPointsBuilder, TextQueryBuilder};

client
    .query(
        QueryPointsBuilder::new("books")
            .query(Query::new_text(
                TextQueryBuilder::new("time travel").k(1.5).b(0.5),
            ))
            .using("title")
            .limit(10)
            .with_payload(true)
            .build(),
    )
    .await?;

import static io.qdrant.client.WithPayloadSelectorFactory.enable;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Points.Bm25Params;
import io.qdrant.client.grpc.Points.Query;
import io.qdrant.client.grpc.Points.QueryPoints;
import io.qdrant.client.grpc.Points.TextQuery;

QdrantClient client =

client
    .queryAsync(
        QueryPoints.newBuilder()
            .setCollectionName("books")
            .setQuery(
                Query.newBuilder()
                    .setText(
                        TextQuery.newBuilder()
                            .setQuery("time travel")
                            .setBm25(Bm25Params.newBuilder().setK(1.5f).setB(0.5f).build())
                            .build())
                    .build())
            .setUsing("title")
            .setLimit(10)
            .setWithPayload(enable(true))
            .build())
    .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.QueryAsync(
    collectionName: "books",
    query: new Query
    {
        Text = new TextQuery
        {
            Query = "time travel",
            Bm25 = new Bm25Params { K = 1.5f, B = 0.5f }
        }
    },
    usingVector: "title",
    payloadSelector: true,
    limit: 10
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.Query(context.Background(), &qdrant.QueryPoints{
	CollectionName: "books",
	Query: qdrant.NewQueryText(qdrant.NewTextQueryBM25("time travel", &qdrant.Bm25Params{
		K: qdrant.PtrOf(float32(1.5)),
		B: qdrant.PtrOf(float32(0.5)),
	})),
	Using:       qdrant.PtrOf("title"),
	WithPayload: qdrant.NewWithPayload(true),
	Limit:       qdrant.PtrOf(uint64(10)),
})

The term frequency parameter is named k, not k1.

Text Processing

Before scoring, Qdrant processes both the indexed text and the query text with the text processing settings of the full-text index, such as the tokenizer, lowercasing, ASCII folding, stemming, and stopword removal. This ensures that query terms match the indexed terms.

By default, a full-text index uses the word tokenizer and lowercasing, without stemming or stopword removal. For better BM25 results, configure stemming and stopwords for the language of your data. For example, to index Spanish book titles:

PUT /collections/books/index?wait=true
{
  "field_name": "title",
  "field_schema": {
    "type": "text",
    "stemmer": {
      "type": "snowball",
      "language": "spanish"
    },
    "stopwords": "spanish",
    "scoring": true
  }
}

from qdrant_client import QdrantClient, models

client.create_payload_index(
    collection_name="books",
    field_name="title",
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        stemmer=models.SnowballParams(
            type=models.Snowball.SNOWBALL,
            language=models.SnowballLanguage.SPANISH,
        ),
        stopwords=models.Language.SPANISH,
        scoring=True,
    ),
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.createPayloadIndex("books", {
  field_name: "title",
  field_schema: {
    type: "text",
    stemmer: {
      type: "snowball",
      language: "spanish",
    },
    stopwords: "spanish",
    scoring: true,
  },
});

use qdrant_client::qdrant::{
    CreateFieldIndexCollectionBuilder, FieldType, TextIndexParamsBuilder, TokenizerType,
};
use qdrant_client::Qdrant;

client
    .create_field_index(
        CreateFieldIndexCollectionBuilder::new("books", "title", FieldType::Text)
            .field_index_params(
                TextIndexParamsBuilder::new(TokenizerType::Word)
                    .snowball_stemmer("spanish".to_string())
                    .stopwords_language("spanish".to_string())
                    .bm25_scoring()
                    .build(),
            ),
    )
    .await?;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Collections.PayloadIndexParams;
import io.qdrant.client.grpc.Collections.PayloadSchemaType;
import io.qdrant.client.grpc.Collections.SnowballParams;
import io.qdrant.client.grpc.Collections.StemmingAlgorithm;
import io.qdrant.client.grpc.Collections.StopwordsSet;
import io.qdrant.client.grpc.Collections.TextIndexParams;
import io.qdrant.client.grpc.Collections.TextScoringParams;
import io.qdrant.client.grpc.Collections.TextScoringType;
import io.qdrant.client.grpc.Collections.TokenizerType;

QdrantClient client =

client
    .createPayloadIndexAsync(
        "books",
        "title",
        PayloadSchemaType.Text,
        PayloadIndexParams.newBuilder()
            .setTextIndexParams(
                TextIndexParams.newBuilder()
                    .setTokenizer(TokenizerType.Word)
                    .setStemmer(
                        StemmingAlgorithm.newBuilder()
                            .setSnowball(
                                SnowballParams.newBuilder().setLanguage("spanish").build())
                            .build())
                    .setStopwords(StopwordsSet.newBuilder().addLanguages("spanish").build())
                    .setScoring(
                        TextScoringParams.newBuilder().setType(TextScoringType.Bm25).build())
                    .build())
            .build(),
        null,
        null,
        null)
    .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.CreatePayloadIndexAsync(
    collectionName: "books",
    fieldName: "title",
    schemaType: PayloadSchemaType.Text,
    indexParams: new PayloadIndexParams
    {
        TextIndexParams = new TextIndexParams
        {
            Tokenizer = TokenizerType.Word,
            Stemmer = new StemmingAlgorithm
            {
                Snowball = new SnowballParams { Language = "spanish" }
            },
            Stopwords = new StopwordsSet { Languages = { "spanish" } },
            Scoring = new TextScoringParams { Type = TextScoringType.Bm25 }
        }
    }
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.CreateFieldIndex(context.Background(), &qdrant.CreateFieldIndexCollection{
	CollectionName: "books",
	FieldName:      "title",
	FieldType:      qdrant.FieldType_FieldTypeText.Enum(),
	FieldIndexParams: qdrant.NewPayloadIndexParamsText(
		&qdrant.TextIndexParams{
			Tokenizer: qdrant.TokenizerType_Word,
			Stemmer: qdrant.NewStemmingAlgorithmSnowball(&qdrant.SnowballParams{
				Language: "spanish",
			}),
			Stopwords: &qdrant.StopwordsSet{
				Languages: []string{"spanish"},
			},
			Scoring: &qdrant.TextScoringParams{
				Type: qdrant.TextScoringType_Bm25,
			},
		}),
})

Corpus Statistics

Qdrant computes the BM25 statistics, such as the number of documents, the number of documents containing each term, and the average document length, over all points in the shard being queried. In a collection with multiple shards, each shard uses its own statistics, so the same document can score slightly differently depending on the shard it’s stored in. To compute statistics over a subset of the data, such as a single tenant, see Per-Tenant Scoring Statistics.

After points are deleted or updated, the statistics can include outdated data until the optimizer rebuilds the affected segments. As a result, scores can shift slightly until optimization. Deleted points never appear in the results.

Score Elements of an Array Separately

If a text field is inside an array of objects, such as reviews[].text, a regular full-text index treats the tokens of all elements as if they were in a single document. When the elements are independent texts, such as reviews or passages of a long document, you may want to score each element as its own document instead. To do so, set the nested parameter when you create the index, which makes it a per-element index. A text query then scores each element separately, and a point gets the score of its best-matching element. Corpus statistics count each element as a separate document, so the average document length is the average length of an element.

Migrate from Sparse Vector BM25

If you use the sparse vector BM25 model, you can switch to BM25 on a full-text index without uploading your data again, as long as the text is stored in the payload:

Create a full-text index with scoring enabled on the payload field that holds the text.
Configure the same text processing as the sparse vector BM25 model. The model uses English stemming and stopword removal by default, while a full-text index doesn’t. If you use the default BM25 sparse vector model settings, set the english Snowball stemmer and english stopwords on the index. If you don’t use the default settings, make sure the index uses the same tokenizer, stemmer, stopwords, and ASCII folding settings as your sparse vector BM25 model.
Replace queries on the sparse vector with text queries, setting using to the payload field instead of the sparse vector name.

The full-text index computes the average document length from your data, so you no longer need to set avg_len. Scores can differ from the sparse vector BM25 model, especially if avg_len didn’t match your data.

Was this page useful?

On this page:

Ready to try Qdrant?