Inference on Qdrant - Vector Search Engine

Inference API

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Inference API

When using Qdrant Cloud Inference or Qdrant’s BM25 model, embeddings are generated server-side. Instead of pre-computed vectors, you pass Inference Objects when ingesting or querying data. An Inference Object tells Qdrant how to generate a vector from your input. It contains the input, such as text or an image, along with the model to use. The API supports three types of Inference Objects:

Document object, used for text inference

// Document
{
 // Text input
 text: "Your text",
 // Name of the model, to do inference with
 model: "<the-model-to-use>",
 // Extra parameters for the model, Optional
 options: {}
}

Image object, used for image inference

BM25

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Server-side Inference: BM25

BM25 (Best Matching 25) is a ranking function for text search. BM25 uses sparse vectors that represent documents, where each dimension corresponds to a word. Qdrant can generate these sparse embeddings from input text directly on the server.

While upserting points, provide the text and the qdrant/bm25 embedding model:

PUT /collections/{collection_name}/points
{
 "points": [
 {
 "id": 1,
 "vector": {
 "my-bm25-vector": {
 "text": "Recipe for baking chocolate chip cookies",
 "model": "qdrant/bm25"
 }
 }
 }
 ]
}

from qdrant_client import QdrantClient, models

client.upsert(
 collection_name="{collection_name}",
 points=[
 models.PointStruct(
 id=1,
 vector={
 "my-bm25-vector": models.Document(
 text="Recipe for baking chocolate chip cookies",
 model="Qdrant/bm25",
 )
 },
 )
 ],
)

import { QdrantClient } from "@qdrant/js-client-rest";

client.upsert("{collection_name}", {
 points: [
 {
 id: 1,
 vector: {
 'my-bm25-vector': {
 text: 'Recipe for baking chocolate chip cookies',
 model: 'Qdrant/bm25',
 },
 },
 },
 ],
});

use qdrant_client::{
 Payload, Qdrant,
 qdrant::{DocumentBuilder, PointStruct, UpsertPointsBuilder},
};
use std::collections::HashMap;

client
 .upsert_points(UpsertPointsBuilder::new(
 "{collection_name}",
 vec![PointStruct::new(
 1,
 HashMap::from([(
 "my-bm25-vector".to_string(),
 DocumentBuilder::new("Recipe for baking chocolate chip cookies", "qdrant/bm25")
 .build(),
 )]),
 Payload::default(),
 )],
 ))
 .await?;

import static io.qdrant.client.PointIdFactory.id;
import static io.qdrant.client.ValueFactory.value;
import static io.qdrant.client.VectorFactory.vector;
import static io.qdrant.client.VectorsFactory.namedVectors;

import io.qdrant.client.QdrantClient;
import io.qdrant.client.QdrantGrpcClient;
import io.qdrant.client.grpc.Points.Document;
import io.qdrant.client.grpc.Points.Image;
import io.qdrant.client.grpc.Points.PointStruct;
import java.util.List;
import java.util.Map;

 client
 .upsertAsync(
 "{collection_name}",
 List.of(
 PointStruct.newBuilder()
 .setId(id(1))
 .setVectors(
 namedVectors(
 Map.of(
 "my-bm25-vector",
 vector(
 Document.newBuilder()
 .setModel("qdrant/bm25")
 .setText("Recipe for baking chocolate chip cookies")
 .build()))))
 .build()))
 .get();

using Qdrant.Client;
using Qdrant.Client.Grpc;

await client.UpsertAsync(
 collectionName: "{collection_name}",
 points: new List<PointStruct>
 {
 new()
 {
 Id = 1,
 Vectors = new Dictionary<string, Vector>
 {
 ["my-bm25-vector"] = new Document()
 {
 Model = "qdrant/bm25",
 Text = "Recipe for baking chocolate chip cookies",
 },
 },
 },
 }
);

import (
	"context"

	"github.com/qdrant/go-client/qdrant"
)

client.Upsert(context.Background(), &qdrant.UpsertPoints{
	CollectionName: "{collection_name}",
	Points: []*qdrant.PointStruct{
		{
			Id: qdrant.NewIDNum(uint64(1)),
			Vectors: qdrant.NewVectorsMap(map[string]*qdrant.Vector{
				"my-bm25-vector": qdrant.NewVectorDocument(&qdrant.Document{
					Model: "qdrant/bm25",
					Text: "Recipe for baking chocolate chip cookies",
				}),
			}),
		},
	},
})

Qdrant uses the model to generate the embeddings and stores the point with the resulting vector. Retrieving the point shows the embeddings that were generated:

Cloud Inference

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Qdrant Cloud Inference

Clusters on Qdrant Managed Cloud can use Qdrant Cloud Inference to generate embeddings. For a list of available models and their dimensions, visit the Inference tab of the Cluster Detail page in the Qdrant Cloud Console. You can also enable Cloud Inference for a cluster from this tab.

Several embedding models are free to use with Qdrant Cloud Inference, including on free-tier clusters. The “Cost: Free” label in the Inference tab identifies these models.

External Providers

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

External Embedding Model Providers

Qdrant Cloud can act as a proxy for the APIs of external embedding model providers:

OpenAI
Cohere
Jina AI
OpenRouter

This enables you to access any of the embedding models provided by these providers through the Qdrant API.

When using an external embedding model, ensure that your collection has been configured for vectors with the correct dimensionality. Refer to the model’s documentation for details on the output dimensions.

Matryoshka Models

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Reduce Vector Dimensionality with Matryoshka Models

Matryoshka Representation Learning (MRL) is a technique used to train embedding models to produce vectors that can be reduced in size with minimal loss of information. On Qdrant Cloud, for supported models, you can specify the mrl parameter in the options object to reduce the vector size to the desired dimension.

MRL on Qdrant Cloud helps minimize costs and latency when you need multiple sizes of the same vector. Instead of making several inference requests for each vector size, the inference service only generates embeddings for the full-sized vector and then reduces the vector to each requested smaller size.