Module 2: First Principles of Vector Search on Qdrant - Vector Search Engine

Module Overview

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

First Principles of Vector Search

Understand collections, points, vectors, payloads, and the HNSW index, and move from theory to actual system design in Qdrant.

Follow-along code: Module 2 notebook

Overview

Module 1 explained why semantic search works. In this module, you’ll learn where your data lives and how Qdrant searches it. You’ll explore collections, points, vectors, payloads, and distance metrics, then see how Qdrant finds the top-k matches without scanning every vector. You’ll also learn how to filter results by metadata and split long documents into smaller chunks before embedding them. By the end, you’ll have created a collection, stored points, and run your first filtered query.

From Idea to System

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

From Idea to System

In Module 1, we saw how search evolved from matching words to understanding meaning. Now we move from theory to actual system design. This module covers every building block you need to go from raw text to a running Qdrant collection.

Raw Text Documents, articles, PDFs
Chunk Split into passages
Embed Convert to dense vectors
Store Upsert to Qdrant: insert a point if its ID is new, update it if the ID already exists

Core Data Model

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Core Data Model

Qdrant organizes data in three levels. Understanding this structure is the foundation for everything else in the course.

Collection

Like a table in a relational database. Stores vectors of a fixed size and a chosen distance metric. Every point in a collection must have a vector of the same dimension.

Distance Metrics

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Distance Metrics

When you query a collection, Qdrant compares your query vector with the stored vectors using the distance metric you chose when creating the collection. For text embeddings, cosine similarity is the most common metric.

Checking every vector would be too slow for large collections. Instead, Qdrant uses an HNSW index to find the closest matches efficiently without scanning the entire collection. Fast Approximate Search: HNSW explains how it works.

Top-K Retrieval

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Top-K Retrieval

A search query is converted into a vector using the same embedding model used to embed your documents. Qdrant finds the K points in the collection whose vectors are most similar to the query vector, ranked by similarity score.

results = client.query_points(
 collection_name="articles",
 query=[0.12, -0.87, 0.33, ...], # your query vector
 limit=3, # return top 3
)

for r in results.points:
 print(r.id, r.score, r.payload)

Why K Matters

Returning too few results (K=3) misses relevant content. Returning too many (K=100) creates noise in results. A common approach is to overfetch: retrieve a larger candidate pool, then rerank it down to the smaller K you actually show the user. Qdrant supports this natively via multi-stage queries - for example, prefetching a large candidate set and reranking it down to a much smaller final limit. We’ll cover reranking in detail later.

Fast Approximate Search: HNSW

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Fast Approximate Search: HNSW

Searching millions of vectors by computing similarity against every single one (brute force) is slow. Qdrant uses HNSW (Hierarchical Navigable Small World), a graph-based approximate nearest neighbor (ANN) index that makes large-scale search fast at a small, measurable recall cost.

How HNSW Works

Graph structure: Each vector is a node. Nodes are connected to their nearest neighbors by bidirectional edges, forming a navigable graph.
Hierarchical layers: The graph has multiple layers. The top layer has few nodes and long-range connections. Lower layers are denser with short-range connections.
Search by traversal: Query entry starts at the top layer. The search “jumps” through neighbors, zooming in on the region of interest at each layer.
Approximate, not exact: HNSW trades some recall (see below) for massive speed gains. Whether that trade-off is worth it depends on your data and queries, so measure recall on queries representative of your actual workload rather than assuming it.

Tunable Parameters

HNSW exposes three tunable parameters: m, ef_construct, and hnsw_ef. They balance search speed, recall (the fraction of true nearest neighbors found), memory usage, and indexing time.

Payload Filtering

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Payload Filtering

Payload filtering lets you apply hard conditions during HNSW traversal, not after retrieval. This keeps results both semantically relevant and legally/logically valid.

This searches by vector similarity as usual, but only among points whose payload passes the filter:

Filter — the overall condition
must — a list of conditions that all have to be true (AND logic)
FieldCondition — checks one payload field; here, that category equals "automotive"

Filter Types

Condition	What it does	Example use case
must	All conditions must be true (AND logic)	Category = automotive AND year >= 2022
should	At least one condition must be true (OR logic)	Category = automotive OR category = transport
must_not	Exclude matching points	Exclude documents flagged as deleted or expired
Range	Numeric range comparisons (gte, lte, gt, lt)	year between 2020 and 2024
Geo	Geospatial radius or bounding box filter	Restaurants within 5 km of user location

from qdrant_client.models import Filter, FieldCondition, MatchValue

results = client.query_points(
 collection_name="articles",
 query=[...],
 query_filter=Filter(
 must=[
 FieldCondition(
 # the payload field to check
 key="category", 
 # keep only points where category == "automotive"
 match=MatchValue(value="automotive") 
 )
 ]
 ),
 limit=5,
)

Index Your Filter Fields

For fields you filter frequently, create a payload index. Without one, Qdrant may need to check payload values across many points at query time. With one, it can look up matching points directly, making filtered queries faster.

Chunking Strategies

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Chunking Strategies

Embedding models have a maximum token limit. all-MiniLM-L6-v2 from Module 1 takes 256 tokens, larger models take 8,000 or more, and anything past the limit is dropped without an error. Check your model’s card for its limit.

Fitting isn’t the only reason to split. A chunk is the unit that gets retrieved, so one vector covering several topics averages them together and matches every query weakly, while a chunk that’s too small loses the context that made the result useful.

Ingestion Pipeline: End-to-End

info@qdrant.tech (Andrey Vasnetsov) — Mon, 01 Jan 0001 00:00:00 +0000

Module 2

Ingestion Pipeline: End-to-End

Let’s put everything together. This section walks through the complete ingestion pipeline from cloud setup to your first query.

Step 1: Connect to Your Cluster

Module 0 walks you through creating a free cluster at Qdrant Cloud and retrieving its URL and API key. Use these credentials to initialize the Qdrant client:

from qdrant_client import QdrantClient

client = QdrantClient(
 url="https://xyz-example.eu-west-1-0.aws.cloud.qdrant.io", # your cluster's URL
 api_key="<your-api-key>", # your API key
)
# In a real project, don't hardcode these; load them from environment
# variables or a secrets manager instead of committing them to source control.

Step 2: Create the Collection

from qdrant_client import models

client.create_collection(
 collection_name="articles",
 vectors_config=models.VectorParams(
 size=384,
 distance=models.Distance.COSINE,
 ))

# Qdrant Cloud runs in strict mode, which rejects filtered queries on payload
# fields that aren't indexed. Step 4 filters on "category", so create that
# index now, before ingesting or querying.
client.create_payload_index(
 collection_name="articles",
 field_name="category",
 field_schema=models.PayloadSchemaType.KEYWORD,
)

Step 3: Ingest Data

!pip install fastembed

from qdrant_client.models import PointStruct
from fastembed import TextEmbedding

model = TextEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2") # 384-dim

documents = [
 {"id": 1, "text": "Car repair guide", "category": "automotive"},
 {"id": 2, "text": "How to cook pasta", "category": "food"},
]

points = [
 PointStruct(
 id=doc["id"],
 vector=vector.tolist(),
 payload={"title": doc["text"], "category": doc["category"]},
 )
 for doc, vector in zip(documents, model.embed([d["text"] for d in documents]))
]
# upload_points handles batching and retries automatically; preferred for lists of points.
# upsert is the raw operation, better for single points or small real-time updates.
client.upload_points(collection_name="articles", points=points)

Step 4: Query

This embeds the user’s question the same way we embedded the documents, then searches with a payload filter on top: same pattern as Payload Filtering, now filtering to only the “automotive” category:

Module 2: First Principles of Vector Search on Qdrant - Vector Search Engine

Module Overview

First Principles of Vector Search

Overview

From Idea to System

From Idea to System

Core Data Model

Core Data Model

Collection

Distance Metrics

Distance Metrics

Top-K Retrieval

Top-K Retrieval

Why K Matters

Fast Approximate Search: HNSW

Fast Approximate Search: HNSW

How HNSW Works

Tunable Parameters

Payload Filtering

Payload Filtering

Filter Types

Index Your Filter Fields

Chunking Strategies

Chunking Strategies

Ingestion Pipeline: End-to-End

Ingestion Pipeline: End-to-End

Step 1: Connect to Your Cluster

Step 2: Create the Collection

Step 3: Ingest Data

Step 4: Query

Further Reading

Further Reading

What’s Next: Module 3