Module 3

Final Project: Build Your Own Multi-Vector Search System

Your Mission

It’s time to bring together everything you’ve learned about multi-vector search, late interaction models, and production optimization. You’ll build a sophisticated document retrieval system that leverages late interaction’s token-level matching for superior search quality.

Your search engine will understand the nuanced relationships between query terms and document content. When someone searches for “machine learning applications in healthcare,” your system will find documents that discuss relevant concepts even when they use different terminology, thanks to late interaction’s fine-grained matching.

This mirrors real-world challenges in enterprise search, research libraries, and knowledge management. You’ll implement the complete pipeline: multi-vector embedding using ColModernVBERT or ColPali, memory-efficient storage with quantization or pooling, and optimized retrieval with multi-stage search.

What You’ll Build

A working multi-vector search system that:

• Indexes real documents using late interaction embeddings
• Applies optimization techniques to manage memory and latency
• Retrieves relevant results with measurable quality
• Documents your design decisions and trade-offs

Both ColModernVBERT and ColPali work well for visual document understanding - choose based on your preference or experiment with both.

The specific dataset, optimization strategy, and retrieval configuration are up to you.

Choose Your Challenge

This is your project. Pick a problem that matters to you.

Use Your Own Data

The most valuable learning comes from working with documents you actually care about:

• Technical documentation you reference frequently
• Research papers in your field of interest
• Internal documents (reports, manuals, wikis) from your work
• Personal collection of PDFs, articles, or notes

Aim for at least 50 documents to have enough variety for meaningful evaluation. More is better for seeing how your system scales.

Public Datasets (If You Prefer)

If you don’t have a suitable personal dataset:

• ArXiv papers on a topic you’re curious about
• Wikipedia articles from a category you’d like to explore
• Open-source documentation from projects you use
• News articles from a domain you follow

The key is picking something where you can judge search quality intuitively. You’ll need to create evaluation queries, and that’s easier when you understand the content.

Project Requirements

Your project should demonstrate:

1. Working Multi-Vector Search

Use ColModernVBERT to index your documents and retrieve results using MaxSim scoring. The system should return relevant documents for natural language queries. Alternatively, consider ColPali.

2. At Least One Optimization Technique

Apply something you learned in this module:

• Binary or scalar quantization
• Token pooling (clustering, attention-based, or hierarchical)
• MUVERA indexing
• Multi-stage retrieval pipeline

Measure the impact of your chosen technique on memory, latency, or search quality.

3. Evaluation with Ground Truth

Create a test set of queries with known relevant documents. This doesn’t need to be exhaustive - 10-20 queries with 3-5 relevant documents each is enough to see meaningful patterns. Measure at least one retrieval metric (precision@k, recall@k, or MRR).

4. Brief Write-Up

Document your decisions:

• Why you chose your dataset
• Whether you used ColModernVBERT or ColPali, and why
• What optimization technique(s) you applied and why
• What worked well and what surprised you
• Key metrics from your evaluation

Suggested Approach

These hints are optional. Feel free to chart your own path.

Start simple. Get a basic multi-vector search working before adding optimizations. It’s easier to measure the impact of changes when you have a baseline.

Create ground truth early. Before optimizing, write your evaluation queries and identify relevant documents. This lets you measure whether changes improve or hurt quality.

Compare configurations. Try at least two different setups (e.g., with and without quantization, or different pooling strategies). The comparison will teach you more than a single configuration.

Keep notes as you go. Document what you try and what happens. Your future self will thank you, and it makes the write-up easier.

Share Your Results

We’d love to see what you build. Share your project on the Qdrant Discord in the #course-submissions channel.

Tell us about:

• Your dataset and why you chose it
• Your model choice (ColModernVBERT or ColPali, and why)
• Your collection configuration (quantization, pooling, indexing)
• Your retrieval metrics (precision@10, recall, latency)
• What you learned and what surprised you

Seeing how others approached the same challenge is one of the best ways to deepen your understanding.

What You’ve Accomplished

By completing this project, you’ve built a production-ready multi-vector search system that:

• Leverages late interaction for superior search quality through token-level matching
• Scales efficiently using quantization, pooling, or optimized indexing
• Delivers fast results through multi-stage retrieval and HNSW optimization
• Measures quality with comprehensive evaluation metrics
• Documents trade-offs between accuracy, speed, and memory

You’ve mastered the full pipeline from multi-vector embeddings to production optimization - skills directly applicable to enterprise search, document management, research platforms, and AI-powered knowledge systems.

Congratulations on completing the Multi-Vector Search course.

Continue your learning journey by exploring advanced topics like fine-tuning late interaction models for domain-specific documents, or building Retrieval Augmented Generation pipelines to derive insights from your retrieved documents.