Dust and Qdrant: Using AI to Unlock Company Knowledge and Drive Employee Productivity
One of the major promises of artificial intelligence is its potential to accelerate efficiency and productivity within businesses, empowering employees and teams in their daily tasks. The French company Dust, co-founded by former Open AI Research Engineer Stanislas Polu, set out to deliver on this promise by providing businesses and teams with an expansive platform for building customizable and secure AI assistants.
Challenge
“The past year has shown that large language models (LLMs) are very useful but complicated to deploy,” Polu says, especially in the context of their application across business functions. This is why he believes that the goal of augmenting human productivity at scale is especially a product unlock and not only a research unlock, with the goal to identify the best way for companies to leverage these models. Therefore, Dust is creating a product that sits between humans and the large language models, with the focus on supporting the work of a team within the company to ultimately enhance employee productivity.
A major challenge in leveraging leading LLMs like OpenAI, Anthropic, or Mistral to their fullest for employees and teams lies in effectively addressing a company’s wide range of internal use cases. These use cases are typically very general and fluid in nature, requiring the use of very large language models. Due to the general nature of these use cases, it is very difficult to finetune the models - even if financial resources and access to the model weights are available. The main reason is that “the data that’s available in a company is a drop in the bucket compared to the data that is needed to finetune such big models accordingly,” Polu says, “which is why we believe that retrieval augmented generation is the way to go until we get much better at fine tuning”.
For successful retrieval augmented generation (RAG) in the context of employee productivity, it is important to get access to the company data and to be able to ingest the data that is considered ‘shared knowledge’ of the company. This data usually sits in various SaaS applications across the organization.
Solution
Dust provides companies with the core platform to execute on their GenAI bet for their teams by deploying LLMs across the organization and providing context aware AI assistants through RAG. Users can manage so-called data sources within Dust and upload files or directly connect to it via APIs to ingest data from tools like Notion, Google Drive, or Slack. Dust then handles the chunking strategy with the embeddings models and performs retrieval augmented generation.
For this, Dust required a vector database and evaluated different options including Pinecone and Weaviate, but ultimately decided on Qdrant as the solution of choice. “We particularly liked Qdrant because it is open-source, written in Rust, and it has a well-designed API,” Polu says. For example, Dust was looking for high control and visibility in the context of their rapidly scaling demand, which made the fact that Qdrant is open-source a key driver for selecting Qdrant. Also, Dust’s existing system which is interfacing with Qdrant, is written in Rust, which allowed Dust to create synergies with regards to library support.
When building their solution with Qdrant, Dust took a two step approach:
Get started quickly: Initially, Dust wanted to get started quickly and opted for Qdrant Cloud, Qdrant’s managed solution, to reduce the administrative load on Dust’s end. In addition, they created clusters and deployed them on Google Cloud since Dust wanted to have those run directly in their existing Google Cloud environment. This added a lot of value as it allowed Dust to centralize billing and increase security by having the instance live within the same VPC. “The early setup worked out of the box nicely,” Polu says.
Scale and optimize: As the load grew, Dust started to take advantage of Qdrant’s features to tune the setup for optimization and scale. They started to look into how they map and cache data, as well as applying some of Qdrant’s built-in compression features. In particular, Dust leveraged the control of the MMAP payload threshold as well as Scalar Quantization, which enabled Dust to manage the balance between storing vectors on disk and keeping quantized vectors in RAM, more effectively. “This allowed us to scale smoothly from there,” Polu says.
Results
Dust has seen success in using Qdrant as their vector database of choice, as Polu acknowledges: “Qdrant’s ability to handle large-scale models and the flexibility it offers in terms of data management has been crucial for us. The observability features, such as historical graphs of RAM, Disk, and CPU, provided by Qdrant are also particularly useful, allowing us to plan our scaling strategy effectively.”
Dust was able to scale its application with Qdrant while maintaining low latency across hundreds of thousands of collections with retrieval only taking milliseconds, as well as maintaining high accuracy. Additionally, Polu highlights the efficiency gains Dust was able to unlock with Qdrant: “We were able to reduce the footprint of vectors in memory, which led to a significant cost reduction as we don’t have to run lots of nodes in parallel. While being memory-bound, we were able to push the same instances further with the help of quantization. While you get pressure on MMAP in this case you maintain very good performance even if the RAM is fully used. With this we were able to reduce our cost by 2x.”
Outlook
Dust will continue to build out their platform, aiming to be the platform of choice for companies to execute on their internal GenAI strategy, unlocking company knowledge and driving team productivity. Over the coming months, Dust will add more connections, such as Intercom, Jira, or Salesforce. Additionally, Dust will expand on its structured data capabilities.
To learn more about how Dust uses Qdrant to help employees in their day to day tasks, check out our Vector Space Talk featuring Stanislas Polu, Co-Founder of Dust.