Log in Start Free

TwelveLabs

TwelveLabs provides Marengo, an embedding model that places video, audio, images, text, and documents in one vector space, so a query in one modality can find content in another. There are two ways to use Marengo with Qdrant:

  • Call the TwelveLabs Embed API v2 from your application and send the vectors to Qdrant. This is what this page covers.
  • Call Marengo 3.0 through Amazon Bedrock in your AWS account.

Models

ModelDimensionsMax Text TokensInput
marengo3.5512 (default), 256, or 1282,000Video, audio, image, text, and PDF, plain text, or Markdown documents
marengo3.0512500Video, audio, image, and text

Use marengo3.5 for new projects. It has no length limit for video and audio, while Marengo 3.0 accepts up to 4 hours. Marengo 2.7 (Marengo-retrieval-2.7) and the Embed API v1 are previous generations. Embeddings from different Marengo versions are not compatible, so re-embed your content when you switch models.

TwelveLabs lists cosine similarity as the metric for Marengo, so create the collection with the Cosine distance.

Setup

Install the TwelveLabs SDK and the Qdrant client:

  • Python: pip install twelvelabs qdrant-client
  • TypeScript: npm install twelvelabs-js @qdrant/js-client-rest

Create both clients and a collection sized for the model. These examples read your API key from the TWELVE_LABS_API_KEY environment variable. The Node.js SDK also reads it when you omit apiKey, but the Python SDK requires the api_key argument.

import os
import time

from qdrant_client import QdrantClient, models
from twelvelabs import (
    AsyncVideoInputRequest,
    MediaSource,
    MultiInputMediaSource,
    MultiInputRequest,
    TwelveLabs,
)

twelvelabs_client = TwelveLabs(api_key=os.environ["TWELVE_LABS_API_KEY"])
client = QdrantClient(url="http://localhost:6333")

MODEL = "marengo3.5"

client.create_collection(
    collection_name="{collection_name}",
    vectors_config=models.VectorParams(size=512, distance=models.Distance.COSINE),
)

import { TwelveLabs } from "twelvelabs-js";
import { QdrantClient } from "@qdrant/js-client-rest";

const twelvelabs = new TwelveLabs({ apiKey: process.env.TWELVE_LABS_API_KEY });
const client = new QdrantClient({ url: "http://localhost:6333" });

const MODEL = "marengo3.5";

await client.createCollection("{collection_name}", {
  vectors: { size: 512, distance: "Cosine" },
});

Embed Videos

TwelveLabs embeds video asynchronously. Create an embedding task, poll it until its status is ready, and read the embeddings from its data field. With embedding_scope set to clip, Marengo splits the video into segments and returns embeddings for each one. By default, Marengo 3.5 returns two embeddings per segment, one visual and one audio, and both are in the same vector space. The response has no index field, so this example uses the position in data as the point ID, and stores the modality and time range of each segment in the payload:

VIDEO_URL = "https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_2MB.mp4"

task = twelvelabs_client.embed.v_2.tasks.create(
    input_type="video",
    model_name=MODEL,
    video=AsyncVideoInputRequest(
        media_source=MediaSource(url=VIDEO_URL),
        embedding_scope=["clip"],
    ),
)

result = twelvelabs_client.embed.v_2.tasks.retrieve(task.id)
while result.status == "processing":
    time.sleep(5)
    result = twelvelabs_client.embed.v_2.tasks.retrieve(task.id)

if result.status != "ready" or result.data is None:
    raise RuntimeError(f"Embedding task {task.id} failed: {result.error}")

client.upsert(
    collection_name="{collection_name}",
    points=[
        models.PointStruct(
            id=idx,
            vector=segment.embedding,
            payload={
                "video_url": VIDEO_URL,
                "embedding_option": segment.embedding_option,
                "start_sec": segment.start_sec,
                "end_sec": segment.end_sec,
            },
        )
        for idx, segment in enumerate(result.data)
    ],
)

const VIDEO_URL =
  "https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_2MB.mp4";

const task = await twelvelabs.embed.v2.tasks.create({
  inputType: "video",
  modelName: MODEL,
  video: {
    mediaSource: { url: VIDEO_URL },
    embeddingScope: ["clip"],
  },
});

let result = await twelvelabs.embed.v2.tasks.retrieve(task.id);
while (result.status === "processing") {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  result = await twelvelabs.embed.v2.tasks.retrieve(task.id);
}

if (result.status !== "ready" || !result.data) {
  throw new Error(`Embedding task ${task.id} failed`);
}

await client.upsert("{collection_name}", {
  points: result.data.map((segment, idx) => ({
    id: idx,
    vector: segment.embedding,
    payload: {
      video_url: VIDEO_URL,
      embedding_option: segment.embeddingOption,
      start_sec: segment.startSec,
      end_sec: segment.endSec,
    },
  })),
});

Each task embeds one file. Video URLs must be direct links to the raw file and can point to files up to 4 GB. To embed a local file, upload it first with assets.create, which accepts up to 200 MB, and pass the asset ID in media_source. The optional segmentation field sets fixed-length or scene-based segments. TwelveLabs keeps task results for seven days, so store the vectors in Qdrant when the task finishes. Audio files, images, and documents use the same task flow with input_type set to audio, image, or document.

Embed Queries

Marengo has no separate query mode. Embed queries with the synchronous embed.v_2.create method and the same model as your content. It returns the embedding in the response instead of creating a task:

response = twelvelabs_client.embed.v_2.create(
    input_type="multi_input",
    model_name=MODEL,
    multi_input=MultiInputRequest(input_text="A big rabbit wakes up in a forest"),
)

hits = client.query_points(
    collection_name="{collection_name}",
    query=response.data[0].embedding,
    limit=3,
).points

const response = await twelvelabs.embed.v2.create({
  inputType: "multi_input",
  modelName: MODEL,
  multiInput: { inputText: "A big rabbit wakes up in a forest" },
});

const { points: hits } = await client.query("{collection_name}", {
  query: response.data[0].embedding,
  limit: 3,
});

The synchronous method accepts up to 2,000 tokens of text and up to 10 media sources per request. Each media source can be up to 32 MB, and video and audio up to 30 seconds. For longer files, use an embedding task.

Search Across Modalities

Because all modalities share one vector space, a query can be an audio clip or an image instead of text. Pass the file in media_sources. These examples also filter on the embedding_option payload field: the audio query searches the audio embeddings of the video, and the image query searches the visual embeddings.

To search with an audio clip:

audio_query = twelvelabs_client.embed.v_2.create(
    input_type="multi_input",
    model_name=MODEL,
    multi_input=MultiInputRequest(
        media_sources=[
            MultiInputMediaSource(
                media_type="audio",
                url="https://codeskulptor-demos.commondatastorage.googleapis.com/descent/background%20music.mp3",
            ),
        ],
    ),
)

hits = client.query_points(
    collection_name="{collection_name}",
    query=audio_query.data[0].embedding,
    query_filter=models.Filter(
        must=[models.FieldCondition(key="embedding_option", match=models.MatchValue(value="audio"))]
    ),
    limit=3,
).points

const audioQuery = await twelvelabs.embed.v2.create({
  inputType: "multi_input",
  modelName: MODEL,
  multiInput: {
    mediaSources: [
      {
        mediaType: "audio",
        url: "https://codeskulptor-demos.commondatastorage.googleapis.com/descent/background%20music.mp3",
      },
    ],
  },
});

const { points: audioHits } = await client.query("{collection_name}", {
  query: audioQuery.data[0].embedding,
  filter: { must: [{ key: "embedding_option", match: { value: "audio" } }] },
  limit: 3,
});

To search with an image:

image_query = twelvelabs_client.embed.v_2.create(
    input_type="multi_input",
    model_name=MODEL,
    multi_input=MultiInputRequest(
        media_sources=[
            MultiInputMediaSource(
                media_type="image",
                url="https://upload.wikimedia.org/wikipedia/commons/c/c5/Big_buck_bunny_poster_big.jpg",
            ),
        ],
    ),
)

hits = client.query_points(
    collection_name="{collection_name}",
    query=image_query.data[0].embedding,
    query_filter=models.Filter(
        must=[models.FieldCondition(key="embedding_option", match=models.MatchValue(value="visual"))]
    ),
    limit=3,
).points

const imageQuery = await twelvelabs.embed.v2.create({
  inputType: "multi_input",
  modelName: MODEL,
  multiInput: {
    mediaSources: [
      {
        mediaType: "image",
        url: "https://upload.wikimedia.org/wikipedia/commons/c/c5/Big_buck_bunny_poster_big.jpg",
      },
    ],
  },
});

const { points: imageHits } = await client.query("{collection_name}", {
  query: imageQuery.data[0].embedding,
  filter: { must: [{ key: "embedding_option", match: { value: "visual" } }] },
  limit: 3,
});

A single query can also combine text with media. Set both input_text and media_sources, give a media source a name, and refer to it in the text as <@name>. The result is one embedding.

Reduce Vector Size

Marengo 3.5 accepts an embedding_dimension parameter of 512, 256, or 128. Shorter vectors make the index smaller and search faster, and longer vectors give better retrieval quality. Set the collection’s size to the same value, and pass the same embedding_dimension when you embed queries:

task = twelvelabs_client.embed.v_2.tasks.create(
    input_type="video",
    model_name=MODEL,
    embedding_dimension=256,
    video=AsyncVideoInputRequest(media_source=MediaSource(url=VIDEO_URL)),
)

response = twelvelabs_client.embed.v_2.create(
    input_type="multi_input",
    model_name=MODEL,
    embedding_dimension=256,
    multi_input=MultiInputRequest(input_text="A big rabbit wakes up in a forest"),
)

const reducedTask = await twelvelabs.embed.v2.tasks.create({
  inputType: "video",
  modelName: MODEL,
  embeddingDimension: 256,
  video: { mediaSource: { url: VIDEO_URL } },
});

const reducedQuery = await twelvelabs.embed.v2.create({
  inputType: "multi_input",
  modelName: MODEL,
  embeddingDimension: 256,
  multiInput: { inputText: "A big rabbit wakes up in a forest" },
});

Store and Search the Vectors

Create a Collection

Match the model's dimensions and distance metric.

Upload Vectors

Batch points with payloads for filtering.

Search

Find matching points with the Query API.

Next Steps

Was this page useful?

Thank you for your feedback! 🙏

We are sorry to hear that. 😔 You can edit this page on GitHub, or create a GitHub issue.

On this page:

Ready to try Qdrant?