DocsGuides

Multimodal RAG

Most enterprise documents are not plain text. They're PDFs with tables, slide decks with charts, screenshots, and scanned forms. Multimodal RAG indexes the visual structure directly instead of throwing it away through OCR.

Shared embedding spaces

The trick that makes multimodal retrieval work is a shared embedding space: an image and a text caption describing it land near each other, even though one came from a vision encoder and the other from a text encoder. CLIP (OpenAI, 2021) trained this on 400M (image, caption) pairs with a contrastive objective; SigLIP and EVA-CLIP are stronger modern replacements.

CLIP / SigLIP

Image ↔ text, one vector each

ColPali / ColQwen

Per-patch vectors for PDFs

CLAP

Audio ↔ text

VideoCLIP

Video frames ↔ text

The simplest setup — CLIP + a vector DB

python

import torch, open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-L-14", pretrained="laion2b_s32b_b82k"
)
tokenizer = open_clip.get_tokenizer("ViT-L-14")
model.eval()

@torch.no_grad()
def embed_image(path):
    img = preprocess(Image.open(path)).unsqueeze(0)
    v = model.encode_image(img)
    return (v / v.norm(dim=-1, keepdim=True)).squeeze().numpy()

@torch.no_grad()
def embed_text(text):
    v = model.encode_text(tokenizer([text]))
    return (v / v.norm(dim=-1, keepdim=True)).squeeze().numpy()

# Index images, query with text — same 768-d space
vector_db.upsert(id=img_id, vector=embed_image("slide_42.png"))
hits = vector_db.search(embed_text("Q3 revenue chart"), k=5)

CLIP is OK, not great, for documents

CLIP was trained on natural photos with short captions. It's fine for stock-photo search; it struggles with dense documents, charts, and screenshots full of small text. For document-heavy corpora, jump straight to ColPali.

ColPali — retrieval over PDFs without OCR

ColPali (Faysse et al., 2024) treats each PDF page as an image, runs it through PaliGemma to get one embedding per visual patch (~1024 per page), and uses ColBERT-style late interaction at query time. It indexes the page as it appears to a human — tables, math, charts, handwriting, fonts — without OCR ever running. On the ViDoRe benchmark it beats text-only pipelines by a wide margin on chart-heavy and tabular PDFs.

Cost: index size is ~1000× larger per document than single-vector CLIP, because you store per-patch vectors. Quantization (PQ, binary) is essential for production.

The generator: vision-language models

Once retrieval returns image chunks (or page screenshots), the generator needs to see them. Modern VLMs — GPT-4o, Claude 3.5 Sonnet, Gemini 2.0, Qwen2-VL — accept images in the prompt natively. You stuff the top-k page images directly in alongside the question instead of converting them to text first.

python

messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": f"Answer based only on these pages:\n{question}"},
        *[{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64(page)}"}}
          for page in retrieved_pages[:5]],
    ],
}]
response = vlm.chat.completions.create(model="gpt-4o", messages=messages)

Hybrid: text + image in one index

Real corpora mix both. The clean pattern: embed text with a strong text retriever (BGE, E5), embed images/pages with CLIP or ColPali, store both in the same vector DB tagged with modality, and run RRF (see advanced RAG) to fuse the two ranked lists. Filter by modality when the question is unambiguous ("show me the slide where...").

What breaks in multimodal RAG

Index size

Patch-level models: 100–1000× larger

Quantization

Mandatory at scale (PQ, BQ)

Eval

ViDoRe for docs, MS-COCO for photos

Cost

VLM tokens-per-image are pricy

Quick Quiz

Test yourself · 3 questions

Q1.

What makes CLIP usable for text-to-image retrieval?

Q2.

What's the main advantage of ColPali over OCR + text retrieval for PDFs?

Q3.

What's the cost trade-off of patch-level multimodal retrievers?