RAG with MongoDB

Retrieval-augmented generation needs somewhere to store and search the embeddings it retrieves context from — and increasingly, teams are choosing to keep that vector data in the same database already holding their application's operational data, rather than standing up a separate specialized vector store. MongoDB, through its Atlas Vector Search capability, has become a popular option for exactly this reason. Here's how RAG with MongoDB actually fits together.

Why Use a General-Purpose Database for Vector Search

As covered in the ANN search and vector search performance posts, dedicated vector databases are purpose-built for similarity search at scale. But many applications already store their documents, user data, and metadata in MongoDB — and keeping embeddings alongside that same data, rather than syncing it to a separate vector store, offers real practical advantages: one system to operate, one data model to reason about, and the ability to combine vector similarity search with traditional filtering and querying in a single request.

The Core Building Block: Atlas Vector Search

MongoDB Atlas Vector Search lets you store vector embeddings as a field within a regular MongoDB document, alongside whatever other structured data that document already holds:

{
  "_id": "doc_123",
  "title": "Q3 Sales Report",
  "content": "Quarterly sales grew across all regions...",
  "category": "finance",
  "date": "2026-07-15",
  "embedding": [0.021, -0.114, 0.203, ...]
}

A vector search index is then created over the embedding field, enabling approximate nearest neighbor search (using an HNSW-based implementation) directly against that collection.

A Typical RAG Pipeline with MongoDB

1. Chunk and embed your source documents. Long documents typically get split into smaller chunks — a paragraph or a few hundred words — since retrieval works better on focused, semantically coherent pieces of text rather than entire documents at once. Each chunk gets converted into a vector embedding using an embedding model.

2. Store chunks and embeddings in MongoDB. Each chunk is inserted as a document, with its embedding and any relevant metadata (source, date, category, permissions) stored alongside it.

3. Create a vector search index. MongoDB Atlas lets you define a vector index on the embedding field, specifying dimensions and similarity function (cosine, dot product, or Euclidean).

4. Embed the incoming user query. When a user asks a question, that question gets converted into an embedding using the same model used for the stored chunks — consistency between query and document embeddings is essential for meaningful similarity comparisons.

5. Run a vector search to retrieve relevant chunks. MongoDB's $vectorSearch aggregation stage finds the chunks whose embeddings are closest to the query embedding, returning the most semantically relevant pieces of content.

6. Combine retrieval with metadata filtering. This is where storing vectors alongside structured data pays off directly — a single query can combine semantic similarity with traditional filters, like restricting results to documents from a certain date range or category, in one request rather than a separate filtering step.

7. Assemble retrieved chunks into the LLM's context. The retrieved chunks get inserted into a prompt (as covered in the context engineering post earlier in this series), giving the model grounded, relevant information to base its answer on rather than relying purely on its training data.

8. Generate the final response. The LLM produces an answer using the retrieved context, ideally citing or grounding its answer in the specific retrieved chunks.

A Simplified Query Example

pipeline = [
    {
        "$vectorSearch": {
            "index": "vector_index",
            "path": "embedding",
            "queryVector": query_embedding,
            "numCandidates": 100,
            "limit": 5,
            "filter": {"category": "finance"}
        }
    }
]

results = collection.aggregate(pipeline)

This single call combines approximate nearest neighbor search with a metadata filter — retrieving only the top financial documents most semantically relevant to the query, in one round trip.

Why the Combined Filtering Matters

As covered in the vector search performance post, integrating filters directly into the search process (rather than filtering after retrieval) avoids the problem of an aggressive filter leaving too few usable results among the initial top matches. MongoDB's approach to combining vector search with its existing query and filtering capabilities addresses this directly, since both are native operations within the same aggregation pipeline.

Trade-Offs Compared to a Dedicated Vector Database

Advantages: operational simplicity (one database instead of two), native combination of vector and traditional queries, and no need to keep a separate vector store in sync with your primary data.

Trade-offs: dedicated vector databases, purpose-built specifically for large-scale similarity search, can sometimes offer more specialized indexing options, tuning flexibility, or raw performance at very large scale. For applications with modest to moderate vector search needs already using MongoDB, the operational simplicity often outweighs that specialization; for applications with vector search as their primary, extremely high-scale workload, a dedicated vector database may still be worth the added operational complexity.

The Bottom Line

RAG with MongoDB works by storing document chunks and their embeddings together in the same collection as your other application data, using Atlas Vector Search to perform approximate nearest neighbor retrieval — often combined directly with metadata filtering — before feeding the retrieved context into an LLM. For teams already invested in MongoDB, it offers a genuinely practical way to add retrieval-augmented generation without introducing and maintaining an entirely separate specialized vector store, trading some of the specialization of dedicated vector databases for meaningful operational simplicity.