LangChain vs LlamaIndex for RAG Pipelines

LangChain and LlamaIndex can both build a production RAG pipeline. They start from different places. LlamaIndex is a data framework: its best parts are about getting documents in, split well and retrieved accurately. LangChain is an orchestration framework: its best parts are about chaining models, tools and steps, and with LangGraph, running agents that keep state.

So the short answer is:

• If the hard part of your project is retrieval quality over messy documents, start with LlamaIndex.
• If the hard part is what happens around retrieval (tools, multi-step logic, approvals, an agent that does more than answer questions), start with LangChain and LangGraph.
• If both are hard, use both. They work well together.

The rest of this guide shows why, with the same pipeline built in each, a component-by-component comparison and a recommendation for each common use case.

How each framework thinks about RAG#

A RAG pipeline has two halves:

1. Ingestion: load documents, parse them, split them into chunks, embed the chunks and store them in a vector database.
2. Query: take a question, retrieve the relevant chunks, optionally rerank them, and have a model write an answer from them with sources.

LlamaIndex is built around this pipeline. Its core objects are the stages themselves: readers, node parsers, indexes, retrievers, postprocessors, response synthesizers and query engines. Defaults are chosen for you, so a working pipeline takes a few lines, and each stage can be swapped out when the defaults stop being good enough.

LangChain gives you the building blocks (loaders, splitters, embeddings, vector stores, retrievers) as interchangeable integrations, and leaves the shape of the pipeline to you. Since LangChain 1.0, the recommended way to put RAG in front of users is an agent built with create_agent, with retrieval exposed as a tool. Agents run on LangGraph, which handles state, memory, streaming and human-in-the-loop steps.

The example pipeline#

An internal assistant that answers employee questions from a folder of HR handbook files, and cites the file each answer came from.

Built in LlamaIndex#

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-5-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Ingestion: load, split, embed and index
documents = SimpleDirectoryReader("./handbook", recursive=True).load_data()
index = VectorStoreIndex.from_documents(
    documents,
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=64)],
)
index.storage_context.persist("./storage")  # or pass a vector store such as Qdrant or pgvector

# Query: retrieve 10 candidates, rerank to the best 3, then answer
query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[
        SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=3),
    ],
)

response = query_engine.query("How many days of parental leave do we offer?")
print(response)
for node in response.source_nodes:
    print(node.metadata["file_name"], round(node.score, 2))

Reranking, source tracking and the answer prompt are all built in. To make it a multi-turn chat, swap as_query_engine() for as_chat_engine(chat_mode="condense_plus_context").

Built in LangChain#

from langchain.agents import create_agent
from langchain.tools import tool
from langchain_chroma import Chroma
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Ingestion: load, split, embed and store
docs = DirectoryLoader("./handbook", glob="**/*.md", loader_cls=TextLoader).load()
chunks = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=200).split_documents(docs)
vector_store = Chroma.from_documents(
    chunks,
    OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma",
)

# Query: retrieval is a tool the agent decides when to call
@tool(response_format="content_and_artifact")
def search_handbook(query: str):
    """Search the employee handbook for policies and procedures."""
    results = vector_store.similarity_search(query, k=4)
    text = "\n\n".join(f"Source: {doc.metadata['source']}\n{doc.page_content}" for doc in results)
    return text, results

agent = create_agent(
    model="openai:gpt-5-mini",
    tools=[search_handbook],
    system_prompt=(
        "You answer questions about the employee handbook. Always search it first "
        "and cite the source file. If the handbook doesn't cover it, say so."
    ),
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "How many days of parental leave do we offer?"}]}
)
print(result["messages"][-1].content)

There is a bit more code, but the shape is different in a useful way: the agent can call search_handbook more than once, rephrase its query, or skip retrieval for "hello". Adding a second tool, such as get_leave_balance(employee_id), is one more function in the tools list. If you want retrieval to always run exactly once, a two-step chain (retrieve, then prompt the model) is simpler and cheaper than an agent.

One detail that catches people moving between the two: LlamaIndex's SentenceSplitter measures chunks in tokens, LangChain's RecursiveCharacterTextSplitter measures in characters unless you build it with from_tiktoken_encoder. That's why the chunk sizes above differ.

Prebuilt components compared#

Note that in LangChain 1.0, several older chains and retriever helpers moved to the langchain-classic package. They still work, but newer tutorials build the same things with agents and plain functions, so check the date on any example you copy.

Advanced retrieval techniques out of the box#

These are the techniques that usually separate a demo from a system people trust. Both frameworks cover most of them. The difference is how much you write yourself.

In general, LlamaIndex has a named component for each technique, and LangChain gives you the pieces to build it. Neither choice fixes bad source documents: clean parsing and sensible chunking matter more than any retriever on this list.

Head-to-head comparison#

Which to choose for each use case#

Using both together#

The two frameworks aren't mutually exclusive, and a common production setup uses each for what it does best: LlamaIndex for ingestion and retrieval, LangChain or LangGraph for the agent.

from langchain.agents import create_agent
from langchain.tools import tool

# Built with LlamaIndex as in the first example: parsing, chunking, hybrid search, reranking
query_engine = index.as_query_engine(similarity_top_k=10, node_postprocessors=[reranker])

@tool
def search_policies(question: str) -> str:
    """Answer a question from the company policy documents, with sources."""
    response = query_engine.query(question)
    sources = ", ".join({node.metadata["file_name"] for node in response.source_nodes})
    return f"{response}\n\nSources: {sources}"

agent = create_agent(
    model="openai:gpt-5-mini",
    tools=[search_policies, get_leave_balance, create_hr_ticket],
    system_prompt="You are the HR assistant...",
)

The retrieval layer can be tuned and evaluated on its own, and the agent can change without touching it.

Mistakes to avoid with either framework#

• Choosing the framework before checking the documents. Scanned PDFs, tables and slide decks need a parsing plan first. The framework comes second.
• Skipping evaluation. Write 30 to 50 real questions with known answers before tuning anything. Without them, every change to chunking or retrieval is a guess.
• Copying old tutorials. Both frameworks have changed a lot. Check that an example matches the version you've installed.
• Using an agent where a pipeline will do. If every question needs exactly one retrieval, a fixed pipeline is faster, cheaper and easier to debug than an agent deciding when to search.
• Over-abstracting a simple system. If you only need embed, search and answer, a hundred lines of plain Python may be the most maintainable option.

For what a RAG system costs to build and run once you've chosen the stack, see how much RAG development costs.

If you're planning a RAG system and aren't sure which way to build it, tell me about your documents and users. The first step of every RAG project I take on is a short evaluation on your real documents, and the framework choice follows from what it shows.