The Landscape in 2026
The AI orchestration space has matured significantly. LangChain is no longer the disorganized framework it was in 2023, the v0.3 rewrite with LCEL brought real composability. Semantic Kernel hit 1.0 and became Microsoft's official SDK for AI orchestration. LlamaIndex carved out a strong niche as the go-to framework for retrieval-augmented generation.
But they overlap significantly. All three can build a RAG pipeline. All three support tool calling. All three connect to vector stores. The differences are in philosophy, developer experience, and where they're strongest.
Architecture Comparison
LangChain: The Swiss Army Knife
LangChain's core abstraction is the chain, a sequence of operations piped together. With LCEL, this becomes genuinely elegant:
# langchain_rag.py - Modern LangChain RAG pipeline
from langchain_anthropic import ChatAnthropic
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
# 1. Load and split documents
loader = DirectoryLoader(
'./legal_docs/',
glob="**/*.pdf",
loader_cls=PyPDFLoader,
show_progress=True
)
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(docs)
# 2. Create vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
chunks,
embeddings,
collection_name="legal_docs",
persist_directory="./chroma_db"
)
retriever = vectorstore.as_retriever(
search_type="mmr",
search_kwargs={"k": 6, "fetch_k": 20, "lambda_mult": 0.7}
)
# 3. Define the RAG chain with LCEL
llm = ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", """You are a legal research assistant. Answer questions based
strictly on the provided context. If the context doesn't contain enough
information, say so explicitly. Cite specific document sections.
Context:
{context}"""),
("human", "{question}")
])
def format_docs(docs):
return "\n\n---\n\n".join(
f"[{doc.metadata.get('source', 'unknown')}] {doc.page_content}"
for doc in docs
)
# The LCEL pipeline - clean, composable, debuggable
rag_chain = (
RunnableParallel(
context=retriever | format_docs,
question=RunnablePassthrough()
)
| prompt
| llm
| StrOutputParser()
)
# Usage
response = rag_chain.invoke("What are the liability limitations in section 4.2?")
# Streaming - works natively with LCEL
for chunk in rag_chain.stream("Summarize the indemnification clauses"):
print(chunk, end="", flush=True)
The strength of LCEL is the pipe operator. retriever | format_docs reads exactly like what it does. Debugging is straightforward because you can invoke any step independently. The RunnableParallel runs the retrieval and question passthrough concurrently.
The downside: the ecosystem is enormous and confusing. There are langchain, langchain-core, langchain-community, langchain-openai, langchain-anthropic, figuring out which package has which class is still a scavenger hunt.
Semantic Kernel: The Enterprise Player
Semantic Kernel thinks in terms of plugins and planners. A plugin is a collection of functions that the AI can call. A planner decides which plugins to invoke and in what order.
# semantic_kernel_rag.py
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import (
AzureChatCompletion,
AzureTextEmbedding
)
from semantic_kernel.connectors.memory.chroma import ChromaMemoryStore
from semantic_kernel.core_plugins.text_memory_plugin import TextMemoryPlugin
from semantic_kernel.functions import kernel_function
from semantic_kernel.memory import SemanticTextMemory
# 1. Initialize the kernel
kernel = sk.Kernel()
# Add AI services
kernel.add_service(AzureChatCompletion(
service_id="chat",
deployment_name="gpt-4o",
endpoint="https://myinstance.openai.azure.com/",
api_key="your-key"
))
embedding_service = AzureTextEmbedding(
service_id="embedding",
deployment_name="text-embedding-3-small",
endpoint="https://myinstance.openai.azure.com/",
api_key="your-key"
)
# 2. Set up memory
memory_store = ChromaMemoryStore(persist_directory="./chroma_sk")
memory = SemanticTextMemory(
storage=memory_store,
embeddings_generator=embedding_service
)
kernel.add_plugin(TextMemoryPlugin(memory), "memory")
# 3. Create a custom RAG plugin
class LegalResearchPlugin:
def __init__(self, memory: SemanticTextMemory):
self._memory = memory
@kernel_function(
name="search_legal_docs",
description="Search legal documents for relevant clauses and sections. "
"Use this when the user asks about contracts, clauses, "
"liability, indemnification, or any legal topic."
)
async def search_legal_docs(
self,
query: str,
collection: str = "legal_docs"
) -> str:
"""Search the legal document corpus and return relevant passages."""
results = await self._memory.search(
collection=collection,
query=query,
limit=6,
min_relevance_score=0.7
)
if not results:
return "No relevant documents found for this query."
passages = []
for i, result in enumerate(results):
source = result.metadata.get("source", "unknown")
passages.append(
f"[Source: {source}, Relevance: {result.relevance:.2f}]\n"
f"{result.text}"
)
return "\n\n---\n\n".join(passages)
@kernel_function(
name="analyze_clause",
description="Analyze a specific legal clause for risks and obligations. "
"Use after retrieving relevant documents."
)
async def analyze_clause(self, clause_text: str) -> str:
return f"Clause for analysis:\n{clause_text}"
# Register plugin
legal_plugin = LegalResearchPlugin(memory)
kernel.add_plugin(legal_plugin, "legal_research")
# 4. Create the chat function with auto function calling
execution_settings = sk.PromptExecutionSettings(
service_id="chat",
function_choice_behavior="auto", # Let the LLM decide which functions to call
temperature=0
)
# The kernel handles the function calling loop automatically
chat_function = kernel.add_function(
plugin_name="chat",
function_name="legal_assistant",
prompt="""You are a legal research assistant with access to a document corpus.
Use the legal_research plugin to find relevant information before answering.
User question: {{$input}}
Provide a thorough answer with citations.""",
prompt_execution_settings=execution_settings
)
# Usage
async def main():
response = await kernel.invoke(
chat_function,
input="What are the liability limitations in section 4.2?"
)
print(response)
The function_choice_behavior="auto" setting is key. The LLM sees all registered plugins with their descriptions and decides which to call. The @kernel_function decorator with a good description is essential, the LLM reads it to decide when to use the function.
Semantic Kernel's strength is in structured enterprise scenarios. The plugin model maps well to organizational boundaries, the legal team owns the legal plugin, the finance team owns the finance plugin. It's also the best option if you're deploying on Azure with managed identities and Azure OpenAI.
LlamaIndex: The RAG Specialist
LlamaIndex was purpose-built for retrieval-augmented generation, and it shows:
# llamaindex_rag.py
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
Settings,
StorageContext,
load_index_from_storage,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import (
SimilarityPostprocessor,
LongContextReorder
)
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.response_synthesizers import get_response_synthesizer
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
# 1. Configure global settings
Settings.llm = Anthropic(model="claude-sonnet-4-20250514", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(
chunk_size=512,
chunk_overlap=50
)
# 2. Load documents with metadata extraction
documents = SimpleDirectoryReader(
input_dir="./legal_docs/",
required_exts=[".pdf", ".docx"],
filename_as_id=True,
).load_data()
# Add custom metadata
for doc in documents:
doc.metadata["doc_type"] = "legal_contract"
doc.metadata["jurisdiction"] = "US"
# 3. Create vector store index
chroma_client = chromadb.PersistentClient(path="./chroma_li")
chroma_collection = chroma_client.get_or_create_collection("legal_docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
show_progress=True
)
# 4. Build a sophisticated query engine
retriever = VectorIndexRetriever(
index=index,
similarity_top_k=10, # Retrieve more, then rerank
)
# Post-processing pipeline
node_postprocessors = [
SimilarityPostprocessor(similarity_cutoff=0.7),
LongContextReorder(), # Put most relevant at start and end
]
response_synthesizer = get_response_synthesizer(
response_mode="tree_summarize", # Hierarchical summarization
use_async=True,
)
query_engine = RetrieverQueryEngine(
retriever=retriever,
node_postprocessors=node_postprocessors,
response_synthesizer=response_synthesizer,
)
# 5. Query with source tracking
response = query_engine.query(
"What are the liability limitations in section 4.2?"
)
print(response.response)
print("\n--- Sources ---")
for node in response.source_nodes:
print(f"Score: {node.score:.3f} | Source: {node.metadata.get('file_name')}")
print(f"Text: {node.text[:200]}...")
print()
LlamaIndex's pipeline is more opinionated about retrieval. The LongContextReorder postprocessor is a perfect example, it reorders retrieved nodes so the most relevant are at the beginning and end of the context, based on research showing LLMs pay more attention to those positions. You'd have to build this yourself in LangChain.
The tree_summarize response mode is another strength. Instead of stuffing all retrieved chunks into one prompt, it builds a tree of summaries, summarizing groups of chunks, then summarizing the summaries. This handles cases where you retrieve 20 relevant chunks that wouldn't fit in a single context window.
Memory Management Compared
Memory is where the frameworks diverge most:
# LangChain - Explicit memory objects
from langchain.memory import ConversationSummaryBufferMemory
memory = ConversationSummaryBufferMemory(
llm=llm,
max_token_limit=2000,
return_messages=True
)
# Keeps recent messages verbatim, summarizes older ones
# Semantic Kernel - ChatHistory with manual control
from semantic_kernel.contents import ChatHistory
history = ChatHistory()
history.add_system_message("You are a legal assistant.")
history.add_user_message("What's in section 4.2?")
history.add_assistant_message("Section 4.2 covers...")
# You control exactly what goes in and what gets pruned
# LlamaIndex - Chat engine with built-in memory
from llama_index.core.memory import ChatMemoryBuffer
memory = ChatMemoryBuffer.from_defaults(token_limit=3000)
chat_engine = index.as_chat_engine(
chat_mode="condense_plus_context",
memory=memory,
system_prompt="You are a legal research assistant."
)
# Automatically condenses conversation history into a query
response = chat_engine.chat("What about section 4.2?")
response = chat_engine.chat("And what about indemnification?")
# ^ This automatically uses conversation context
LangChain gives you the most memory strategies but requires manual wiring. Semantic Kernel gives you the most control but the least magic. LlamaIndex's condense_plus_context mode is the most turnkey, it automatically reformulates follow-up questions using conversation history, which is exactly what you need for a chat-based RAG system.
Verdict After Building Three Systems
LlamaIndex is the strongest choice for legal document search platforms. The document processing pipeline, metadata extraction, and retrieval postprocessors save weeks of custom development. If your primary job is "search a corpus and answer questions," start here.
LangChain is the strongest choice for customer support automation. Complex agent behaviors are often needed, routing between human agents and AI, calling external APIs for order status, switching between different LLM providers based on query complexity. LangChain's LCEL made these compositions clean and debuggable.
Semantic Kernel is the strongest choice for internal knowledge bases in Microsoft-heavy enterprises. Azure AD integration, managed identity authentication to Azure OpenAI, and the plugin model that mapped to their organizational structure. The C# SDK is also a factor when the majority of the backend is .NET.
None of these frameworks is bad. They've all matured enormously. But they have different strengths, and pretending they're interchangeable does everyone a disservice. Pick the one that matches your primary use case, not the one with the most GitHub stars.