7 Document summarization and RAG with Langchain.js

This chapter puts LangChain.js and the Vercel AI SDK to work by building two production-minded web apps: a document summarizer and a Retrieval-Augmented Generation (RAG) system. You’ll see how Next.js, LangChain.js, and modern LLMs can be combined into pipelines that accept user inputs, process documents, and stream grounded, readable outputs back to the UI. Along the way, the chapter frames the practical challenges—context limits, accuracy, latency, and cost—and shows how grounding and careful prompt design help keep outputs faithful to source material. Familiarity with prior chapters on the Vercel AI SDK, prompt engineering, and LangChain.js is assumed.

The summarization app ingests PDFs and DOCX files, normalizes and cleans content, then splits text with semantic overlap to fit LLM context windows. It compares three summarization strategies—stuffing (best for short texts), refine (iterative improvements), and MapReduce (parallel chunk summaries combined into a final synthesis)—and wires them up with LangChain’s loadSummarizationChain. The chapter addresses real-world friction points: imperfect PDF parsing, noisy headers/footers, token costs, and long prompts. It introduces practical mitigations such as token-usage tracking via callbacks, prompt compression to shrink inputs, and k-means clustering over embeddings to merge semantically similar chunks before summarization, noting trade-offs between efficiency, speed, and potential information loss. A streamlined UI lets users paste text or upload files, while server actions orchestrate processing and streaming results.

The RAG app extends the pipeline to retrieve context from an indexed corpus before generation. Documents are chunked, embedded (e.g., with Google or OpenAI embeddings), and stored in an HNSWLib vector index for efficient semantic search. At query time, a retriever selects top matches, an augmentation layer composes a focused prompt, and an LLM produces the final answer—optionally returning source snippets for transparency. To bolster factuality, the chapter discusses grounding strategies, including citation and confidence-aware generation via per-token log probabilities when available; low-confidence answers can trigger fallbacks such as asking the user for clarification. The result is a responsive UI that can answer corpus-specific questions with higher accuracy, while calling out the operational realities of cost, rate limits, and context-window management.

The architecture of a document summarization system integrating LangChain.js and Vercel AI SDK. The user uploads a document which is processed by LangChain.js - it checks if the document exceeds the context window, splits it into chunks, and summarizes each chunk using LLMs. The individual summaries are then combined by the summarization (LLM) component which is displayed to the user through the UI.

The Map-Reduce summarization pattern, behind the scenes. The input document, which is divided into smaller chunks. Each chunk is then processed independently by the map function, generating intermediate summary pairs. These pairs are subsequently organized and are grouped together. The reduce step is then applied to these grouped pairs, aggregating the values to produce a final summary.

How the refine summarization method operates behind the scenes. This method enhances summarization quality by iteratively updating the summary with each new chunk of text. Initially, the first chunk is summarized, and for each subsequent chunk, it is concatenated to the existing summary. A new summary is generated based on this updated text, repeating the process until all chunks have been processed.

Demonstration of how the web application uses a MapReduce summarization technique to generate a concise summary from a large document through the UI. The application then sends the input to the backend service, which leverages the Langchain.js library to split the document into manageable chunks, process them through the summarization model, and return the final summary to be displayed in the user interface.

The user in this case can directly paste longform text that utilizes the same service to summarize it.

The process of process of compressing a prompt before sending it to a language model (LLM) using LLMLingua. The original prompt is compressed into a more concise version, which is then passed to the LLM for processing.

The sequential process of document clustering using k-means algorithm. The input text is first split into chunks using RecursiveCharacterTextSplitter, which preserves semantic context. These chunks are then transformed into vector embeddings to capture their semantic meaning. The k-means clustering algorithm groups similar chunks based on their vector representations. A distance-based comparison determines whether chunks should be merged or kept separate, ultimately producing semantically coherent sections.

Architecture of a RAG system. The system first indexes a collection of documents offline into a vector database, enabling efficient semantic search. When a user query is received, the system embeds the query, searches the vector database for relevant documents, and retrieves the most relevant contexts. These contexts are then used to augment the original query, creating an enriched prompt that is passed to a large language model to generate a final response.

Screenshot of an Astra AI conversation interface where the user asks questions regarding the indexed list of documents using RAG

RAG approach for grounding large language models (LLMs) with relevant information. The system performs a grounding search to extract the most relevant context, which is combined with the original user input to create an enhanced prompt. This prompt is then passed to the LLM, which generates the final grounded response that is provided to the user.

Comparison of AI responses with and without grounding context. The left panel shows the AI's response without grounding, while the right panel shows the AI's response when provided with additional contextual information to ground its understanding.

• Building a document summarization web application involves creating a user-friendly interface for uploading documents and generating concise summaries using advanced techniques.
• Implementing Retrieval-Augmented Generation (RAG) allows for dynamic information retrieval, enhancing the accuracy of responses by integrating real-time data with generated content.
• Developing reliable RAG web applications requires a thoughtful approach and combines a plethora of tools and techniques. Incorporating a grounding evaluation mechanism is a very useful step to ensure the system provides transparent and trustworthy outputs.
• Providing grounding support for AI outputs ensures that generated summaries are anchored to the source documents, improving reliability and reducing inaccuracies.
• Advanced document processing techniques, such as semantic chunking and k-means clustering, enhance context retention and summary quality in large documents.
• Handling various document formats, including PDFs and DOCX files, requires specialized loaders to extract text effectively for summarization.
• Utilizing the MapReduce method for summarization allows for parallel processing of document chunks, improving efficiency and enabling the handling of extensive texts.
• The RecursiveCharacterTextSplitter and TokenTextSplitter methods facilitate effective chunking of text, preserving context while managing LLM input limitations.
• The integration of LangChain.js with Vercel AI SDK streamlines the document processing pipeline, providing a seamless user experience during summarization tasks.
• Addressing challenges related to performance and user experience is crucial when implementing large-scale document summarization applications.
• Exploring different summarization strategies, such as the stuffing and refine methods, provides flexibility in how summaries are generated based on document complexity.

FAQ