Appendix B. LLM-Powered Summarization

This appendix extends the news reader from vector search into LLM-powered summarization. Instead of stopping at a ranked list of relevant articles, the system now feeds those retrieved results into a local language model to produce readable prose. The result is a /summary endpoint that can either digest the most recent articles or, when given a topic, summarize only the articles most semantically related to that topic.

The core architecture keeps retrieval and generation cleanly separated. Qdrant remains responsible for finding similar articles, while Ollama serves both the embedding model used for search and the generation model used for summaries. A new summarizer package wraps the LLM integration through a stable interface, and the project uses a retrieval-augmented generation flow so summaries are grounded in the feed’s actual content rather than general model knowledge. Prompt design, temperature, token limits, and timeouts are treated as important tuning points for balancing quality, speed, and reliability.

The API wiring is intentionally additive and modular. The summary handler accepts storage and search interfaces plus the summarizer, then chooses between a fast recent-articles path and a topic-based search-and-summarize path. The main server only gains configuration for the LLM model and a new route, while the rest of the application stays unchanged. The chapter closes by testing both modes end to end and encouraging further exploration through smaller models, prompt refinement, streaming, caching, and failure testing to better understand the trade-offs of local AI integration.

Figure B.1 RAG request pipeline: the two code paths through the system. When a topic is provided, the query is embedded by nomic-embed-text and Qdrant retrieves the most semantically similar articles; those articles are then sent to llama3 for synthesis into prose. Without a topic, the most recent articles are fetched directly from bbolt and passed to llama3, skipping the vector search step entirely.

Figure B.2 Component architecture showing how the packages depend on one another and the external services each talks to

• Retrieval-augmented generation (RAG) grounds LLM output in real data: the /summary?prompt=X endpoint uses vector search to find relevant articles first, then synthesizes them with the LLM, ensuring every claim in the summary comes from your actual feed.
• The Summarizer package wraps Ollama’s llms.Model interface via langchaingo. Swapping to a different LLM provider (OpenAI, Anthropic) requires changing only the initialization in New(); Summarize() and buildPrompt() are unchanged.
• Interface segregation carries forward from Appendix B: SummaryHandlers depends on domain.Storage for article retrieval and domain.Searchable for vector search, accepting them as separate parameters even though one concrete value satisfies both.
• Prompt engineering is the primary control surface for LLM behavior. The same model, same articles, and same code produce substantially different output depending on whether focusTopic is set. Adjusting the prompt is the first tool to reach for when output quality isn’t where you want it.
• Temperature and token limits are the primary dials for controlling LLM output. Temperature governs randomness (0.0 = deterministic, 1.0 = highly creative); token limits govern response length and generation time. Tuning both is a normal part of integrating an LLM, not a one-time setup step.
• Ollama serves both the embedding model (nomic-embed-text) and the generation model (llama3) from the same port. No additional infrastructure is required beyond what Appendix B already established.

FAQ