3 Encoder models in action: Semantic-Based Retrieval Systems
This chapter deepens the focus on encoder models by situating them within the evolution of information retrieval, from early postwar ideas through web-scale keyword search to modern semantic systems. It defines the core functions of IR—indexing, storage, and retrieval—and illustrates their ubiquity in web, enterprise, e-discovery, and library search. Building on the transformer groundwork from earlier, the text frames semantic IR as a foundation for retrieval-augmented generation: first retrieving contextually relevant passages using neural methods, then feeding them to a generator to produce accurate, grounded responses. Encoders are highlighted as central not only to semantic retrieval but also to broader NLP applications such as machine translation and question answering.
The chapter revisits classic keyword search, explaining how inverted indexes enable fast lookups and how TF-IDF weights terms by their within-document frequency and across-corpus rarity to rank results. A step-by-step Python implementation shows tokenization, index construction, TF-IDF scoring, and ranking, demonstrating both the practicality and the limits of lexical matching. While efficient and scalable, keyword methods cannot capture meaning or intent; examples with overlapping terms can be ranked highly despite weak conceptual relevance, revealing why purely lexical techniques struggle with nuance, ambiguity, and user intent.
Semantic search addresses these gaps by encoding queries and documents into dense vectors and ranking via similarity (primarily cosine), often after L2 normalization to stabilize comparisons. Using Sentence Transformers (e.g., all-MiniLM-L6-v2), the chapter demonstrates producing 384-dimensional embeddings and performing semantic matching that recovers conceptually related content even without exact term overlap, yielding more graded and meaningful rankings than TF-IDF. It also introduces alternative similarity metrics, clarifies vector similarity with intuitive examples, and underscores how encoder-driven retrieval powers RAG to produce more accurate, context-aware answers. The chapter closes by noting challenges in language understanding, the practical advantages of semantic methods, and a preview of hybrid search that combines BM25 with embeddings for robust, real-world retrieval.
Recap from Chapter 1: Retrieval Augmentation Generation is used to enhance the capabilities of LLMs, especially in generating relevant and contextually appropriate responses.
Imagine a map where countries are placed based on how similar their names are. This plot is like that map but for words instead of countries. Words close together have similar meanings. For example, China is closer to Russia than France because their names are more related.
Simplified steps to follow in building semantic search.
A graphical demonstration of cosine similarity using two dimensions ("pet" and "furry"). Three words are drawn as vectors into the coordinate system. θ 1 denotes the angle between bird and dog, θ 2 depicts the angle between dog and cat, respectively. The smaller angle θ 2 illustrates a higher similarity of dogs with cats compared to birds.’
Summary
- An Information Retrieval (IR) system is a software system that stores and manages information (documents, web pages, articles, etc.) and provides interfaces for users to find and access the information they need.
- A very popular example of IR is Keyword search using inverted indices and Term Frequency Inverse Document Frequency (TF-IDF) scoring. It provides a scalable baseline for search relevancy. However, keyword matching is mostly based on word frequency and has limited semantic understanding beyond the exact phrases mentioned in the query.
- Recent advances in deep learning have enabled a more sophisticated for of search, called semantic search. The foundation of semantic search lies in the semantics, or meaning, of the words and phrases used in the search query.
- Using a pre-trained model framework like Sentence Transformers, we encode language into vector representations capturing meaning.
- Similarity metrics such as cosine similarity is a crucial metric in semantic search that measures the angle between two vectors - in this case, the vectors representing the query and the document. It effectively measures how closely related two pieces of text are, regardless of their size.
- Semantic search performs substantially better than keyword search in retrieval as it tries to achieve the context of the query and can go beyond surface form keyword matching to better model meaning. This allows discovering relevant content even without word-for-word query matches.
FAQ
What are the core components of an Information Retrieval (IR) system?
Indexing, Storage, and Search & Retrieval. Indexing analyzes documents and builds structures (like inverted indexes) for fast lookup. Storage keeps documents and indexes in scalable data stores. Search & Retrieval parses user queries, matches against the index, ranks results by relevance, and returns them quickly.How does a keyword search using an inverted index work?
An inverted index maps each term (token) to the list of document IDs that contain it. At query time, the system tokenizes the query, looks up each term’s postings list, merges the candidates, and ranks them. This avoids scanning all documents and enables fast keyword-based matching at scale.What is TF-IDF, and why is it useful for ranking keyword matches?
TF-IDF assigns a weight to a term within a document: TF reflects how frequently the term appears in that document (often normalized by length), while IDF down-weights terms that are common across the corpus. Multiplying TF and IDF highlights terms that are both prominent in a document and distinctive across documents, improving ranking quality.What are the main limitations of keyword-based search like TF-IDF?
- It relies on exact or near-exact term overlap and ignores context and meaning.- It can rank documents highly that share words but aren’t conceptually relevant.
- It struggles with synonyms, paraphrases, and nuanced intent.
What is semantic search and how does it differ from keyword search?
Semantic search encodes text into dense vectors that capture meaning, then retrieves by vector similarity rather than exact token overlap. It can recognize related concepts (e.g., “dog,” “canine,” “puppy”) and understand context, often returning more relevant results even without keyword matches.What is Retrieval-Augmented Generation (RAG) and how does it work?
RAG combines neural retrieval with language generation in two stages: (1) retrieve semantically relevant passages for a query, and (2) feed them to a generator (e.g., an LLM) to synthesize an answer. Encoders power retrieval; the generator crafts a coherent, context-aware response from both retrieved text and its own knowledge.How do Sentence-Transformers create sentence embeddings, and what roles do pooling and normalization play?
A Transformer encoder (e.g., MiniLM/BERT) produces token embeddings; pooling (often mean pooling) aggregates tokens into a single sentence vector; L2 normalization scales vectors to unit length to make similarity focus on direction (semantics) rather than magnitude, improving retrieval robustness.Why is cosine similarity commonly used for semantic search, and how does it compare to other metrics?
Cosine similarity measures the angle between vectors, emphasizing direction (meaning) over length—ideal for L2-normalized embeddings. Alternatives include Euclidean and Manhattan distances (less reliable in high dimensions) and Jaccard (set-based, more suited to sparse features than dense embeddings).How can I implement a simple semantic search pipeline with Sentence-Transformers?
- Install libraries (sentence-transformers, scipy, pandas).- Load a pretrained encoder (e.g., all-MiniLM-L6-v2).
- Encode your corpus into embeddings (one vector per document).
- Encode the query into an embedding.
- Compute cosine similarity between the query vector and corpus vectors.
- Sort by similarity and return the top results.
When should I use keyword search, semantic search, or a hybrid approach?
- Keyword search: precise term matching, boolean filters, and speed on well-defined vocabularies.- Semantic search: intent understanding, synonyms/paraphrases, and concept-level retrieval.
- Hybrid: combine both to leverage exact matches and semantic relevance, often yielding the most robust performance across varied queries.
Build an Advanced RAG Application (From Scratch) ebook for free
