2 Revolutions in semantics, scale, and similarity

RAG is presented as an architectural pattern rather than a single technology, and this chapter explains how three earlier innovations converged to make it possible: Word2Vec for semantic representation, FAISS for scalable similarity search, and Sentence-BERT for practical sentence-level retrieval. Together, they addressed the biggest barriers in information retrieval: vocabulary mismatch, computational scale, and the difficulty of turning meaning into a usable ranking signal. The chapter emphasizes that modern RAG systems inherit both the strengths and the limitations of these foundations, so understanding them is essential for diagnosing retrieval errors and improving system behavior.

Word2Vec marked the shift from exact keyword matching to geometric meaning by learning dense word vectors from context, operationalizing the idea that words with similar usage have similar semantics. This made it possible to connect terms like “car” and “automobile,” which traditional methods such as TF-IDF and BM25 treated as unrelated. However, Word2Vec remained limited to static word meanings and could not represent full sentences or adapt to context-dependent ambiguity, even though its efficiency and scalability made large-scale semantic modeling practical.

FAISS solved the next bottleneck by making billion-scale approximate nearest neighbor search fast enough for production, while Sentence-BERT adapted transformer models into a bi-encoder format that could precompute embeddings and compare them efficiently. SBERT turned BERT’s contextual understanding into sentence vectors suitable for indexing, enabling real-time semantic search across large document collections. The chapter concludes that the combination of semantic embeddings, efficient vector search, and hybrid retrieval laid the groundwork for RAG, and that the next step was to connect retrieval with generation so systems could not only find relevant knowledge but also synthesize answers from it.

CBOW vs skip-gram architectures.

What cosine similarity measures and discards. Top left: the query and doc A sit at angle θ, and doc B, twice as long and pointing the same way, scores identically, because normalizing removes magnitude before the dot product. The other three panels draw the values listed above: identical direction scores 1.0, right angles score 0.0, and opposite directions score −1.0, which text embeddings almost never produce.

FAISS's Two-Stage Search Strategy.

BERT vs Sentence-BERT: Architecture comparison.

• Word2Vec (2013) popularized the concept of dense vector embeddings, showing that semantic relationships can be learned from data and captured in mathematical representations. This innovation enabled systems to understand that "automobile maintenance" and "car repair" refer to the same concept. While previous methods like LSA existed, Word2Vec made it cheap enough to train on billions of words, which is what made semantic search practical.
• FAISS (2017) turned vector search into production infrastructure. While algorithms like Locality-Sensitive Hashing (LSH) had existed for decades, FAISS optimized them for modern hardware (GPUs) and billion-scale datasets. Its unified API for indexing strategies (Flat, IVF, HNSW) became an industry standard for deploying semantic search at scale.
• Sentence-BERT (2019), transformers adapted for retrieval, solved the bottleneck of encoding full sentences. Using Siamese networks, it allowed transformers to generate independent sentence embeddings that could be precomputed and indexed. This reduced retrieval time from hours to milliseconds compared to raw BERT cross-encoders, finally enabling real-time semantic ranking.
• The convergence created a working technology stack: Python development environments, vector databases, embedding models, approximate nearest neighbor algorithms, and cloud infrastructure. By 2020, the pieces were in place. We had a way to represent meaning (embeddings), a way to search it at scale (FAISS), and a way to encode complex queries (SBERT). All that remained was to combine this retrieval stack with the generative power of the emerging large language models.

FAQ