5 Word embeddings based ranking

This chapter explains how a search engine translates a user’s query into an ordered list of results and why ranking quality is central to perceived relevance. After queries are analyzed and matched in inverted indexes, a ranking function scores candidate documents so that the most relevant appear first—crucial because users rarely look beyond the first page. Beyond pure term matching, effective ranking can and should incorporate contextual signals such as user history, location, and recency, all of which can boost documents aligned with the user’s likely intent.

The chapter first reviews traditional retrieval models. Statistical models like TF-IDF within the Vector Space Model represent queries and documents as high-dimensional bag-of-words vectors and rank via cosine similarity; they are simple and strong baselines but ignore word order, grow with vocabulary size, and have limited semantic awareness. Probabilistic models such as Okapi BM25 dampen runaway term-frequency effects and improve length normalization, yet still operate as bag-of-words with heuristic approximations. In Lucene, these models are encapsulated by the Similarity API, enabling practical configuration and experimentation. The chapter then pivots to neural information retrieval, showing how dense word embeddings (e.g., word2vec) provide compact, semantically meaningful representations. By averaging word vectors to build query/document embeddings and comparing them with cosine similarity—and further “smoothing” with term frequency or TF-IDF weights to emphasize informative terms—one can correct failures of pure frequency-based ranking and better reflect intent.

To move from promising examples to robust practice, the chapter emphasizes evaluation. Using Lucene4IR and trec_eval on benchmark data, it compares TF-IDF, BM25, language models, and a custom WordEmbeddingsSimilarity that scores with averaged, smoothed embeddings. Results show that embeddings can slightly outperform classic baselines for top-k precision and NDCG, though gains depend on data and tuning; no single model dominates universally. A practical takeaway is to measure against clear objectives (precision, recall, NDCG) and consider hybrid ranking: combining multiple scoring functions (e.g., embeddings with BM25 or language models) often yields the best overall effectiveness. The chapter concludes that modern ranking benefits from mixing semantic embeddings with proven lexical signals, guided by systematic evaluation and real-world constraints.

Figure 5.1. Querying, retrieving and ranking

Figure 5.2. Cosine similarity

Figure 5.3. Word vectors for words bernhard, riemann and influence

Figure 5.4. Averaging bernhard, riemann and influence word vectors

Figure 5.5. Similarity between query and document embeddings

Figure 5.6. Averaged word vector smoothed by term frequencies

Figure 5.7. Averaged word vector smoothed by TF-IDF

• Classic retrieval models like VSM and BM25 provide good baseline for ranking documents but also lack semantic understanding of text capabilities
  
• Neural information retrieval models aim to provide better semantic understanding capabilities for ranking documents
  
• Distributed representations of words (like the ones generated by word2vec) can be combined together to generated document embeddings for queries and documents
  
• Averaged word embeddings can be used to generate effective Lucene Similarities which can achieve good results when evaluated against IR datasets
  

FAQ