This chapter moves beyond averaging word vectors and focuses on learning document embeddings directly so search systems can compare texts by meaning rather than surface term overlap. It introduces paragraph vectors as an extension of word2vec that associates each sentence, paragraph, or document with a learned label and dense vector, enabling semantic representations at multiple granularities. With these embeddings, the model can return vectors for known documents or infer vectors for unseen text, powering tasks like ranking and related-content retrieval where semantic similarity matters more than exact keyword matches.
For ranking, the workflow is straightforward: train paragraph vectors over the indexed corpus using document identifiers as labels, obtain a vector for the query (reusing a trained label or inferring a new one), fetch vectors for candidate documents, and re-score with cosine similarity. This can be embedded in a custom Similarity to integrate with a standard retrieval pipeline. The chapter contrasts this with traditional “related content” based on TF-IDF term extraction (e.g., MoreLikeThis), showing how frequency-driven terms can miss the true topic and yield off-topic suggestions even when swapping ranking functions. Using Wikipedia categories as a proxy for relevance, such term-based relatedness achieves low accuracy, highlighting the limits of frequency-only signals.
Embedding-driven related content performs much better: nearest neighbors in the paragraph-vector space tend to be topically coherent, and measured accuracy improves substantially over TF-IDF baselines. The chapter further explores sequence-to-sequence encoder–decoder models, harvesting the encoder’s “thought vectors” as document embeddings; by supervising training with pairs from the same category, these vectors capture category-level semantics and boost related-content accuracy even more. Overall, document embeddings offer a practical bridge between semantic understanding and search, improving ranking quality and recommendation relevance while allowing flexible training choices—from unsupervised paragraph vectors to supervised thought vectors—across different text granularities.
Figure 6.1. Word2vec continuous bag of words model
Figure 6.2. Distributed memory model from paragraph vectors
Figure 6.3. Distributed bag of words model from paragraph vectors
Figure 6.4. Using neural networks to retrieve related content
Figure 6.5. Retrieving related content by using a document’s most important terms, weighted by TF-IDF scheme
Figure 6.6. Building a taxonomy out of Wikipedia categories
Figure 6.7. Navigating relations between "Ledgewood Circle" and "New Jersey" pages in DBPedia
Figure 6.8. Paragraph vectors for Ledgewood Circle and its neighbours, compared with music related paragraph vectors
Figure 6.9. Encoder Decoder model
Summary
- Paragraph vectors models provide distributed representations for sentences and documents at configurable granularity (sentence, paragraph, document)
- Ranking functions based on paragraph vectors can be more effective than "old school" statistical models and word embeddings based ones.
- Paragraph vectors can be used to effectively retrieve related content based on document semantics
- Thought vectors can be extracted from seq2seq models to retrieve related content based on document semantics
FAQ
What are document embeddings and why use them in search?
Document embeddings are dense vectors that represent the meaning of an entire text span (sentence, paragraph, or document). Unlike TF-IDF/BM25, which rely on individual term frequencies, embeddings capture context and semantics, enabling better ranking, semantic matching, and “related content” retrieval when exact term overlap is weak or noisy.How do Paragraph Vectors (PV-DM and PV-DBOW) extend word2vec?
Paragraph Vectors add a document label into the learning objective so the model learns a vector per label (document/paragraph/sentence). PV-DM (Distributed Memory) augments the CBOW context with a doc label to predict a target word. PV-DBOW (Distributed Bag of Words) inputs only the doc label and predicts words from that document. Both produce document-level vectors that capture semantics.What granularity should I label: document, paragraph, or sentence?
Choose the granularity that matches your task and text length. Short pieces (titles, short articles) can use a single label. Long pages (e.g., Wikipedia) often benefit from labeling smaller units (paragraphs/sentences) if you need fine-grained similarity (e.g., “find similar sentences”).How do I train ParagraphVectors from a Lucene index?
- Implement a LabelAwareIterator that streams each Lucene document’s content and assigns a stable label (e.g., “doc_1234”).- Build and fit the model (e.g., set layerSize, minWordFrequency, tokenizer).
- After training, each label has a learned embedding you can retrieve via the lookup table.
How do I use Paragraph Vectors for ranking queries and documents?
- Get a query vector: look it up if already trained; otherwise, infer it (inferVector) for unseen text.- Fetch each candidate document’s vector by its label (e.g., “doc_1234”).
- Compute cosine similarity between query and document vectors and use it to re-rank the initial results (or implement a custom Similarity that returns this score).
How does embedding-based related content differ from Lucene MoreLikeThis (MLT)?
MLT extracts “important” terms via TF-IDF and re-queries the index, which can be skewed by frequent yet uninformative terms. Embedding-based retrieval finds nearest document vectors (e.g., with cosine similarity or nearestLabels), which tends to surface semantically coherent items. In the chapter’s experiments, Paragraph Vectors achieved much higher related-content accuracy than MLT.How can I evaluate the quality of related content suggestions?
- Use category overlap: a suggested item is “relevant” if it shares at least one category with the source document; average this over many queries.- Improve the metric by using hierarchical category relations (taxonomy) and external knowledge (e.g., DBPedia links).
- When categories are missing/noisy, unsupervised clustering of embeddings can provide a proxy for topical relatedness.
Deep Learning for Search ebook for free
