4 More sensitive suggestions
This chapter explores how to help users compose better queries through smarter autocomplete, moving beyond backend synonym expansion toward interactive, context-aware suggestions. It motivates suggestions as a way to reduce zero-result queries and improve relevance, while also giving search systems a lever to guide user intent. The chapter frames where suggestions can come from—static dictionaries, query logs, and indexed content—and sets goals such as token-level and infix completion, generating natural-sounding unseen sequences, aligning suggestions with available results, and assisting disambiguation.
The text first examines traditional approaches with Lucene: dictionary-based lookups (e.g., ternary search trees) that are fast but limited to strict prefixes and full strings; analyzed suggesters that apply tokenization, stopword removal, and finite state transducers to support richer prefix and infix matches; and language-model-based suggesters (ngrams with “stupid backoff”) that can always return something and even synthesize unseen phrases, yet struggle with long-range coherence and may produce awkward continuations. To keep suggestions faithful to what can be retrieved—and to allow business-driven weighting—the chapter introduces content-based suggesters that draw from index fields (titles, authors, etc.) and leverage per-suggestion weights (such as price or ratings).
It then develops neural language models for suggestions, implementing a character-level LSTM as a Lucene Lookup fed directly from indexed content. The chapter details training and inference (sampling), highlights early issues (fragmented or nonsensical outputs), and shows how tuning learning rate, network depth/size, and truncated BPTT—monitored via a training UI—improves fluency and stability while avoiding overfitting. Although character models can generate fluid, human-like completions, they require careful setup and sufficient data; quality further benefits from combining LSTM outputs with word vectors (word2vec) to diversify and semantically steer suggestions (e.g., handling synonyms like aircraft/aeroplane). The overall takeaway is a practical recipe: blend IR-driven sources and analyzers with neural language models to produce accurate, natural, and context-sensitive autocomplete that reflects the corpus and user needs.
Figure 4.1. Suggestions for "book"
Figure 4.2. Suggestions for "books about sear"
Figure 4.3. Suggestions for "books about search engines and"
Figure 4.4. Suggestions for "books about search engines and dee"
Figure 4.5. A finite state transducer
Figure 4.6. RNN for sequence learning
Figure 4.7. DL4J Training UI
Figure 4.8. Char LSTM Neural language model with 2 hidden layers (300 neurons each)
Figure 4.9. Char LSTM Neural language model with 1 hidden layer (100 neurons each)
Figure 4.10. More parameters but still slow convergence
Figure 4.11. More parameters but still slow convergence
Figure 4.12. Model view in DL4J Training UI
Summary
- Search suggestions are very important in order to help users write good queries
- The data for generating such suggestions can be both static (e.g. dictionaries of previously entered queries) or dynamic (e.g. the documents stored in the search engine)
- We can use text analysis and / or ngram language models to build good suggester algorithms
- Neural language models are language models based on neural networks, like RNNs (or LSTMs)
- By using neural language models we can get better sounding suggestions
- It’s important to monitor the neural network training process in order to make sure that we’ll get good results
- We can combine the results of original suggester with word vectors to augment the diversity of the suggestions
FAQ
Why do autocomplete and suggestions matter in a search engine?
They reduce zero-result queries and low-relevance queries, speed up typing, help users avoid typos, and let the engine “hint” better terms or full queries. They also allow ranking and biasing suggestions (for relevance, popularity, or business goals).What data sources can suggestions be built from?
- Static dictionaries (hand-crafted terms or phrases)- Query logs (previously entered queries, often frequency-weighted)
- Indexed content (titles, authors, fields from the search index)
Mixing these sources lets you keep suggestions fresh and aligned with what the engine can actually return.
How do dictionary-based suggesters in Lucene work, and what are their limits?
They build a Lookup from a dictionary (for example, past queries) and return strings that start with the user’s prefix. They’re fast and simple but can’t split or analyze text, won’t suggest mid-query (infix) matches, and don’t compose unseen sequences.What is a ternary search tree (trie) and why is it used?
A TST is a prefix tree where each node represents a character with up to three children. It excels at prefix lookups, making it well-suited for classic autocompletion where suggestions must start with the typed prefix.How do analyzed suggesters (AnalyzingSuggester and AnalyzingInfixSuggester) improve results?
They apply text analysis at build and lookup time (tokenization, stopword removal, etc.), compile an efficient model (FST), and support richer behaviors like infix matches and token-based suggestions, not just whole-string prefixes.What is a finite state transducer (FST) in this context?
An FST is a graph whose edges carry characters and optional weights. At build time, analyzed suggestion terms are compiled into a single FST. At lookup time, traversing the FST with the analyzed user input yields fast, weighted suggestions.How do n-gram language models (e.g., FreeTextSuggester) help with suggestions?
They estimate probabilities of token sequences (ngrams) and can generate plausible, unseen continuations. Lucene’s FreeTextSuggester uses “stupid backoff” to back off from higher-order to lower-order ngrams. Pros: constant suggestions and novel combinations. Cons: weaker for long sentences and can produce odd completions.What are content-based suggesters and how do I build them from the index?
They read suggestion values and optional weights directly from the search index (for example, book titles weighted by price or ratings) using DocumentDictionary + IndexReader. This keeps suggestions aligned with current inventory and business priorities.What is a neural language model for suggestions and how is a char-level LSTM used?
A neural LM predicts next-character probabilities given prior characters. A char-level LSTM (e.g., CharLSTMNeuralLookup) is trained on indexed text (like lyrics) and, at runtime, samples likely continuations from the model’s SoftMax output. It can produce fluent, human-like suggestions when well tuned.How do I tune and evaluate an LSTM suggester, and how can I diversify suggestions?
- Tuning: Adjust learning rate, hidden size, number of layers, and sequence lengths; avoid over/underfitting. Use DL4J’s Training UI (StatsListener) to monitor loss trending down.- Diversification: Combine neural suggestions with word vectors (word2vec) to inject semantic neighbors and varied, yet relevant, alternatives.
Deep Learning for Search ebook for free
