This chapter shifts from simple synonym expansion to teaching a search engine to generate alternative phrasings of a user’s intent, thereby bridging the gap between terse queries and richer information needs. The core idea is automatic query expansion: keep the user’s original query but add machine-generated alternatives to reduce zero-result queries, raise recall by surfacing missed documents, and improve precision by boosting results that match both. To power this, the chapter motivates recurrent neural networks (RNNs)—especially Long Short-Term Memory (LSTM) models—for their strength on sequences, and frames the problem around learning to express the same intent in multiple ways without user intervention.
A substantial portion focuses on data preparation strategies, because what the model learns depends on how inputs and targets are paired. Options include: correlating queries that share results, grouping queries from the same users within time windows, linking queries via shared or expanded terms (with careful synonym use), and even deriving targets from indexed document titles (e.g., mapping a keyword query to a relevant title). When labeled pairs are scarce, an unsupervised approach treats the query log as a simple text stream (one query per line). The chapter then builds intuition for RNNs—unrolling over time, backpropagation through time, and the need for LSTMs to capture long-term dependencies. Practical modeling choices are outlined (character vs word inputs, softmax outputs with cross-entropy loss, and the impact of unroll length, hidden size, and depth).
Implementation proceeds in two tracks. First, unsupervised, character-level LSTMs are used to learn next-character prediction (validated on literary text) and then applied to query logs to generate alternatives; integration is shown via a Lucene QueryParser that runs both the original and sampled alternatives, with sampling tricks (e.g., injecting a line break) to elicit wholly new queries rather than mere completions. Second, a supervised sequence-to-sequence setup encodes an input query and decodes a related output query at the word level, yielding a “thought vector” that captures intent. The chapter closes with production guidance: expect iterative tuning, align sequence lengths and layers to the domain, monitor loss and learning rate, mind data freshness and quality, and leverage modern tooling and hardware. Done well, these models measurably widen and refine result sets, improving the end-to-end effectiveness of search.
Figure 3.1. Alternative query generation
Figure 3.2. Learning from queries
Figure 3.3. Learning from relevant documents' titles
Figure 3.4. Feed forward neural network with 9 inputs and 4 outputs for language detection
Figure 3.5. Neural network ingesting an input sequence with a fixed input layer of 5 neurons
Figure 3.6. A recurrent neural network
Figure 3.7. A recurrent neuron in the hidden layer of a RNN
Figure 3.8. An unrolled recurrent neural network reading "my name is Yoda"
Figure 3.9. generating Shakespeare text
Figure 3.10. Feeding the unrolled RNN with unsupervised sequence learning
Figure 3.11. Plotting loss trend
Figure 3.12. Supervised sequence learning with single LSTM
Figure 3.13. Sequence to sequence modelling for queries
Summary
Neural networks can learn to generate text, even in the form of natural language. This is very useful for silently generating queries that are executed together with the user-entered ones in order to provide better search results.
Recurrent neural networks are very helpful for the task of text generation, as they are adept at handling possibly long sequences of text.
Long Short Term Memory networks are an extension of recurrent neural networks that can deal with so called long term dependencies. They work better than plain RNNs in dealing with natural language text where related concepts / words may sit at significant distances in a sentence.
Providing deeper layers in neural networks can help in cases where the network requires more computational power for handling larger data sets and / or more complex patterns.
Sometimes it is useful to look close at how a neural network is generating its outputs, small adjustments (like the CR trick) can make a difference in the quality of the results
Sequence to sequence models and thought vectors are powerful tools to learn to generate sequences from sequences in a supervised manner
FAQ
What problem does alternative query generation solve, and how does it improve search effectiveness?It bridges the gap between a user’s information need and the short query they type. By adding alternative queries behind the scenes, the engine can:
- Reduce zero-result queries
- Improve recall by surfacing documents missed by the original phrasing
- Improve precision by boosting documents that match both the original and generated queriesHow is this different from autocomplete and synonym expansion?Autocomplete helps while the user is typing (UI-time hints). Alternative query generation is post-processing at query time: the system augments the submitted query with additional, optional queries. Synonym expansion is a special case of automatic query expansion (often term-level); alternative queries can be full rephrasings, not just term substitutions.Why use recurrent neural networks (RNNs) for generating queries instead of simpler models or the search engine itself?RNNs (especially LSTMs) model sequences and retain context, so they can generate coherent text and handle unseen inputs. Heuristic or index-only methods are limited to term matching and cannot generalize (e.g., “movie spielberg future” might fail to yield “Back to the Future,” while an RNN can still produce it by learning sequence patterns).What are practical ways to prepare training data from query logs?Three common strategies:
- Shared results: correlate queries that return overlapping document IDs
- Time-window sessions: correlate queries issued by the same user within short intervals
- Term similarity: correlate queries sharing terms; optionally enhance with synonym expansion (e.g., mapping “AI” to “artificial intelligence”) while carefully avoiding noisy expansionsHow can I use indexed data (like document titles) to produce better targets?Map each user query to the titles of its top relevant results and train the model to generate those titles (or title-like queries). This biases generation toward content that actually exists in the index, reducing zero-result alternatives (e.g., “movie Zemeckis future” → “Back to the Future”).What is unsupervised query-stream training and when should I use it?Concatenate queries into a plain-text stream, one per line, and train a character-level LSTM to predict the next character. It’s simple, needs minimal preprocessing, and captures natural transitions between queries/topics. Use it when you lack reliable input→target pairs but have ample query text.How do RNNs handle sequence context, and why are LSTMs preferred?RNNs are “unrolled” over time steps; outputs depend on current input and prior hidden state. They learn via backpropagation through time (BPTT). Vanilla RNNs struggle with long-term dependencies; LSTMs add gating to preserve/forget information over longer spans, making them effective for text generation and query reformulation.How does character-level LSTM generation work, and what training settings matter?Each character maps to a one-hot vector; the model outputs a softmax distribution over the character set and samples the next character iteratively. Key points:
- Loss: cross-entropy with softmax outputs
- Hyperparameters: sequence/example length, unroll length, hidden layer size, number of LSTM layers
- More layers can improve quality (up to a point) but need more data/time
- Monitor loss during training; tune learning rate to ensure steady decrease (too high diverges, too low stalls)How do I integrate the generator into a search engine, and force a fresh alternative instead of just “completing” the query?Wrap parsing to build a Boolean query:
- Original user query as MUST
- Generated alternatives as SHOULD
To bias the model toward a brand-new alternative (not just a completion), append a carriage return to the input; the LSTM learns that CR ends one query and begins another, prompting it to start a fresh query.When should I use sequence-to-sequence (encoder–decoder) models, and what is a “thought vector”?Use seq2seq when you can build supervised pairs of input→target queries (e.g., close in time and sharing results). The encoder converts the input sequence (often word-level) into a “thought vector” (final hidden state), and the decoder generates the target query from it. The thought vector captures an intent-like representation that maps an input query to a relevant alternative.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Deep Learning for Search ebook for free
