This chapter introduces cross-language information retrieval and shows how to serve users who search in one language while relevant content exists in others. It motivates the problem with web and mission-critical vertical scenarios (for example, legal information for refugees), and contrasts two practical strategies: translating queries at search time versus translating documents ahead of time. A production-friendly pipeline detects the query language, selects appropriate machine translation models, translates the text, and submits the original and translated queries together (as OR clauses), followed by careful ranking to balance recall and user preferences. The chapter demonstrates this integration on top of Apache Lucene with a custom query parser, highlighting design choices such as language detection, model selection, and scoring considerations for mixed-language results.
It then surveys statistical machine translation (SMT): learning from parallel corpora to estimate word/phrase translation probabilities, using Bayes-inspired scoring and alignment to handle reordering, one-to-many, and many-to-one mappings. Phrase-based SMT improves context handling by translating and reordering phrases rather than isolated words. The chapter underscores the centrality of high-quality parallel data, explains common formats like TMX, and provides lightweight tooling to parse parallel sentence pairs for training or evaluation. While SMT was long the standard, it demands substantial engineering for alignment and feature design across language pairs, which limits scalability when many directions must be supported in a search setting.
Neural machine translation (NMT) is presented as the modern, end-to-end alternative that typically surpasses SMT in accuracy and maintainability for search applications. Using sequence-to-sequence encoder–decoder LSTMs, the encoder produces a thought vector from source tokens (via an embedding layer), and the decoder generates target tokens with a softmax over the target vocabulary at each step; no explicit alignment tooling is required. The chapter details a practical implementation: building the computation graph, preparing dictionaries and token indices from parallel corpora, training over epochs, and exporting the model. The trained translator is plugged into the Lucene query parser as a TranslatorTool to translate queries at runtime. Compared to SMT, NMT offers better quality with less manual tuning, robust handling of variable-length sequences, reusable embeddings, and more consistent gains in both recall and precision when ranking multilingual search results.
Figure 7.1. Searching for "rete neurale", Italian for "neural network"
Figure 7.2. Searching for "artificial neural network", getting results in Italian too
Figure 7.3. Query translation flow
Figure 7.4. An Encoder Decoder model
Figure 7.5. An encoder network with 4 hidden recurrent layers
Figure 7.6. Encoder network layers (up to second hidden recurrent layer) with a dictionary of 10 words
Figure 7.7. A decoder network with 4 hidden recurrent layers
Figure 7.8. Encoder Decoder model with two LSTMs layers per side
Figure 7.9. Encoder decoder network training graph
Summary
Machine translation can be very useful in the context of search to improve the user experience of multi language speaking users
Statistical models exist that can reach a good translation accuracy, however the amount of per language pair tuning required is non trivial
Neural machine translation models look into ways of learning to translate sequence of text into different languages in a less articulated yet more powerful way
FAQ
What is cross-language information retrieval (CLIR) and why does it matter for search?CLIR lets users query in one language and retrieve relevant documents in other languages. It improves recall (more relevant results) and user satisfaction (results in a preferred or native language), which is especially valuable in web search, technical topics with richer English content, and critical scenarios like refugees seeking legal documents in a host country’s language.Should I translate queries or translate documents?Both are valid and can be combined. Translating queries is often best for web search because it expands recall across languages and preserves ranking flexibility. Translating documents (or snippets/answers) is useful when consumption in the user’s language is paramount (e.g., customer reviews, question answering). The choice depends on use case, content ownership, latency, and ranking strategy.How are translated results combined with the original query for retrieval?Detect the query language, translate the query into one or more target languages, then issue a combined boolean OR query: original_query OR translated_query. This increases recall by matching documents in multiple languages. Ranking may then prioritize translated or original-language results based on user context and informativeness.How does a search system pick the right translation model for a query?Use a language detector first to identify the source language (e.g., “ita” for Italian). Then select only the machine translation models that can translate from that detected source language to desired target languages. This avoids wasted computation and bad translations from mismatched models.How can I implement query-time translation on top of Apache Lucene?- Extend Lucene’s QueryParser to: read the query, detect its language, select the appropriate TranslatorTool(s), translate the query, and build a BooleanQuery with SHOULD clauses for original and translated texts.
- Keep translation models preloaded and keyed by source language.
- The result is a single query that searches across languages in one pass.What is Statistical Machine Translation (SMT) and how does it work?SMT estimates the most probable translation using statistics learned from a parallel corpus. It computes probabilities like p(e|f) using counts of words/phrases and their co-occurrences across aligned sentence pairs, often requiring word/phrase alignment and reordering. Phrase-based SMT improves accuracy by translating phrases instead of isolated words.What makes alignment challenging in SMT?- Word positions differ across languages (reordering).
- Non 1-to-1 mappings (one-to-many or many-to-one) are common (e.g., “I live” → “vivo”, “USA” → “Estados Unidos”).
- High co-occurrence doesn’t imply correct translation without alignment signals.
Alignment models map positions between source and target tokens to improve translation quality.What are parallel corpora and how do I work with them?Parallel corpora contain aligned source–target sentence pairs used to train MT. Resources like OPUS provide datasets in formats such as TMX. You can parse TMX to extract ParallelSentence(source, target) pairs, build vocabularies/dictionaries, and feed them to SMT or NMT training pipelines.Why switch to Neural Machine Translation (NMT) and what is the encoder–decoder approach?NMT (seq2seq) learns translations end-to-end, reducing manual components like explicit alignment. An encoder RNN (often LSTM/GRU) reads the source and produces a fixed-length thought vector; a decoder RNN generates the target sequence token by token. Benefits: simpler setup, variable-length handling, reusable embeddings, and often better quality than SMT.How do I train and deploy an NMT model for query translation?- Prepare a parallel corpus (e.g., TMX), tokenize, and build a dictionary.
- Define an encoder–decoder graph (embedding + LSTM layers for encoder; LSTM + softmax output for decoder).
- Train over multiple epochs with a DataSetIterator.
- Serialize the trained model, load it in the search service, wrap it as a TranslatorTool, and plug it into the MT-aware QueryParser to translate queries at runtime.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Deep Learning for Search ebook for free
