9 Peaking into performance
This chapter turns the book’s ideas into production realities, focusing on how to set up, deploy, and operate deep learning models inside search systems without sacrificing performance. It frames the practical trade-offs between training for accuracy and meeting time and space constraints in indexing and retrieval, and it surfaces questions engineers face in real deployments: how to order training and indexing, how to handle updates, how to store and use embeddings efficiently, and how to keep query latency within strict user expectations.
Through a guided optimization journey, the chapter shows how to take a model from design to production. Using an image-classification CNN as a running example, it demonstrates iterative experimentation: begin with small datasets for rapid feedback, then scale; watch loss curves and evaluation metrics; tune learning rate, batch size, and epochs; adjust model capacity judiciously; and apply effective weight initialization and regularization. The narrative emphasizes that better accuracy usually costs more training time and compute, that query-time latency is paramount, and that embeddings introduce storage and computation overhead. It also reviews options for storing dense vectors in search engines and cautions about concurrency, reindexing costs, and the operational burden of recomputation.
To handle continuously arriving data, the chapter advocates streaming architectures that couple neural models with indexing pipelines. An example pipeline ingests social posts, computes multiple document embeddings, indexes them alongside text, and periodically executes benchmark queries to compare classic term-based scoring with neural ranking. Results show that embedding-based methods adapt more readily to changing content, providing a basis for selecting models for production. The overarching guidance is to measure everything—training and inference time, index growth, and ranking quality—so neural and search components can be orchestrated to meet real-world service levels while improving relevance.
Figure 9.1. LeNet model
Figure 9.2. LeNet training
Figure 9.3. Loss plotted for the same neural networks but trained with different learning rates
Figure 9.4. CNN full training loss graph
Figure 9.5. Training with 48 sized batches
Figure 9.6. Training with 24 sized batches
Figure 9.7. Loss curve for 20 epochs
Figure 9.8. Error surface with some points of interest
Figure 9.9. Sharpening loss curve with smaller batch sizes
Figure 9.10. Tuning best
Figure 9.11. Tuning best for 20 epochs
Figure 9.12. Whole dataset training loss curve
Figure 9.13. Neural network and search engine interactions, indexing time
Figure 9.14. Neural network and search engine interactions, search time
Summary
- training deep learning models is not always straightforward; tuning and adjustments for real life scenarios are often needed
- search engines and neural networks are often two different systems which interact both at indexing and search time. It is essential to monitor their performance in order to keep the overall user experience good in terms of response times
- real life deployments, like the common streaming scenario, need to account for load, concurrency and quality evaluation for the best possible search solution to be used
FAQ
How do I balance model accuracy with training and inference performance for a search system?
Start with small, simple models and small data slices to iterate quickly, then scale up. Track both quality (accuracy, precision/recall, F1, confusion matrix) and cost (training time, GPU/CPU use, memory). Optimize in this order: get a reasonable architecture, tune learning rate and batch size, add regularization/weight init, then consider more data/epochs. Always measure query-time latency for the embedding step and ranking function, as user-perceived speed matters most.Should training happen before indexing, or should I index first? Can the feeds be combined?
It depends on your pipeline and freshness needs. In many setups: train or fine-tune the model first, then index documents with their embeddings so ranking can use them. If data arrives continuously, combine steps in a pipeline: ingest -> embed -> index. For model changes that affect embeddings, plan a reindex (possibly rolling) or dual-write strategy to avoid downtime.What are the main performance risks when adding neural models to search?
- Query-time latency: embedding the query and applying neural scoring must be fast enough for SLAs.- Indexing throughput: generating embeddings can slow ingest and reindexing.
- Index size: storing dense vectors increases disk/memory footprint.
- Concurrency: the model service must handle parallel requests from indexing jobs and many concurrent search users.
How should I store embeddings in Lucene-based systems, and what are the trade-offs?
- Binary doc values: compact storage; compute similarity after fetch; flexible but requires custom scoring.- N-dimensional points: enables nearest-neighbor style queries; better for vector search but with indexing/search trade-offs.
- Text encoding: feasible with careful design; leverages existing text infra but is indirect. Consider vector-native platforms (e.g., Vespa) or features in Solr/Elasticsearch if they fit your use case and scale.
How do batch size and learning rate affect convergence and training time?
- Learning rate: too high can diverge; too low converges slowly. The same architecture can converge at different speeds with different rates.- Batch size: smaller batches can generalize better and reduce overfitting on frequent patterns but increase training time and noise; larger batches train faster per epoch but may smooth out rare signals. Tune both empirically; monitor loss curves and validation metrics.
Why do weight initialization and regularization matter in production models?
Good initialization (e.g., Xavier/Glorot variants) places weights near useful regions of the error surface, speeding and stabilizing training. Regularization (L1/L2, gradient normalization) reduces overfitting and sharp loss oscillations, especially with small batches or noisy data, leading to more robust models for real traffic.When should I add layers or increase model size?
Only after simpler fixes fail. If the model underfits (loss plateaus high and validation accuracy is low even with tuning), consider more capacity. But larger models increase training time, inference latency, and memory. Prefer architectural and optimization tweaks first; add capacity incrementally and re-evaluate latency budgets.How can I evaluate models before reindexing a large corpus?
- Prototype with small data subsets to compare settings quickly.- Keep a separate validation/test set; watch F1 and confusion matrices, not just training loss.
- Save models with their metrics and timings for later choice.
- Run offline retrieval experiments by indexing a sample with embeddings and testing ranking end-to-end before committing to a full reindex.
How can streaming architectures help with neural ranking on live data?
Use a stream processor (e.g., Apache Flink) to ingest items continuously, window them (e.g., count windows) to control workload, compute embeddings (paragraph vectors, averaged word vectors, etc.), index into Lucene, and execute fixed queries periodically. Persist results (e.g., CSV) to track how classic vs neural ranking adapt to fresh content and to choose the best model for trends.How do I handle concurrency and scaling for indexing and query-time inference?
- Make the model service stateless and horizontally scalable; batch inputs where possible.- Use asynchronous pipelines for indexing; commit in chunks.
- Cache frequent query embeddings and hot document vectors.
- For reindexing, consider rolling updates or dual indices to avoid downtime.
- Load test both indexing and search paths to validate throughput and tail latency.
Deep Learning for Search ebook for free
