8 Content based image search

Modern search experiences increasingly go beyond text, allowing users to find relevant images by their visual content rather than by manually written captions or tags. The chapter motivates content-based image retrieval by highlighting the ambiguity, cost, and limited recall of text-only metadata, and introduces “query by example,” where a user submits an image to retrieve visually similar results. It frames the core challenge as learning robust, compact representations of images that capture semantics (objects, shapes, textures) while being resilient to variations like lighting, viewpoint, and scale.

The chapter first surveys classical, hand-crafted representations. Global features, such as color histograms across RGB (or other color spaces), are simple and fast but ignore spatial layout and can miss object-specific detail. Local features, like SIFT and SURF, detect stable keypoints (edges, corners, textured patches) and describe them in a way that is robust to rotation and scale. Because each image yields many local descriptors, they are aggregated into a fixed-length vector using the Bag of Visual Words model: cluster all local descriptors (e.g., with k-means) to form a vocabulary of “visual words,” then represent each image by its distribution over these words. This pipeline enables efficient similarity search but still requires careful engineering and can struggle to capture higher-level semantics.

Deep learning—specifically convolutional neural networks—supersedes manual feature design by learning hierarchical features directly from pixels: early layers capture edges and simple patterns; deeper layers learn shapes and object parts. Trained in a supervised fashion on labeled images, CNNs yield both class predictions and powerful intermediate embeddings (from a dense layer) that serve as semantic feature vectors for retrieval. The chapter shows how to build such a model, extract embeddings for query and catalog images, and index them for efficient nearest-neighbor search (e.g., using Lucene’s n-dimensional point structures based on k-d trees). The end-to-end flow is: feed the query image to the CNN, extract its embedding, then retrieve and rank the most similar embeddings to deliver visually relevant results.

Figure 8.1. Learning image abstractions incrementally

Figure 8.2. Manually captioned sport car images

Figure 8.3. Some images described as "tall man"

Figure 8.4. Some images described as "basketball player wearing a 35 numbered jersey"

Figure 8.5. Some visually similar images

Figure 8.6. A red sport car photo retrieved by the image search engine based on a picture of a red sport car model

Figure 8.7. A picture of a butterfly

Figure 8.8. Color histogram for the butterfly picture

Figure 8.9. Another picture of a butterfly

Figure 8.10. Comparing histograms of two butterfly images

Figure 8.11. Analyzing image cells

Figure 8.12. Stars and clusters

Figure 8.13. Image of dogs classified by AlexNet

Figure 8.14. Building blocks (and flow) of convolutional neural networks

Figure 8.15. Convolutional layer

Figure 8.16. Some examples from the Cifar dataset

Figure 8.17. A simple CNN with 2 convolution + pooling layers

Figure 8.18. Testing a CNN on the Cifar dataset

Figure 8.19. Resulting model from DL4J UI

• Searching through binary content like images requires learning a representation which can capture visual semantics that can be compared across different images
  
• Traditional techniques for feature extraction have limits and require significant engineering efforts
  
• Convolutional neural networks are at the core of the recent rise of deep learning as they can learn image representation abstractions (edges, shapes, objects) incrementally during network training
  
• CNNs can be used to extract feature vectors from images that can be leveraged to perform similar images search
  

FAQ