6 Analyzing images and videos

This chapter broadens data analysis beyond text and tables to the visual world, showing how large language models, specifically a multimodal model, can interpret images and videos from natural-language instructions alone. It outlines three mini-projects—answering free-form questions about images, tagging people in photos, and generating titles for videos—to demonstrate practical capabilities and design patterns. A minimal setup adds OpenCV for image/video handling and a simple HTTP client for direct API calls. Throughout, the core technique is to build multimodal prompts that mix text with one or more images (or video frames), enabling flexible tasks without task-specific training.

The first project builds a question-answering tool for images: users supply a picture and a natural-language question, and the model returns a textual answer. This generalizes classical, narrowly trained vision models by letting you specify tasks at inference time (for example, specialized classifications) through prompt wording. The implementation uses the chat completions interface with a message that combines text and an image reference, and it supports multiple images if needed. The chapter also highlights cost control: image processing fees depend on resolution and detail settings, with a low-detail mode to reduce expense and a high-detail mode whose price scales with image size, reminding readers to be mindful when processing large collections.

The second project automates tagging people in photos by comparing a reference portrait to each candidate image via a two-image prompt that asks if they depict the same person; matching images are copied to an output folder with a name prefix that serves as the tag. To keep photos private while still usable by the model, images stored locally are converted to base64, UTF-8 strings, and embedded as data URLs in the prompt; the code also illustrates constructing HTTP requests directly to the API. The third project generates concise video titles by extracting a small set of frames with OpenCV, encoding them like images, and prompting the model to propose a title—trading comprehensive coverage for affordability by limiting frames. Together, these patterns show how a few lines of code can deliver practical visual analytics while balancing privacy, performance, and cost.

Multimodal prompt containing an image and text: the prompt instructs the language model to decide whether the picture shows a banana. In this case, the expected output is “No” (otherwise “Yes”).

Multimodal prompt containing two images and text: the prompt instructs the language model to check whether the two pictures show the same person (expected answer: “Yes”) or not (expected answer: “No”).

Multimodal prompt for video processing: based on a selection of video frames, the language model is instructed to generate a suitable title.

• GPT-4o processes images as well as text.
• Prompts can integrate text snippets and images.
• GPT-4o supports multiple image formats.
• Images can be specified via a public image URL.
• Locally stored images can be uploaded to OpenAI.
• GPT-4o processes images in string encoding.
• Processing images is costly, compared to processing text.
• The costs of image processing may depend on the image size.
• Processing images with a low degree of detail reduces costs.
• The base64 library can encode images as strings.
• Decompose videos into their frames to send them to GPT-4o.
• The OpenCV library can be used to extract frames from videos.

FAQ