11 AI-powered web scraping

Building on the earlier hand-coded scraping pipeline, this chapter shifts the focus from brittle, site-specific selectors to places where AI can replace or augment manual work. The manual approach proved workable but hard to scale because every new site, variant page, or HTML change demanded more selectors, more fallback logic, and more maintenance. The core idea is to treat a data pipeline as a set of stages and ask where unstructured content, variability, or judgment calls make AI genuinely useful, while leaving simple deterministic tasks to ordinary code.

The chapter organizes AI use into three kinds of generated data: extracted data that already exists but is hard to retrieve, enriched data that requires inference or categorization, and synthetic data that is generated from scratch for testing or development. For the RuckZone pipeline, the highest-value AI opportunities are URL discovery, HTML cleaning, and product extraction. URL discovery becomes a two-step process in which a search API gathers candidate pages and an LLM ranks them using criteria such as official source, product-page quality, and likelihood of containing useful fields. HTML cleaning is expanded beyond basic tag removal by stripping common noise like headers, banners, reviews, and related-product modules, reducing token usage and improving downstream extraction quality.

The extraction workflow itself replaces manual CSS selectors with a schema-driven prompt and structured output. A Pydantic model defines the fields the model should return, and the prompt tells it to extract only what is explicitly present, prefer null over guessing, and preserve values like price and weight as strings for later normalization. Compared with the manual approach, the AI version extracts more complete product records across many sites with one reusable function, though it still needs validation, confidence review, and cost awareness. The chapter closes by emphasizing batch processing, failure handling, token budgeting, and a practical checklist for deciding when AI is appropriate, along with guidance on when deterministic methods remain the better choice.

Example output after running listing 11.1. Notice that four candidate URLs were found with titles, URLs and search positions.

Example output after running listing 11.2. Notice that the reasoning is based on our prompt and what we deem important to accomplish the task at hand. This is how you can fine tune results and configuration based on your specific use case using natural language instead of code.

A high level example of the type of content listing 11.3 aims to remove. HTML and CSS tags help a website understand how to render, style and process it’s content. While this is valuable for the operation of a website, these “tags” are white noise when trying to extract information from a website.

A comparison of the manual extraction approach versus the AI first approach, clearly showing the value AI brings when extracting valuable data from a website. You should see similar results after running listing 11.6.

The results of running listing 11.7 and the subsequent pipeline outlined in this chapter. Please note that results will vary, but now you have the blueprint for making your extraction pipelines more deterministic when using AI.

Flowchart: "Should I use AI for this step?" Decision tree walking through the key questions from the checklist.

FAQ