12 AI-powered web scraping data pipelines in action
This chapter turns a collection of working scraping snippets into a real, schedulable data pipeline. It frames the key shift as moving from one-off notebook cells to an orchestrated flow that can run across many products, survive failures, and produce outputs you can trust. The pipeline is built around small, single-purpose agents with typed inputs and outputs, while ordinary Python controls the sequence, retries, validation, and logging.
The core design centers on a strict data contract and four main agents: URL discovery, fetch and clean, AI extraction, and validate and map. Pydantic models define the shapes for raw input, LLM extraction output, the final enriched record, and the state-carrying pipeline record. Shared infrastructure such as logging and exponential-backoff retries makes the system resilient, while the agents themselves stay focused: the LLM is used only where messy text needs to become structured data, and deterministic code handles normalization and sanity checks.
The orchestrator threads each product through the agents, tracks the stage where it is at every moment, and classifies outcomes as success, failed, or needs_review. Low-confidence URL picks and validation issues are routed into a review queue instead of being pushed into the catalog, which keeps the final dataset clean and honest. The chapter concludes by showing how to run the pipeline end to end, save only trusted records, and reuse the pattern for other messy data sources by swapping in new discovery or extraction logic while keeping the same validation-and-review structure.
In this design, an “agent” is not an autonomous decision-maker. It is a small, single-purpose function with a typed input/output contract. Ordinary code orchestrates the sequence, handles retries and validation, and passes data from one agent to the next until messy content becomes validated structured output.
The orchestrator coordinates a four-step pipeline in which each agent performs one clearly bounded task: discovering the best source URL, fetching and cleaning the page, extracting structured product data with an LLM, and validating and mapping the result into a final enriched record, with shared logging/retry infrastructure and a final branch to either the database or a human review queue.
Output of the end-to-end pipeline run. Each product becomes one row with a final status, the stage it reached, the normalized price, and any validation issues. Results vary by run because live pages change, but the shape of the output is stable: a clean table plus a count of how many records need human review.
Lab answers
- The categorization agent. We constrain the model's output to the allowed list with a Pydantic Literal.
- Wiring it into the orchestrator. We call the new agent only when the category is missing, and we guard it so a failure is non-fatal. One detail matters: decide the record's status from the real validation issues before inferring a category, so that filling in a blank category does not, by itself, push an otherwise-clean record into the review queue.
- Adding timing. We add a field to the record and measure elapsed time around the agent chain. A finally block guarantees the duration is recorded even on the early return when discovery fails.
- The report. With timing and categorization in place, the summary is a few lines of pandas.
The categorization count tells you how often the source pages omitted a clean category, which is useful signal about your data sources. The timing almost always shows the extract and fetch stages dominating, the two stages that make network and LLM calls, which is exactly where you would focus if you needed to make the pipeline faster (for example, by running products concurrently rather than one at a time).
Learn AI Data Engineering ebook for free
