10 Incorporating External Data into Analyses

Incorporating external data expands the scope and impact of analysis beyond what internal systems alone can offer. This chapter emphasizes the mindset and mechanics needed to find, retrieve, and structure outside information so it becomes analysis-ready. It frames data acquisition as an ETL process—extract, transform, load—showing how raw, messy sources (from city services to weather and market signals) must be reshaped into coherent, reliable datasets that align with defined analytical purposes. The goal is to help analysts strategically identify and integrate third-party inputs that enrich context, improve models, and ultimately support better decisions.

Two core acquisition routes are covered: APIs and web scraping. With APIs, you learn to make requests, parse JSON responses, and transform nested structures into tabular form, then automate refreshes and align formats with existing models. Practical constraints—errors, authentication, rate limits, pagination, and cost—inform when API pipelines are worth the investment versus manual downloads. When APIs aren’t available, web scraping offers a fallback: use HTTP requests to retrieve HTML, parse with tools like BeautifulSoup, and standardize values with techniques such as regular expressions. Because websites change and scraping can be brittle, robust error handling, throttling, and adherence to site rules (robots.txt, usage policies) are essential.

The chapter also shows how to tap the rich landscape of public data from governments, NGOs, and academia to add new dimensions and population-level context at low cost. It traces the rise of open data and highlights common domains (economic, demographic, environmental, health) and widely used surveys that can inform many industries. Access options include detailed data catalogs/codebooks, interactive portals for filtering and previewing, and APIs for automated refresh. A practical workflow emerges: define the question, use portals to explore and narrow fields, generate API queries where refresh is needed, and fold the pipeline into your ETL so stakeholders receive timely, well-structured external data.

Each of these datasets was retrieved/constructed for a specific analytical purpose.

Extract, transform, load (ETL) is the process of gathering and structuring data for analysis.

APIs allow you to request and return information from a software application.

A dictionary nested inside a list containing the first set of forecast data.

Dataframe containing one row per forecast record from the API response

Subset of the forecast dataset with the extraneous text information removed.

Forecast dataset aggregated by day.

If manual download and manipulating of data frustrates you and creates additional work, you can streamline process if an API is available.

Unstructured data doesn’t have a predefined format and may require extensive processing.

The first few rows of the HTML content received from our request.

The first few rows of the “prettified” HTML, which shows the nested document structure.

Cleanly printed text from the selected container

The same printed items, including their HTML tags.

Partially structured and transformed forecast data

Dataframe with the temperature integer extracted.

Full text of the forecast description that contains the wind speed.

Result of the regular expression pattern matching, formatted as a list

Final result matching the format of the API and our historical weather dataset from chapter 9.

Truthfully, if the places we worked at did operate in a vacuum or from within a fortress, that would make an analyst’s job so much easier.

Sample question and answers downloaded from the Monitoring the Future survey.

The NYC Open Data Portal allows you to filter, aggregate, and create simple charts.

A sample of the JSON results from the NYC Open Data 311 dataset endpoint

• APIs are a way for two or more computers or services to communicate with each other. They contain a set of protocols that allow you to request specific data and information and receive a response in a specified format. APIs are instrumental for the efficient routine extraction of up-to-date data in analysis.
• The Python requests library makes it easy to retrieve data from an API. With a successful call, you’ll receive a response object that can be formatted as JSON data.
• The JSON data returned by calling an API often requires processing to turn its nested document structure into tabular data that can be read and analyzed with pandas.
• When gathering data from an API, it’s essential to ensure you are capturing the information you want. For example, many APIs implement guardrails to limit improper usage of their service. These guardrails include requiring authentication with an API key, implementing pagination to limit the number of results per API call, and rate limiting the frequency in which you can make API calls.
• Part of your responsibility as an analyst includes determining the value and viability of calling data from an API. This includes assessing the time you will save, the available resources to automate the API calls, and the ability to incorporate the data returned into a centralized data warehouse.
• Web scraping is a method of extracting unstructured data from a website’s HTML, enabling you to transform it into a structured format for analysis.
• Web scraping is performed by extracting information contained within specific tags in the HTML. These tags can be headers, links, paragraphs, sections, etc., that can be selected. From there, you can directly extract information that’s easy to process or use regular expressions for complex pattern matching.
• While web scraping enables you to tap into a wide range of resources, it does offer challenges compared to leveraging data from an API:
• Website content tends to be dynamic, making your processing steps far more brittle and prone to breaking when a website changes.
• Many websites have rules for web scrapers (e.g., pages allowed to be scraped, rate limits). Disregarding these rules can block your IP address on that website.
• Public data sources contain information that is freely accessible to the general population and are usually maintained by the organization that publishes them.
• Public data is often published by government agencies (e.g., the U.S. Department of Labor), international organizations (e.g., the European Union), and academic institutions (e.g., the University of Chicago).
• Public data can usually be accessed with an organization’s data explorer on their website, downloaded as a CSV, or retrieved from an API.
• Public data can be complex, containing hundreds or thousands of columns. Thus, it’s important to review the data catalog of all available variables in your dataset.

FAQ