Overview

Appendix D. Troubleshooting Common Errors

This appendix is a practical troubleshooting guide for the kinds of failures that most often appear in the book’s examples. It emphasizes that problems usually come from a small set of repeat causes: missing configuration values, absent packages, API changes, or data that does not match what the code expects. Rather than treating every issue as unique, it organizes errors by area and gives the symptom, likely cause, and a direct fix, often pointing back to setup guidance when a fuller installation is needed.

Many of the problems begin with environment setup and authentication. Common fixes include confirming that Python 3.8+ is installed, using the right command names on different systems, activating the virtual environment, reinstalling dependencies, and making sure Jupyter is using the same interpreter as the rest of the project. For API access, the text stresses loading environment variables correctly, checking that keys are named and stored properly, verifying that the secret is actually available without printing it, and updating or replacing expired keys or missing billing setup when requests fail.

The appendix also covers runtime issues across OpenAI calls, web scraping, data handling, and PostgreSQL. It explains how to handle SDK version mismatches, retired model names, rate limits, timeout errors, blocked sites, JavaScript-rendered pages, missing files, absent Excel engines, missing dictionary keys, unexpected NaN values, connection problems, bad credentials, missing drivers, and databases that have not been created yet. The overall message is to diagnose by category, confirm assumptions about version, data shape, and connectivity, and then apply simple corrective steps such as upgrading packages, adjusting code to newer APIs, adding retries, or handling missing data more safely.

D.7 When you are still stuck

If none of the above fixes it:

  • Read the full error message, not just the last line. The actual cause is often a few lines up.
  • Re-run the failing step in isolation in a fresh notebook cell so you can see exactly what input it received.
  • Print the shape of your data (the JSON keys, the DataFrame columns, the response status code) before the line that fails. Most data-engineering bugs are shape mismatches, not logic errors.
  • Check the chapter's README.md and setup/ folder in the repository for chapter-specific guidance, and open an issue on the GitHub repository for persistent problems.

FAQ

Why do I get “command not found: python” when starting the examples?Python may not be installed, or your system may only expose it as python3. Confirm Python 3.8+ is installed, use python3 and pip3 on Mac/Linux, and make sure your virtual environment is active.
What should I do if I see “ModuleNotFoundError: No module named 'X'”?This usually means the package is not installed or the wrong Python environment is active. Activate your virtual environment, run pip install -r requirements.txt, and verify the package with pip list.
Why can Jupyter not find a package that is installed in my virtual environment?Jupyter is likely running a different Python interpreter than your virtual environment. Install Jupyter inside the venv and register a kernel with python -m ipykernel install --user --name=venv, then select that kernel in Jupyter.
Why does pip install fail or take a very long time?Common causes are an old version of pip, the wrong Python version, or a slow network connection. Try pip install --upgrade pip and confirm you are using Python 3.8+.
What does it mean when OpenAI says the API key must be set?The OPENAI_API_KEY environment variable is not set. Load it before calling the API, for example with load_dotenv(), and confirm the key is available first.
Why does os.getenv(...) return None for my API key?Your .env file may not have been loaded, or the variable name may be misspelled. Call load_dotenv() first, verify the name matches exactly, and make sure the .env file exists.
Why do I get AttributeError: module 'openai' has no attribute 'ChatCompletion'?You are using the old-style OpenAI call with the newer v1.0.0+ SDK. Switch to openai.chat.completions.create(...) instead of openai.ChatCompletion.create(...).
What causes RateLimitError or HTTP 429 errors from the OpenAI API?These errors usually mean you are sending too many requests too quickly. Add a small delay between calls and retry with exponential backoff.
Why do I get 404 Not Found when scraping a page or calling an ID-based API?The URL or resource ID may not exist. Verify the URL, check the ID range, and handle 404 as data rather than as a crash.
Why do scraped pages sometimes come back nearly empty or missing the data I expect?The content may be rendered by JavaScript after the initial HTML loads, which requests cannot execute. Use a fully rendered fixture, switch to the site’s API if available, or treat it as a failed fetch if cleaning leaves almost no text.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Learn AI Data Engineering ebook for free
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Learn AI Data Engineering ebook for free