Overview

Appendix A. Setting Up Your Environment

This appendix provides the practical setup needed to work through the book’s examples without interruption. It introduces the core toolkit—Python, Jupyter Lab, PostgreSQL, an OpenAI account, and a few free data APIs—and explains that you do not need to install everything at once. Instead, the recommended approach is to get the basics in place first, then add extra tools as later chapters require them, with the companion repository supplying the book’s code, notebooks, sample data, and setup files.

The foundation of the workflow is a clean Python installation paired with a virtual environment, which keeps the book’s packages isolated from the rest of your system. After cloning the repository, readers install the shared dependencies and use Jupyter Lab as the main notebook interface for running examples. The appendix also introduces several AI coding companion options—such as ChatGPT, Cursor, Windsurf, and GitHub Copilot—while noting that the early chapters keep the assistant separate from the execution environment so learners can better understand how prompts become code.

Later sections focus on access and safety for external services. Readers create an OpenAI API key, install the current OpenAI Python SDK, and verify that the client can list models by reading the key from environment variables. The appendix also covers PostgreSQL and pgAdmin for SQL and pipeline work, including an optional Docker-based setup and loading the Pagila sample database. Finally, it explains how to manage secrets with environment variables or .env files, where to obtain NewsAPI and SerpAPI keys when needed, and how to run a quick smoke test to confirm that Python, libraries, notebooks, and credentials are all working properly.

A.11 Troubleshooting

"command not found: python" or wrong version. On macOS and Linux, use python3 and pip3. Confirm your virtual environment is active (your prompt should show (deai-venv)).

API key errors. Confirm the key is spelled correctly in .env, that you called load_dotenv() before reading it, and that your OpenAI account has billing set up and available credit.

Package installation fails, or a `ModuleNotFoundError` at import. Make sure your virtual environment is active, upgrade pip with pip install --upgrade pip, and confirm your Python version is compatible. If an import fails even after installing, confirm the package actually landed in this environment with pip show <package> (or list everything with pip list) — a common cause is installing into a different Python than the one running your notebook.

Jupyter can't find installed packages. Jupyter is probably running a different Python than your virtual environment. Install Jupyter inside the environment, register it as a kernel with python -m ipykernel install --user --name=deai-venv, and select that kernel.

PostgreSQL connection refused. Confirm the service is running (brew services list, systemctl status postgresql, or check Docker), that you are using port 5432, and that your username and password match what you set during installation.

Still stuck. Each chapter's README.md and setup/ folder in the repository include chapter-specific guidance, and you can open an issue on the GitHub repository for persistent problems.

With your environment in place, you are ready to follow along with every example in the book. Head back to whichever chapter you came from and pick up where you left off.

FAQ

What tools do I need to start the book, and what can I install later?Python, the repository, and Jupyter Lab are the basics to begin with. You do not need every tool before Chapter 1; additional tools like OpenAI, PostgreSQL, API keys, and scraping libraries are added when the chapters require them.
Which tools are required for Part 1, Part 2, and Part 3 of the book?Part 1 needs Python, the repo, Jupyter Lab, an AI coding companion, PostgreSQL + pgAdmin, and the Pagila sample database. Part 2 adds an OpenAI API key, the OpenAI Python SDK, and NewsAPI keys. Part 3 adds BeautifulSoup, requests, and a SerpAPI key.
What version of Python does the book require?The examples run on Python 3.8 or newer, and the appendix recommends using the latest stable release if possible.
Why should I use a virtual environment for this book?A virtual environment keeps the book’s packages isolated from the rest of your system, prevents dependency conflicts, and lets you delete and recreate the setup without affecting your system Python.
How do I clone the companion GitHub repository?Run git clone https://github.com/dave-melillo/data_eng_ai.git and then cd data_eng_ai. The repository contains the notebooks, listings, data, and setup files used throughout the book.
How do I install and launch Jupyter Lab?Install it with pip install jupyterlab inside your virtual environment, then launch it with jupyter lab. If that command is not found, use python -m jupyterlab instead.
Which AI coding companions can I use with the book?You only need one of these: ChatGPT, Cursor, Windsurf, or GitHub Copilot. The appendix explains how to set up each one, but the early chapters prefer a separate chat-based workflow so you can see prompts and responses clearly.
How do I create and verify an OpenAI API key?Create a key at the OpenAI platform dashboard, store it securely, and set it as OPENAI_API_KEY. Then test it in Python with from openai import OpenAI, client = OpenAI(), and print(client.models.list()).
How do I install PostgreSQL and pgAdmin for the book?Install PostgreSQL using your platform’s recommended method: Windows installer, Homebrew or installer on macOS, or apt on Linux. pgAdmin is included on Windows and macOS, while Linux users install it separately. You can also run PostgreSQL with Docker if you prefer a disposable setup.
How do I know my environment is set up correctly before starting a chapter?Run the smoke test in the appendix: confirm Python is 3.8+, your API keys load, the core libraries import successfully, and Jupyter Lab opens and runs a cell. If all of those work, your setup is ready.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Learn AI Data Engineering ebook for free
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Learn AI Data Engineering ebook for free