19 Using Python libraries
Python embraces a “batteries included” approach, and this chapter surveys the breadth of its standard library so you can solve common problems without extra installs. It highlights support for text and bytes handling, core data types and collections, and rich numeric and mathematical utilities. You also get tools for files, storage, and persistence (including compression and lightweight databases), deep integration with the operating system (paths, processes, threading, and more), comprehensive internet and networking capabilities (encodings, JSON/XML/HTML, HTTP clients/servers), and development aids for testing, debugging, profiling, and runtime introspection. The emphasis is on recognizing what’s already available and consulting the library reference before reaching for external packages.
When needs extend beyond the standard library, the chapter outlines how to locate and evaluate third-party modules, noting the pros and cons of OS-level installers versus Python-native packaging. It presents the Python Package Index (PyPI) as the primary, searchable repository and shows how pip streamlines discovery, dependency resolution, installation, upgrades, and version pinning. Practical guidance includes installing into a user-specific location when system-wide changes aren’t possible or desirable, while keeping security and source trust in mind.
For clean dependency management, virtual environments created with venv provide isolated Python installations per project, preventing conflicts across applications and enabling reproducible setups; within them, pip manages packages as usual. The chapter also mentions complementary tools and ecosystems—such as pyenv for managing Python versions, alternative package/project managers like poetry, rye, and uv, and the Anaconda distribution with conda for data-science–oriented stacks. Overall, the recommended practice is to favor the standard library when it fits, use PyPI and pip for well-scoped additions, and rely on virtual environments to keep projects stable and maintainable.
Summary
- Python has a rich standard library that covers more common situations than many other languages, and you should check what’s in the standard library carefully before looking for external modules.
- Among other things, the standard library has modules for handling different data types, manipulating files and the file systems, accessing operating system services, handing Internet protocols, and developing and debugging Python code.
- If you do need an external module, prebuilt packages for your operating system are the easiest option, but they’re sometimes older and often hard to find.
- The standard way to install from source is to use pip, and the best way to prevent conflicts among multiple projects is to create virtual environments with the venv module.
- Usually, the logical first step in searching for external modules is the Python Package Index (PyPI).
FAQ
What does Python’s “batteries included” philosophy mean?
It means a stock Python install ships with a rich standard library that covers common needs without extra installs: working with data types and numbers, files and storage, operating system services, internet protocols and formats, and development/debugging tools.Which standard modules help with strings, bytes, and text processing?
Common picks include: - string (constants and helpers) - re (regular expressions) - struct (pack/unpack binary data) - difflib (compute differences, create patches) - textwrap (wrap/fill/format text)What modules support core data types, containers, dates/times, and typing?
Useful modules: - datetime, calendar (date/time operations) - collections (container types) - enum (symbolic constants) - array (efficient numeric arrays) - sched, queue (scheduling and synchronized queues) - copy, pprint (copy structures, pretty-print) - typing (type hints)Which numeric and mathematical modules should I know?
Highlights: - numbers (numeric ABCs) - math, cmath (real/complex math) - decimal (decimal arithmetic) - statistics (basic stats) - fractions (rationals) - random (PRNG, sampling, shuffling) - itertools, functools, operator (iterators and functional tools)How can I manipulate files, paths, archives, and persistence with the standard library?
Key tools: - Paths/files: os.path, pathlib, fileinput, filecmp, tempfile, glob, fnmatch, linecache, shutil - Persistence/DB: pickle, shelve, sqlite3 - Archives/compression: zlib, gzip, bz2, zipfile, tarfile - Formats/config: csv, configparserWhich modules expose operating system services like logging, threads, and processes?
Common choices: - os, io, time (OS interfaces, streams, time) - optparse (command-line options) - logging (logging framework) - getpass, curses, platform (passwords, terminal UIs, platform info) - ctypes (call native libs), select (I/O readiness) - threading, multiprocessing, subprocess (concurrency and subprocesses)How does Python support internet protocols, data formats, and simple servers?
Frequently used modules: - socket, ssl (low-level networking and TLS) - email, mailbox, mimetypes (mail and MIME) - json (JSON), base64/binhex/binascii/quopri/uu (encodings) - html.parser, html.entities; xml.parsers.expat, xml.dom, xml.sax, xml.etree.ElementTree - cgi, cgitb; wsgiref (WSGI utilities) - urllib.request, urllib.parse (URLs) - ftplib, poplib, imaplib, nntplib, smtplib, telnetlib (protocol clients) - socketserver, http.server; xmlrpc.client, xmlrpc.serverHow do I find third‑party packages and what is PyPI?
You can search the web for “your need + Python” or use OS/Distro package managers (easy, but often older). The official central index is the Python Package Index (PyPI), which hosts hundreds of thousands of projects and is searchable by name, category, and version.How do I install or upgrade a package with pip, choose versions, or install just for my user?
- Install:python3 -m pip install requests- Upgrade:
python3 -m pip install --upgrade requests- Pin/version range:
python3 -m pip install requests==2.32.3 or requests>=2.32- Per-user install:
python3 -m pip install --user requestsNotes: Some Linux distros package pip separately. In Google Colab many packages are preinstalled; you can install others with
!python3 -m pip install package.What are virtual environments and how do I create/activate one? Are there other tools?
A virtual environment is an isolated Python + packages directory tree, preventing conflicts between projects and with system Python.- Create:
python -m venv myenv- Activate (Windows):
myenv\Scripts\activate.bat- Activate (macOS/Linux):
source myenv/bin/activate- Then use
pip install ... inside the env. Alternatives and helpers include pyenv (manage Python versions/envs), poetry/rye/uv (packaging/workflows), and Anaconda/conda (popular in data science, strong dependency handling, smaller curated ecosystem than PyPI).
The Quick Python Book, Fourth Edition ebook for free
