This chapter surveys Python’s practical toolkit for working with data on disk and at the terminal. It starts from the basics of opening files, choosing text or binary modes, and using context managers for reliability, then moves through everyday reading and writing patterns, newline and encoding concerns, and convenient path-based I/O with pathlib. It also covers interacting with users and the shell via standard input/output and redirection, and introduces three powerful persistence/format tools—struct for structured binary data, pickle for serializing Python objects, and shelve for dictionary-like storage on disk—along with their trade-offs. A capstone exercise applies these ideas by refactoring a word-count utility to read from files or standard input, mirroring common command-line behavior.
Core file mechanics emphasize using open with paths (built safely with os.path.join), closing files promptly, and preferring with for automatic cleanup. Modes such as 'r', 'w', and 'a' control reading, overwriting, or appending; buffering, encoding, and newline parameters fine-tune performance and cross-platform text handling. For reading, readline, readlines, and iterating over the file each balance simplicity and memory usage; for writing, write and writelines mirror their counterparts (noting that you must add newline characters explicitly). Binary mode ('rb'/'wb') returns bytes, supports fixed-size reads, and requires explicit decoding to text when needed.
Beyond the basics, pathlib offers concise text/byte helpers without manual open/close calls. Terminal I/O uses input for prompts and the sys module’s stdin/stdout/stderr for low-level control, enabling redirection and capturing output (including print’s file parameter). For structured binary formats, struct’s pack/unpack and calcsize provide precise control over layout and endianness. For persistence, pickle can serialize most Python objects (ideal for caches or small state files) but is neither space- nor speed-efficient and is unsafe with untrusted data; shelve layers a persistent, dictionary-like interface over pickle, trading convenience for restrictions such as string-only keys, slower writes, and no concurrent access controls. The chapter’s lab underscores how to integrate these techniques to match real command-line utilities while remaining mindful of edge cases and correctness.
Summary
File input and output in Python uses file objects with methods to open, read, write, and close files.
While by default Python treats files as text, there is also binary mode which processes files as bytes.
Python makes it easy to redirect standard input, output, error channels to read and write from files.
Path objects from the pathlib library can also be used for reading and writing text or binary objects.
In addition to reading and writing text, the struct module gives you the ability to read or write data packed into structured binary formats.
The pickle module provides a simple, safe, and powerful way of saving and accessing arbitrarily complex Python datastructures.
The shelve module offers a convenient way to save and load objects, similar to using a dictionary, so long as the keys are strings.
FAQ
What’s the safest way to open and close a file in Python?Use a context manager so the file always closes: with open('path', 'mode') as f: ... The file is automatically closed when the with-block exits, even on errors.Which file mode should I use for reading, writing, appending, and binary I/O?'r' reads, 'w' overwrites, 'a' appends, and adding 'b' enables binary mode (for example, 'rb', 'wb'). To append to an existing text file and then read from the start: open('myfile.txt', 'a') to add, close it, then open('myfile.txt', 'r') to read from the beginning.What’s the best way to read a large text file line by line?Iterate over the file object: for line in f: ... It streams one line at a time and is memory efficient. readline() gets one line per call; readlines() loads all lines into memory (simpler but can be costly for big files).How does Python handle different newline conventions (Windows, macOS, Linux)?Text mode uses “universal newlines,” translating platform endings to '\n'. Control this with open(..., newline='\n' or '\r' or '\r\n') to force a specific newline, or newline='' to accept all and return the original endings. In binary mode, bytes are returned unchanged.What’s the difference between write and writelines?write(s) writes the string exactly as given (no automatic newline). writelines(list_of_strings) writes each string in sequence without adding newlines; include '\n' yourself if you need line breaks.When should I use binary mode, and how do I convert between bytes and text?Use binary mode ('rb'/'wb') when handling non-text data or fixed-size byte chunks. Reads return bytes, not str. Decode bytes to text with b.decode('utf-8') (or another encoding) and encode text to bytes with s.encode('utf-8').How can pathlib simplify simple file reads and writes?Path objects offer read_text(), write_text(), read_bytes(), and write_bytes() without calling open/close. Note: write_text/write_bytes overwrite content; there’s no append method—use open(..., 'a') if you need to append.How do I work with standard input/output and redirect them?Use input() for simple prompts. For lower-level access, use sys.stdin, sys.stdout, and sys.stderr like file objects. You can redirect print to a file with print(..., file=f). Temporarily reassigning sys.stdout works in scripts; in Jupyter/Colab, save the old value and restore it afterward (sys.stdout = old_stdout).When is the struct module useful, and how do I use it?Use struct for structured binary data from/to other programs or protocols. Define a format string (for example, 'hd7s'), pack with struct.pack(fmt, ...), compute record size with struct.calcsize(fmt), read fixed-size chunks, and unpack with struct.unpack(fmt, data). You can also control endianness and standard sizes in the format.What’s the difference between pickling and shelving, and what are the caveats?pickle serializes Python objects to bytes for storage and later restoration; it’s easy but not secure (never unpickle untrusted data) and isn’t the most space/CPU efficient. shelve provides a persistent, dictionary-like store on disk; keys must be strings, stored values must be pickleable, writes can be slower, there’s no concurrency control, and you should close the shelf to flush changes.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
The Quick Python Book, Fourth Edition ebook for free
