22 Data over the network

This chapter surveys practical ways to move data across networks with Python and how to turn what you fetch into usable structures. It starts by framing when automation in Python makes sense versus simpler shell tools, then explores common transports for moving files or retrieving data programmatically. Along the way, it stresses workflow concerns such as security, repeatability, scheduling, and the trade-offs between quick, manual downloads and robust, scriptable pipelines.

The chapter presents hands-on patterns for acquiring data: using the standard library to work with FTP (and its TLS variant) for simple, low-security transfers; employing an external library for SFTP over SSH when stronger security is required; and relying on a high-level HTTP client to fetch resources via HTTP/HTTPS, including handling headers, bodies, and automatic URL quoting. It then shifts to APIs—particularly REST-style endpoints—covering query strings, HTTP verbs, and receiving structured responses. JSON is the primary format: you parse it with the built-in module or convenience methods, pretty-print results, and serialize back to files, with cautions about writing multiple objects versus a single container and about memory with very large payloads.

For structured data beyond JSON, the chapter demonstrates turning XML into familiar Python dictionaries using a lightweight parser that preserves attributes and nested structure in an intuitive way. When no API or downloadable file exists, it introduces basic web scraping: fetching pages and extracting content with an HTML parser via tag searches and CSS selectors, balanced with legal/ethical and load considerations. A short lab ties the techniques together by pulling historical weather data from an API, aligning parallel lists (dates and measurements), and exporting a clean CSV for downstream analysis—illustrating the overarching theme: pick the right transport, parse with format-aware tools, and write concise, maintainable, “Pythonic” data pipelines.

• Python has the FTP client in the ftplib library to fetch data over the network via FTP.
• The parameiko library can be used to fetch files via SFTP
• Using a Python script may not be the best choice for fetching files. Be sure to consider options like curl and wget from the command line.
• Using the requests module is your best bet for fetching files by using HTTP/HTTPS and Python.
• Fetching files from an API using requests is very similar to fetching static files.
• Parameters for API requests often need to be quoted and added as a query string to the request URL.
• The most common structured format used for serving data is JSON.
• XML is a more complex data format for serving data and is still also used.
• Using a library like Beautiful Soup you can get data by scraping a web site and parsing the HTML.
• Scraping sites that you don’t control may not be legal or ethical and requires consideration not to overload the server.

FAQ

import ftplib

ftp = ftplib.FTP("example.org")
ftp.login()                 # or ftp.login(user, password)
ftp.cwd("data")             # change directory
print(ftp.nlst())           # list names

with open("remote.txt", "wb") as f:
    ftp.retrbinary("RETR path/to/remote.txt", f.write)

ftp.quit()

import paramiko

t = paramiko.Transport((host, 22))
t.connect(username="user", password="secret")
sftp = paramiko.SFTPClient.from_transport(t)
sftp.get("/remote/path/file.txt", "file.txt")  # download
sftp.put("local.txt", "/remote/path/local.txt") # upload
sftp.close(); t.close()

import requests

r = requests.get("https://example.com/data.txt", timeout=30)
r.raise_for_status()
open("data.txt", "wb").write(r.content)  # use .content for bytes

import requests

url = "https://api.open-meteo.com/v1/forecast"
params = {
    "latitude": 41.882,
    "longitude": -87.623,
    "current": "temperature_2m,wind_speed_10m",
}
r = requests.get(url, params=params)

import requests, json

r = requests.get("https://api.example.com/data")
data = r.json()                       # parse JSON
print(json.dumps(data, indent=2))     # pretty print

with open("data.json", "w") as f:
    json.dump(data, f, indent=2)      # save pretty JSON

import requests, xmltodict

xml = requests.get("https://api.worldbank.org/v2/...&format=xml").text
doc = xmltodict.parse(xml)
# Access attributes with "@attr", text with "#text"; repeated nodes become lists.

import requests, bs4

html = requests.get("https://example.com").text
soup = bs4.BeautifulSoup(html, "html.parser")

# All links
for a in soup("a"):
    print(a.get("href"), a.get_text(strip=True))

# Elements with class "special"
for el in soup.select(".special"):
    print(el.get_text(strip=True))

import csv

dates = ["2024-07-01", "2024-07-02"]
values = [23.1, 22.5]

with open("out.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["Date", "Value"])
    w.writerows(zip(dates, values))   # zip returns an iterator (fine for writerows)