3 Data privacy and safety: Technical and legal controls

This chapter surveys how data privacy, safety, and governance shape the development and use of generative AI. It explains how vast, largely uncurated training corpora embed bias, underrepresent many languages and cultures, and may contain sensitive personal data that models can memorize and leak. The authors frame safety as both a technical and institutional problem: LLM outputs must be steered away from harmful or policy-violating content while protecting users’ information, and these goals must be supported by documentation, transparency, and accountable governance that can keep pace with evolving social norms and regulations.

On the technical side, the chapter examines controls across the AI pipeline. Post-processing with classifiers can block toxic outputs, but must balance helpfulness, harmlessness, and honesty. Data filtering and conditional pre-training can reduce toxic tendencies, though at scale they are costly and imperfect. Post-training methods—supervised fine-tuning, RLHF, and AI-driven approaches like Constitutional AI—align models with safety policies while reducing dependence on human raters. The authors also explore privacy risks from unintended memorization and data extraction and review mitigations such as PETs (pseudonymization, blocklists), differential privacy trade-offs, membership inference testing, and novel training-objective tweaks (e.g., preventing verbatim sequence learning). Emerging research in unlearning and interpretability suggests future avenues to suppress harmful capabilities. Beyond model training, the chapter highlights user-side risks from inadvertent data leakage in prompts, practical workplace safeguards, and enterprise deployments that isolate or avoid logging sensitive data.

Legally and organizationally, the chapter traces privacy frameworks from early Fair Information Practices and OECD principles to modern regimes like GDPR, Brazil’s LGPD, and China’s PIPL, contrasting them with the United States’ sectoral and state-based approach (e.g., CCPA/CPRA). It reviews recent enforcement and scrutiny around generative AI, including questions about lawful bases for processing, transparency, children’s access, and the feasibility of rights like erasure once data is embedded in model weights. Complementary instruments—the EU AI Act, the Digital Services Act, product liability and cybersecurity rules—and sectoral protections such as FERPA expand oversight. Companies respond with privacy-by-design practices, PETs, enterprise safeguards, and documentation (dataset/model cards, data nutrition labels), though maturity varies. The chapter closes by emphasizing that compliance is a baseline: robust, lifecycle governance grounded in transparency, accountability, fairness, and human rights is essential—and users should remain cautious and informed about how their data is collected and used.

The logic of a response decision with a post-hoc toxicity classifier.

A simplified version of the architecture in the Constitutional AI method for improving model generations’ compliance with content policies.

Five core principles of FIPs [69]

A timeline of data protection laws.

• Large language models are trained on gigantic amounts of text from the web, which encodes linguistic and cultural biases and sensitive information into the models.
• The term AI alignment refers to the alignment between the goals of a given machine learning system and the intended goals of its human creators, or—more broadly—the alignment between powerful AI systems and human values.
• Researchers are using several strategies to try to prevent the model from generating inappropriate responses, including post-processing detection algorithms, content filtering or conditional pre-training, safety post-training, and machine unlearning.
• Another privacy risk is exposing personal or sensitive data through user prompts in conversational AI systems. This information can be used to further improve or train the tool and potentially be leaked in responses to other users’ prompts.
• Existing privacy laws and data protection frameworks are often limited in nature. Companies have taken internal measures to prevent their proprietary data from leaking into LLMs through employees’ use. Users can also take individual precautions to avoid the risk of incidental exposure of sensitive data.

FAQ