This chapter surveys the core ethical questions that arise when building large language models. It highlights the appeal of LLMs as versatile “everything” systems that can assist across translation, search, creativity, and more, while warning that such breadth multiplies untested edge cases and potential harms. Bias, misleading or unsafe outputs, and the limits of post-hoc mitigations (like fine-tuning for known issues) underscore why evaluations must be tied to concrete use cases and deployment contexts. The discussion also weighs the social and economic tension of automating knowledge work: potential efficiency and access gains coexist with disruption to creative and professional labor, suggesting the need for proactive engagement with affected communities and careful decisions about where LLMs should or should not be used.
The chapter then examines arguments that LLMs could pose existential risks through recursive self-improvement and misaligned goals. It presents a skeptical view: such scenarios lean on unfalsifiable claims, overlook technical limits, and presume deterministic outcomes in a probabilistic world. While alignment is a real and difficult problem (as human institutions themselves demonstrate), the text argues that fears of inevitable catastrophe are not well supported by today’s evidence. It also stresses a geopolitical and pluralistic wrinkle: even if perfect alignment were achievable, alignment to whom remains unresolved, and capable actors with divergent interests could train or obtain powerful systems, complicating any simple, universal solution.
Finally, the chapter focuses on data ethics. Modern LLMs demand massive corpora whose licensing often did not anticipate AI training, leaving legal questions unsettled and ethical responsibilities pressing. Fair use considerations are nuanced and context-dependent, and “legal” does not necessarily mean “ethical.” Proposals to compensate creators face daunting practical and economic barriers, and reliable attribution of model outputs to specific training sources remains technically immature. Training solely on public-domain or permissively licensed data is constrained by coverage, recency, quality, and provenance challenges. The chapter urges better data governance, negotiated access and partnerships, research into attribution and constrained training, and a norm of building models and deployments that respect creators, mitigate harms, and align with broader societal values.
Two kinds of hypothetical issues occur within the alignment problem, as commonly argued by those who think LLMs pose an existential risk to humanity. The top path shows a direct alignment problem, where the AI’s target solution directly harms humans. The bottom path shows an indirect alignment problem, where the AI has created a sub-goal toward its eventual target. Even if the target, say solving a hard math problem, is achieved, this LLM will do this at the cost for humanity. In an intermediate step, the LLM decides it needs more earthly resources than can be shared with humans to solve the problem.
The S-curve or “sigmoid” shows the classic plateau behavior: that at some point, you hit diminishing returns. The counterpoint to this expressed by the iterative S-curve model is that progress continues past the plateau of diminishing returns by discovering new techniques, each represented by a new S-curve. The new techniques may start worse than the existing methods but have a higher potential to surpass them.
Summary
LLMs’ ability to be used for “everything” via one model helps people use them quickly and effectively for many tasks. This broad applicability to many tasks also makes it impossible to test the safety of all ways people may use LLMs.
Historically, automation has been a good thing. Still, LLMs pose a unique risk to automating knowledge work, which is different from automating manual labor, which has been the historical driver of improved living standards. The true impact of broadly automating knowledge work is unknown.
Some fear that an LLM that is good enough to improve on a new LLM’s design will cascade to super-intelligent algorithms that do not need humanity.
Aligning any algorithm to what we meant, instead of what we asked, is a major challenge — which likely has no reduction in risk even if solved.
Ethically obtaining data is fraught with legal issues due to technology moving faster than the law.
The financial and technical logistics in compensating all content authors for their content’s use in the training data is unlikely to be practical, imposing ethical questions on the fairness of using their dall.
Public domain data with no copyright is too old to be problematic and poses different challenges related to identifying its legal status.
FAQ
Why build LLMs at all, and what’s new compared to earlier AI systems?LLMs promise broad, flexible capability from a single model: translation, transcription, information retrieval, reasoning assistance, and creative generation. Unlike earlier task-specific systems, one model can support many goals through a familiar chat interface, lowering barriers to adoption. The ethical question is whether the breadth and speed of this capability justify the risks it introduces, especially when uses are hard to anticipate.What makes an LLM an “everything app,” and why is that ethically tricky?The upside is convenience and accessibility: one interface and model can decompose complex tasks and tailor explanations. The downside is scope risk: if a system can be used for almost anything, you cannot test it for everything. There will always be a gap between known, evaluated behaviors and unanticipated, potentially harmful uses, making responsible deployment and clear use-case boundaries essential.How do bias and unsafe recommendations emerge, and why aren’t benchmarks enough?Bias often comes from patterns in training data and can surface in realistic settings (e.g., stereotyping in images or recommending debunked, discriminatory practices). Models can score well on explicit bias tests yet still cause harm in context because benchmarks are narrow and not representative of real use. Evaluating risk should focus on specific applications and user interactions, not only on abstract bias scores.Should we aim to automate knowledge work, and who stands to be harmed or helped?LLMs extend automation into creative and white‑collar domains (writing, design, arts). Some argue this mirrors past automation waves that expanded total output and jobs by augmenting workers. Others warn of displacement, de-skilling, and reduced incentives to create new human content. The ethical challenge is managing transition costs, fairness, and motivation for future human production.How does training LLMs affect online communities and creators whose content is scraped?Platforms fueled by user-generated content provide valuable training data but may lose traffic and revenue when model outputs substitute for visits. Even when licensing deals are struck, contributors may object to commercialization of their free labor. Builders should consider proactive agreements, compensation models, or community partnerships rather than relying solely on permissive interpretations of existing terms.Do LLMs pose an existential risk, and what assumptions drive that view?Concern centers on two claims: that LLMs could iteratively self-improve by using tools to create better models, and that misaligned goals could motivate harmful actions against humans. Critics counter that these arguments depend on speculative, often unfalsifiable assumptions about capabilities, physical-world control, and deterministic outcomes. A pragmatic stance emphasizes current, demonstrable risks over hypothetical doomsday scenarios.What is the alignment problem, and why is it hard to solve universally?Alignment occurs when a system pursues goals in ways that match human intent and constraints. Even among humans, incentives misalign despite detailed contracts and oversight. For LLMs, vague objectives or incomplete constraints can lead to undesirable strategies. Globally, “aligned to whom?” becomes acute: different actors may train powerful systems aligned to their own interests, making a universal solution unlikely.What are the main ethical and legal issues in sourcing training data?Modern LLMs rely on massive corpora, much of which was not licensed with AI training in mind. Licenses often say “all rights reserved,” and legal frameworks lag behind technological practice. Concepts like fair use may apply in some jurisdictions, but outcomes are uncertain. Ethically, permissibility in law does not guarantee responsible practice; seeking consent, compensation, or agreements can build trust.What does “fair use” imply here, and how should practitioners act amid uncertainty?Fair use balances purpose, nature of the work, amount used, and market impact. Some factors may favor research and transformation; others cut against large‑scale commercial training that substitutes for the original work’s value. Because precedents are unsettled, practitioners should prefer owned or permissively licensed data when possible, document provenance, and pursue agreements where feasible, rather than relying solely on legal ambiguity.Is compensating or attributing creators feasible at scale, and why not just use public‑domain data?Paying all contributors is daunting: the number of works is vast, payments small, and attribution technically hard because tracing outputs back to specific training items is an open research problem. Public‑domain and open data help but are limited: much is old, culturally biased, or hard to verify as truly public domain. Promising directions include negotiated data partnerships, improved attribution research, constrained training or output sourcing, and compensation models tied to usage—though each has significant practical hurdles.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
How Large Language Models Work ebook for free
