The chapter opens by showing how a single malicious sentence can turn a well-tested chatbot into a data leak, making the core lesson clear: prompt security is prompt design security. Because language models process system instructions, retrieved content, and user input as one flat sequence, every prompt is vulnerable unless trusted and untrusted text are deliberately separated. The chapter builds on earlier prompt-structuring skills to explain that strong prompts are not only better written, but also harder to attack.
From there, the text explains three main attack patterns. Prompt injection works when hostile instructions are placed inside untrusted input and the model treats them as if they were part of the prompt’s authority. Jailbreaking relies on framing, roleplay, or persuasion to make the model relax its constraints, while instruction conflict appears when a prompt contains competing goals that users can exploit. The chapter emphasizes that vague safety language such as “be helpful” or “decline inappropriate requests” is weak because it leaves interpretation to the model under pressure.
To reduce risk, the chapter presents five hardening techniques: delimiter-based separation, explicit trust framing, closed-world instruction sets, explicit fallback responses, and minimal context exposure. Together, these make prompts more resilient by clarifying boundaries, narrowing behavior, and reducing what the model can reveal. However, the chapter also stresses that prompt-level defenses have limits: persistent adversarial pressure, sensitive data already in context, and model-to-model variability can still defeat them, so some problems require system-level controls beyond the prompt itself.
How a language model processes a prompt with no boundary between trusted and untrusted input
Unhardened vs. hardened prompts for a deployment notification assistant
Summary
Prompts are vulnerable by default because models treat the entire prompt as a single flat sequence. There is no built-in privilege hierarchy between developer instructions and user-supplied text, and structural separation is the only mechanism available at the prompt level for establishing that hierarchy.
Prompt injection is a structural attack. When untrusted input is concatenated with developer instructions without explicit boundaries, the model has no basis for preferring developer instructions over injected directives. Directives phrased to claim authority can override instructions written earlier in the sequence.
Jailbreaking is an interpretive attack. Vague, qualitative constraint language leaves the model room to resolve ambiguity in the user’s favor, especially under adversarial framing such as roleplay requests, persona substitution, or gradual escalation across turns.
Instruction conflict, in which a prompt contains two instructions that cannot both be satisfied simultaneously, is often written unintentionally but creates the same exploitable ambiguity as deliberately weak constraints. The model will resolve the conflict somehow, and not always as the developer intended.
Five hardening techniques address the most common failure modes at the prompt level: delimiter-based separation, explicit trust framing, closed-world instruction sets, explicit fallback instructions, and minimal context exposure. These techniques work best when applied together, so the failure of any single one does not open a gap.
Prompt-level defenses have a real ceiling. Persistent adversarial pressure across many turns, sensitive data that should not be in the context window at all, and model compliance variability across updates are conditions where a better prompt is not the right response.
FAQ
What is prompt security in this chapter’s context?Prompt security here means designing prompts so they resist manipulation at the prompt level. The chapter treats vulnerabilities and defenses as consequences of how the prompt is written, especially when untrusted input is present.Why is every prompt considered vulnerable by default?Because language models process the entire prompt as one continuous sequence with no built-in privilege hierarchy. System instructions and user-supplied text are not inherently separated, so any text in the prompt can influence behavior unless the prompt explicitly creates boundaries.What is the difference between trusted and untrusted input?Trusted input is text you control, such as system instructions, task framing, and few-shot examples. Untrusted input comes from external sources at runtime, such as user messages, retrieved documents, or API responses, and should be treated as potentially arbitrary text.What is prompt injection?Prompt injection is when an attacker embeds instructions in untrusted input to override or contradict the developer’s instructions. It works because the model may treat the injected text as part of the same instruction sequence.How does jailbreaking differ from prompt injection?Jailbreaking does not rely on embedding instructions inside the prompt structure. Instead, it uses framing, persuasion, or roleplay to convince the model to abandon or reinterpret its constraints.What is instruction conflict?Instruction conflict happens when a prompt contains two instructions that cannot both be satisfied. Under pressure, the model may choose the interpretation that seems more helpful, more recent, or more persuasive, which can be exploited by users.Why are vague safety phrases like “be helpful but safe” not enough?Because they require the model to interpret terms like “helpful,” “harmful,” or “inappropriate” in the moment. The chapter recommends precise, closed-world instructions instead of qualitative guidance that leaves room for interpretation.What are the five prompt-hardening techniques in this chapter?The five techniques are delimiter-based separation, explicit trust framing, closed-world instruction sets, explicit fallback instructions, and minimal context exposure. Together they reduce the attack surface of a production prompt.Why is explicit fallback behavior important?It gives the model a fixed response when a request is out of scope or conflicts with constraints. That reduces improvisation, which is often where adversarial framing can influence the model’s behavior.What are the limits of prompt-only security?Prompt-level defenses reduce risk, but they do not guarantee safety. They can fail under persistent adversarial pressure, when sensitive data is already in context, or when model behavior changes across versions, so some problems require system-level controls beyond the prompt.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Prompt Engineering in Practice ebook for free
