table of content

1 Introduction to agents and their world

1.1 Defining agents

1.2 Understanding the component systems of an agent

1.3 Examining the rise of the agent era: Why agents?

1.4 Peeling back the AI interface

1.5 Navigating the agent landscape

Summary

2 Harnessing the power of large language models

2.1 Mastering the OpenAI API

2.1.1 Connecting to the chat completions model

2.1.2 Understanding the request and response

2.2 Exploring open source LLMs with LM Studio

2.2.1 Installing and running LM Studio

2.2.2 Serving an LLM locally with LM Studio

2.3 Prompting LLMs with prompt engineering

2.3.1 Creating detailed queries

2.3.2 Adopting personas

2.3.3 Using delimiters

2.3.4 Specifying steps

2.3.5 Providing examples

2.3.6 Specifying output length

2.4 Choosing the optimal LLM for your specific needs

2.5 Exercises

Summary

3 Engaging GPT assistants

3.1 Exploring GPT assistants through ChatGPT

3.2 Building a GPT that can do data science

3.3 Customizing a GPT and adding custom actions

3.3.1 Creating an assistant to build an assistant

3.3.2 Connecting the custom action to an assistant

3.4 Extending an assistant’s knowledge using file uploads

3.4.1 Building the Calculus Made Easy GPT

3.4.2 Knowledge search and more with file uploads

3.5 Publishing your GPT

3.5.1 Expensive GPT assistants

3.5.2 Understanding the economics of GPTs

3.5.3 Releasing the GPT

3.6 Exercises

Summary

4 Exploring multi-agent systems

4.1 Introducing multi-agent systems with AutoGen Studio

4.1.1 Installing and using AutoGen Studio

4.1.2 Adding skills in AutoGen Studio

4.2 Exploring AutoGen

4.2.1 Installing and consuming AutoGen

4.2.2 Enhancing code output with agent critics

4.2.3 Understanding the AutoGen cache

4.3 Group chat with agents and AutoGen

4.4 Building an agent crew with CrewAI

4.4.1 Creating a jokester crew of CrewAI agents

4.4.2 Observing agents working with AgentOps

4.5 Revisiting coding agents with CrewAI

4.6 Exercises

Summary

5 Empowering agents with actions

5.1 Defining agent actions

5.2 Executing OpenAI functions

5.2.1 Adding functions to LLM API calls

5.2.2 Actioning function calls

5.3 Introducing Semantic Kernel

5.3.1 Getting started with SK semantic functions

5.3.2 Semantic functions and context variables

5.4 Synergizing semantic and native functions

5.4.1 Creating and registering a semantic skill/plugin

5.4.2 Applying native functions

5.4.3 Embedding native functions within semantic functions

5.5 Semantic Kernel as an interactive service agent

5.5.1 Building a semantic GPT interface

5.5.2 Testing semantic services

5.5.3 Interactive chat with the semantic service layer

5.6 Thinking semantically when writing semantic services

5.7 Exercises

Summary

6 Building autonomous assistants

6.1 Introducing behavior trees

6.1.1 Understanding behavior tree execution

6.1.2 Deciding on behavior trees

6.1.3 Running behavior trees with Python and py_trees

6.2 Exploring the GPT Assistants Playground

6.2.1 Installing and running the Playground

6.2.2 Using and building custom actions

6.2.3 Installing the assistants database

6.2.4 Getting an assistant to run code locally

6.2.5 Investigating the assistant process through logs

6.3 Introducing agentic behavior trees

6.3.1 Managing assistants with assistants

6.3.2 Building a coding challenge ABT

6.3.3 Conversational AI systems vs. other methods

6.3.4 Posting YouTube videos to X

6.3.5 Required X setup

6.4 Building conversational autonomous multi-agents

6.5 Building ABTs with back chaining

6.6 Exercises

Summary

7 Assembling and using an agent platform

7.1 Introducing Nexus, not just another agent platform

7.1.1 Running Nexus

7.1.2 Developing Nexus

7.2 Introducing Streamlit for chat application development

7.2.1 Building a Streamlit chat application

7.2.2 Creating a streaming chat application

7.3 Developing profiles and personas for agents

7.4 Powering the agent and understanding the agent engine

7.5 Giving an agent actions and tools

7.6 Exercises

Summary

8 Understanding agent memory and knowledge

8.1 Understanding retrieval in AI applications

8.2 The basics of retrieval augmented generation (RAG)

8.3 Delving into semantic search and document indexing

8.3.1 Applying vector similarity search

8.3.2 Vector databases and similarity search

8.3.3 Demystifying document embeddings

8.3.4 Querying document embeddings from Chroma

8.4 Constructing RAG with LangChain

8.4.1 Splitting and loading documents with LangChain

8.4.2 Splitting documents by token with LangChain

8.5 Applying RAG to building agent knowledge

8.6 Implementing memory in agentic systems

8.6.1 Consuming memory stores in Nexus

8.6.2 Semantic memory and applications to semantic, episodic, and procedural memory

8.7 Understanding memory and knowledge compression

8.8 Exercises

Summary

9 Mastering agent prompts with prompt flow

9.1 Why we need systematic prompt engineering

9.2 Understanding agent profiles and personas

9.3 Setting up your first prompt flow

9.3.1 Getting started

9.3.2 Creating profiles with Jinja2 templates

9.3.3 Deploying a prompt flow API

9.4 Evaluating profiles: Rubrics and grounding

9.5 Understanding rubrics and grounding

9.6 Grounding evaluation with an LLM profile

9.7 Comparing profiles: Getting the perfect profile

9.7.1 Parsing the LLM evaluation output

9.7.2 Running batch processing in prompt flow

9.7.3 Creating an evaluation flow for grounding

9.7.4 Exercises

Summary

10 Agent reasoning and evaluation

10.1 Understanding direct solution prompting

10.1.1 Question-and-answer prompting

10.1.2 Implementing few-shot prompting

10.1.3 Extracting generalities with zero-shot prompting

10.2 Reasoning in prompt engineering

10.2.1 Chain of thought prompting

10.2.2 Zero-shot CoT prompting

10.2.3 Step by step with prompt chaining

10.3 Employing evaluation for consistent solutions

10.3.1 Evaluating self-consistency prompting

10.3.2 Evaluating tree of thought prompting

10.4 Exercises

Summary

11 Agent planning and feedback

11.1 Planning: The essential tool for all agents/assistants

11.2 Understanding the sequential planning process

11.3 Building a sequential planner

11.4 Reviewing a stepwise planner: OpenAI Strawberry

11.5 Applying planning, reasoning, evaluation, and feedback to assistant and agentic systems

11.5.1 Application of assistant/agentic planning

11.5.2 Application of assistant/agentic reasoning

11.5.3 Application of evaluation to agentic systems

11.5.4 Application of feedback to agentic/assistant applications

11.6 Exercises

Summary

Appendices

Appendix A: Accessing OpenAI large language models

A.1 Accessing OpenAI accounts and keys

A.2 Azure OpenAI Studio, keys, and deployments

Appendix B: Python development environment

B.1 Downloading the source code

B.2 Installing Python

B.3 Installing VS Code

B.4 Installing VS Code Python extensions

B.5 Creating a new Python environment with VS Code

B.6 Using VS Code Dev Containers (Docker)

Overview

9 Mastering agent prompts with Prompt Flow

This chapter presents a disciplined approach to building AI agents by moving from ad-hoc prompt tinkering to systematic iteration and evaluation. It clarifies the difference between a single prompt and a full agent profile—an orchestrated set of prompts and components such as persona, special instructions, tools, memory, knowledge, reasoning, evaluation, and feedback. Microsoft’s Prompt Flow is introduced as the practical workbench for this process, providing templated prompts (via Jinja2), configurable LLM settings, reproducible flows, and scalable execution to develop and refine robust agent profiles.

Readers are guided through setting up Prompt Flow in VS Code, connecting to an LLM, and configuring critical parameters (API type, model/deployment, temperature, token limits). The chapter shows how system versus user prompts, clear instructions, and persona shaping can materially affect outputs; how to run flows locally or in containers; and how to create prompt variants to compare templates, models, and parameter settings. It also demonstrates batch execution over JSONL input sets, parsing evaluator outputs into structured data, and capturing run artifacts so that experiments remain consistent, repeatable, and auditable.

Evaluation centers on rubrics and grounding: define explicit criteria and scales aligned to task objectives, assess how well responses adhere to them, and automate scoring with a second LLM to deliver consistent, unbiased assessments. Scores are aggregated across runs to compare variants and surface the strongest profiles. A key finding is that injecting key inputs into the system prompt produced more aligned recommendations, illustrating how Prompt Flow enables data-driven iteration toward an optimal profile. The chapter concludes by encouraging continued experimentation—adding criteria, exploring new use cases, and benchmarking alternative models—within this systematic evaluation loop.

The differences in applying prompt engineering and iterating

OpenAI prompt engineering strategies, broken down by agent component

The messages to and from an LLM conversation and the iteration of messages

The effect of adding a system prompt to our previous conversation

The systemic method of prompt engineering

The component parts of an agent profile

Creating a new Prompt Flow LLM connection

Setting the connection information for your LLM resource

Opening the visual Prompt Flow editor

Setting the LLM connection details

Setting the Inputs and starting the flow

Opening the prompt jinja2 template and examining the parts of the profile/prompt

Building and starting the flow as a local app

Running the flow as a local web application

The recommender, with variations in flow and expanded inputs

The user interaction options for interfacing with the agent profile to prime inputs into the agent profile

Side-by-side comparison of variant profile templates for the recommender

Running both prompt variations at the same time

The evaluation prompt, with each of the parts of the rubric outlined

Opening the parsing_results.py file in VS Code

Loading the bulk JSONL file to run the flow on multiple input variations

Running the batch run of inputs

An opening run visualization and an examination of a batch run

Looking at the evaluate_groundings flow used to ground recommendation runs

Loading a previous run to be grounded and evaluated

Visualizing the metrics for multiple runs and comparing them

Summary

An agent profile consists of several other component prompts that can drive functions such as actions/tools, knowledge, memory, evaluation, reasoning, feedback and planning.
Prompt Flow can be used to evaluate an agent's component prompts.
Systemic prompt engineering is an iterative process evaluating a prompt and agent profile.
The Test Changes Systematically strategy describes iterating and evaluating prompts, and system prompt engineering implements this strategy.
Agent profiles and prompt engineering have many similarities. We define an agent profile as the combination of prompt engineering elements that guide and help an agent through its task.
Prompt Flow is an open-source tool from Microsoft that provides several features for developing and evaluating profiles and prompts.
An LLM connection in Prompt Flow supports additional parameters, including temperature, stop token, max tokens, and other advanced parameters.
LLM blocks support prompt and profile variants, which allow for evaluating changes to the prompt/profile or other connection parameters.
A rubric applied to an LLM prompt is the criteria and standards a prompt/profile must fulfill to be grounded. Grounding is the scoring and evaluation of a rubric.
Prompt Flow supports running multiple variations simultaneously in single or through batches.
In Prompt Flow, an evaluation flow is run after a generative flow to score and aggregate the results. The Visualize runs window can compare the aggregated criteria from scoring the rubric across multiple runs.

FAQ

What is “systematic prompt engineering,” and why does it matter?

It’s the disciplined practice of iterating and evaluating prompts with clear criteria instead of ad‑hoc trial and error. Systematic testing helps you isolate changes, compare variants, measure impact (e.g., quality, cost, tokens), and converge on reliable, repeatable prompts and agent profiles.

What is Prompt Flow, and how does it help with agent prompts?

Prompt Flow (PF) is a Microsoft tool (VS Code extension and OSS) for building, running, and evaluating LLM workflows. It lets you: design flows visually, template prompts, connect to LLMs, run variants side-by-side, batch test at scale, log metrics, and compare runs to pick the best profile.

How is an agent profile different from a single prompt or persona?

An agent profile bundles multiple components: system persona, special instructions, tools/actions, knowledge, memory, reasoning, evaluation, planning, and feedback. A single prompt is just one message; a profile orchestrates all the prompts and settings that define the agent’s behavior end-to-end.

How do I set up my first Prompt Flow in VS Code?

- Install VS Code, the Prompt Flow extension, and a Python virtual environment
- pip install promptflow promptflow-tools
- Create an LLM connection (OpenAI or Azure OpenAI) in the PF extension
- Open a flow (flow.dag.yaml), set the connection/model in the LLM block, and click Run

Which LLM parameters matter most in Prompt Flow?

- connection: the service/resource to use
- api: chat vs completion API
- model: model or deployment name
- temperature: randomness (0 = deterministic, 1 = diverse)
- max_tokens: response length, affects cost and latency
- stop and advanced options (top_p, penalties) for further tuning

How do Jinja2 templates fit into profiles?

PF uses Jinja2 files to define prompt content (system/user messages) and inject inputs. Templates make profiles modular, readable, and easy to version. You can create multiple templates (variants) to test different instruction styles or where inputs are injected (system vs user).

Can I deploy a Prompt Flow as an app or API?

Yes. From the Visual Editor, use Build to generate a local web app/API configuration, then Start application. You can also containerize flows (Docker) for service deployment.

What are rubrics and grounding in prompt evaluation?

- Rubric: a clear set of criteria (e.g., subject/format/genre fit) with a scoring scale (e.g., 1–5) and descriptions for each level
- Grounding: checking how well outputs align with the rubric and context to make evaluation objective, consistent, and comparable across runs

How do I use an LLM to evaluate another LLM’s output?

Add an evaluation block with a rubric-based prompt that scores each response against your criteria. PF runs the recommender, then feeds its outputs to the evaluator prompt, which returns structured scores. Optionally parse results, average scores, and log metrics for comparison.

How do I batch test and compare prompt/profile variations?

- Prepare a JSONL file of input cases (subject, format, genre, custom)
- Use PF’s batch run to process all cases across variants in parallel
- Parse and aggregate evaluation scores (Python blocks) to produce metrics
- Open Batch Run History and use Visualize Runs to compare metrics across variants and pick the best-performing profile

with subscription

$24.99

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more