3 Classifying user intent
This chapter introduces intent classification as a practical LM application and uses airline customer service messages as the running example. The core problem is to map many different ways a user might ask for something into a smaller, predefined set of intents, such as booking a flight or asking about airport information. It emphasizes that these utterances are usually short and that the chapter focuses first on building a simple baseline before later steps like evaluation and optimization.
To prepare for the classifier, the chapter walks through downloading the ATIS dataset, inspecting its structure, and cleaning the labels. It reframes the original short intent names into clearer, more descriptive labels so the language model has better semantic clues during zero-shot classification. It also simplifies the task by dropping multi-label cases at first, leaving a single-label, multi-class, mostly static classification problem with a manageable set of intents.
The DSPy solution is built from three main pieces: an LM, a signature, and a module. The signature defines the inputs and output fields, while the Predict module turns that specification into a reusable program that generates prompts automatically and returns structured predictions. The chapter then compares this to calling the OpenAI API directly, showing that DSPy reduces prompt-construction boilerplate, adapts more easily across providers, and makes outputs easier to handle, while also covering dynamic labels, multi-label classification, and tools for inspecting prompt and response history for debugging and cost tracking.
The three main stages of development with prompt programming. Almost all DSPy projects start with creating a baseline DSPy program (in this example, a classifier), followed by creating a framework to evaluate the program, and then optimizing the program.
Summary
- A classification task may be binary or multi-class, depending on how many classes are defined.
- A classification task may also be either a single-label or multi-label problem, depending on if it’s possible to assign multiple labels to each item. Single-label problems are simpler but may be too limiting.
- A classification problem may have either static or dynamic labels.
- When using DSPy, creating a baseline classifier involves specifying a language model, a DSPy signature, and a DSPy Module.
- DSPy creates a prompt by transforming the signature and module into the system prompt and user prompt.
- DSPy signatures come in two flavors: a short string-based format and a class-based format.
- Using class-based signatures allows us to specify docstrings and descriptions of the input and output fields.
- DSPy modules are initialized by DSPy signatures and are responsible for running the queries.
- DSPy allows for simpler, cleaner code than working with the LMs’ APIs directly.
- Inspecting the prompts created by DSPy is made easy by running lm.inspect_history().
- Calling lm.history or get_lm_usage() allows you to see information about token usage and costs.
FAQ
What is intent classification in the context of chatbot applications?
Intent classification is the task of mapping a user’s message to one label from a predefined set of actions or intents. In chatbot applications, it helps determine what the user wants so the system can respond appropriately, such as turning on a light or booking a flight.Why does the chapter use short user utterances instead of long paragraphs?
Short utterances are easier to classify because they usually express a single intent clearly. Longer text is harder because it may contain multiple intents, more ambiguity, and extra unrelated information.What is the difference between binary and multi-class classification?
Binary classification has two possible classes, while multi-class classification has three or more. In this chapter, intent classification is treated as a multi-class problem because there are many possible user intents.Why are the ATIS intent labels renamed to more descriptive names?
The original labels are short and may not be meaningful to the language model. Renaming them to clearer labels like “Flight Booking Request” gives the model better context and can improve classification accuracy, especially in zero-shot prompting.Why does the chapter drop multi-label intent rows from the dataset at first?
To simplify the baseline classifier, the chapter treats the problem as single-label classification. Rows with combined intents are removed so the model only has to predict one intent per utterance during the initial setup.What are the three core components needed to build a DSPy classifier?
You need a language model, a DSPy signature, and a DSPy module. The LM performs the generation, the signature defines inputs and outputs, and the module executes the prompt-based program.What is the purpose of a DSPy signature?
A DSPy signature defines what inputs the model receives and what outputs it should produce. It can also include a docstring and field descriptions to help guide the model more clearly.How is DSPy different from calling an LM API directly?
DSPy automates prompt construction and output formatting, making code cleaner and easier to maintain. Direct API use often requires manually building prompts, managing roles, and parsing more complex responses yourself.How can DSPy help with dynamic or unknown intents?
You can instruct the model to return None when no label matches well, which is useful for dynamic classification. This helps handle cases where the set of possible labels may change over time or where a user request does not fit existing intents.How do you inspect the prompt and response history in DSPy?
You can use methods likelm.inspect_history(n=1) to view the most recent prompt and response, or lm.history to see detailed call records. DSPy also provides tools like dspy.inspect_history() and usage tracking for monitoring token use and cost.
Building LLM Applications with DSPy ebook for free
