18 Dress rehearsal patterns for building confidence
The chapter explains how BDD teams can keep confidence high when scenarios must run against real services such as databases or payment gateways. Stubs remain useful for fast development feedback, but integration-style “dress rehearsals” are still needed in continuous integration to reveal issues that only appear with actual dependencies. The main difficulty is balancing reliability, isolation, and speed when shared external resources are involved.
To make database-backed scenarios repeatable, the chapter focuses first on establishing a stable baseline and resetting it before each scenario rather than cleaning up afterward. A reset-before approach preserves failure evidence for debugging and avoids leaving behind data when tests abort unexpectedly. From there, the chapter develops a progression of techniques: starting with a clean, empty database, then improving efficiency by seeding shared reference data, and finally introducing optimizations such as skipping unnecessary resets when a scenario is known not to have changed relevant data.
The chapter also covers cases where teams do not fully control the underlying dataset. For large or fixed snapshot databases, it describes undoing changes instead of rebuilding everything, and finding suitable existing entities instead of creating them from scratch. Throughout, the guidance is pragmatic: measure before optimizing, use the simplest reliable reset strategy first, and only adopt more complex patterns when the performance or integration constraints justify the added maintenance risk.
Conflicting modifications when running tests in parallel using a shared database
The effort required to automate new scenarios depending on the quantity of data in the baseline
Comparison of Reset Before and Reset After patterns
Summary
- For developers to get fast, local feedback they use stub external services during scenario execution. To mitigate the risk of integration failures, the same scenarios must regularly run against the real, production-like services in the CI pipeline.
- A database is a common example of an external service, but this chapter’s patterns are not restricted to databases.
- To ensure repeatable test results, the starting state of a service should be effectively identical for every scenario. This starting state is called the baseline.
- The simplest baseline is the clean start, which is the default, empty state of the service.
- A seeded baseline is an optimization of the default, empty state that includes a basic set of entities that are commonly needed by scenarios.
- When an external service does not offer automation code flexible ways to manipulate its state, it is common to use a snapshot baseline taken from a running version of the service.
- To exercise specific behaviors, a scenario may need very specific entities to be present in a service. We consider this to be an aspect of test data management (TDM). The approach you take will depend on what affordances a service provides for automation code to manipulate its state.
- When the automation code creates the entities it needs on demand, it is following the Insert TDM approach. The automation code must be able to freely manipulate the service’s state.
- When the automation code cannot manipulate the service’s state, either a Query or Hardcoded TDM approach must be adopted. These approaches rely on the underlying baseline containing the entities needed for the scenarios to exercise the behavior they illustrate.
- There are three significant challenges that must be considered whenever scenarios are run against external services: isolation from other scenarios, repeatability of scenario results, and performance of the test suite.
- In the context of using external services, Isolation means that no changes made to service state by a scenario should impact any other scenario running in parallel.
- Strategies that address isolation include serial execution, resource pooling, and exclusive data partitions.
- Serial execution effectively restricts scenarios running in parallel that each use the service. This is the strategy chosen by WIMP in this chapter.
- Resource pooling requires multiple, identical instances of the service so that every running scenario has exclusive access to an instance. See chapter 21 for more on this topic.
- Exclusive data partitions depend on the service being able to partition state in such a way that changes made by one scenario are not seen by any other running scenario.
- Repeatability of scenario results is not guaranteed by using the serial execution strategy. The persistent nature of a service’s state may cause changes made during one scenario to impact the outcome of a scenario that follows it.
- Repeatability of scenario results can be ensured by resetting the service’s state to the baseline or, if available, by using exclusive data partitions.
- Although scenario execution using production-like services does not typically need to match the speed of execution using stubs, performance is still an issue to ensure timely feedback.
- Parallelization is often used to get timely feedback, but it has a significant impact on isolation. See chapter 21 for a discussion of relevant patterns.
- The design and overall size of the baseline (whether seeded or snapshot) can have a significant impact on the performance of resets in between scenarios.
- The Reset Before pattern improves on the often used restore-after approach by making sure that the reset takes place no matter how the previously executed scenario terminated.
- The Reset Before pattern can help diagnose scenario failures during development.
- The Clean Start pattern ensures that every scenario starts off with a service in a default, empty state.
- The Seeded Data pattern preserves performance and reduces the amount of explicit context setup that each scenario must perform.
- The Skippable Reset pattern preserves performance by ensuring that only essential reset work is performed between scenarios.
- The Skippable Reset pattern, when combined with the Restore Before pattern, may need to preserve some state between scenarios.
- The Undo Changes pattern can be useful when working with snapshot baselines to improve reset performance.
- The record keeping needed for the Undo Changes pattern can be onerous, especially if a reset requires more than simple deletion of data added to the service’s state.
- The Entity Finder pattern encompasses the Query and Hardcoded TDM approaches for using snapshot baselines.
Effective Behavior-Driven Development ebook for free