Overview

Appendix B. Benchmarks and Datasets

This appendix curates benchmarks and datasets that enable fair, reproducible evaluation of optimization algorithms across continuous, combinatorial, geospatial, and machine-learning contexts. It explains why standardized test functions and problem instances matter, outlines typical properties used to stress algorithms (multimodality, convexity, separability, differentiability), and points to practical ways to obtain or synthesize data for experimentation. The material is geared toward helping practitioners select representative tasks, generate instances, and measure progress consistently.

For continuous optimization, it highlights canonical test functions used to probe solver behavior. Examples include Ackley (flat plateaus with a central basin), Bohachevsky (smooth, convex, separable), Bukin (rugged ridge with many local minima, non-differentiable, favoring derivative-free search), Gramacy & Lee (1-D with mixed local/global structure), and Griewank (many widespread local minima). By contrasting characteristics—convex versus non-convex, separable versus non-separable, multi- versus unimodal—the appendix motivates when gradient-based methods suffice and when black-box metaheuristics are preferable. It also notes that implementations can be coded from scratch or accessed through common Python libraries such as DEAP, pymoo, and PySwarms.

For combinatorial optimization, the appendix compiles widely used benchmark families—TSP, VRP, job-shop scheduling, assembly-line balancing, quadratic assignment, knapsack, set covering, and bin packing—so algorithms can be compared on standard instances. It then turns to geospatial data, describing spatial attributes and variables relevant to real-world planning (locations, infrastructure, demographics, mobility, environment, and emergency response) and illustrating how open sources and APIs such as OpenStreetMap, Overpass, and elevation services can be tapped for optimization inputs. Finally, for machine-learning-driven and neural combinatorial optimization, it notes that “datasets” are often generated problem instances, with examples including convex-hull point sets, TSP instances for sequence-model training, and large mobility datasets like taxi trip records; accompanying examples show how to prepare these for training and evaluation.

B.5 Data Folder

The data folder included in the GitHub repo of the book includes the following sample data:

  • ALBP: dataset for assembly line balancing problem covered in Chapter 6.
  • AdministrativeBoundaries: administrative boundaries of different regions of interest around the world in geojason format that can be used for map visualization.
  • BikeShare: Bike Share Toronto (TBS) ridership data that contains anonymized trip data, including: trip start day and time, trip end day and time, trip duration, trip start station, trip end station and user type (see Listing B.4).
  • CanadaTraffic: data from Statistics Canada’s dataset for Motor Vehicle Collisions in 2018 such as fatalities per 100,000 population, fatalities per Billion vehicles-kilometres, injuries per Billion vehicles-kilometres, fatalities per 100,000 licensed drivers and injuries per 100,000 licensed drivers (see Listing A.2).
  • OntarioHealth: health region of Ontario, Canada (see Listing A.2).
  • Police: Toronto Police Service Public Safety data (see Listing A.2).
  • PoliticalDistricting: Political districting data used in Chapter 8.
  • PtrNets: convex hull and tsp data used by pointer networks (Chapter 11).
  • TSP: travelling salesman problem instances.

FAQ

What are optimization test functions and why are they used?Optimization test functions (benchmark functions) are mathematical functions used to evaluate and compare optimization algorithms. They expose algorithm strengths and weaknesses under known properties (e.g., convexity, separability, differentiability, multimodality). Common examples include Ackley (flat outer region with a central hole), Bohachevsky (unimodal, convex, separable), Bukin (many local minima, non-differentiable), Gramacy & Lee (1D with multiple local minima), and Griewank (many widespread local minima in 1D–3D).
How do properties like separability, convexity, and differentiability affect solver choice?- Convex + differentiable: gradient-based methods typically perform well (e.g., Bohachevsky).
- Non-differentiable or black-box: prefer derivative-free methods (e.g., simulated annealing for Bukin).
- Separable: variables can be optimized independently, simplifying the search.
- Multimodal: algorithms need mechanisms to escape local minima (e.g., population-based or annealing).
When should I use derivative-free (black-box) solvers?Use derivative-free solvers when the objective is non-differentiable, noisy, or gradients are unavailable/expensive. For instance, the Bukin function is continuous but non-differentiable and multimodal, motivating derivative-free methods such as simulated annealing.
Where can I find code examples for optimization test functions?See Listing B.1 for implementations and references to Python frameworks: DEAP, pymoo, and PySwarms. Link: https://github.com/Optimization-Algorithms-Book/Code-Listings/blob/main/Appendix%20B/Listing%20B.1_Optimization_test_functions.ipynb
What benchmark datasets are available for the Traveling Salesman Problem (TSP) and Vehicle Routing Problem (VRP)?- TSP: https://github.com/coin-or/jorlib/tree/master/jorlib-core/src/test/resources/tspLib/tsp
- VRP: https://github.com/coin-or/jorlib/tree/master/jorlib-core/src/test/resources/tspLib/vrp and http://neumann.hec.ca/chairedistributique/data/
Where can I get benchmark instances for JSS, ALBP, QAP, Knapsack, SCP, and Bin Packing?- Job Shop Scheduling (JSS): http://people.brunel.ac.uk/~mastjjb/jeb/orlib/files/jobshop1.txt and http://people.brunel.ac.uk/~mastjjb/jeb/orlib/files/jobshop2.txt
- Assembly Line Balancing (ALBP): https://assembly-line-balancing.de/
- Quadratic Assignment (QAP): http://mistic.heig-vd.ch/taillard/problemes.dir/qap.dir/qap.html
- Knapsack: http://people.brunel.ac.uk/~mastjjb/jeb/orlib/mknapinfo.html
- Set Covering (SCP): http://people.brunel.ac.uk/~mastjjb/jeb/orlib/scpinfo.html
- Bin Packing: http://people.brunel.ac.uk/~mastjjb/jeb/orlib/binpackinfo.html
What kinds of geospatial data are relevant to optimization problems?Examples include locations of people, businesses, assets, and infrastructure; spatially distributed variables (traffic, health, demographics, weather); environmental change data (ecology, sea level, pollution, temperature); and data for emergency and disaster response (floods, epidemics, terrorism).
How can I fetch open geospatial data for my models?Use sources like OpenStreetMap (OSM), Overpass API, and Open-Elevation API. See Listing B.3 and B.4 for examples: https://github.com/Optimization-Algorithms-Book/Code-Listings/blob/main/Appendix%20B/Listing%20B.3_Geospatial_data.ipynb and https://github.com/Optimization-Algorithms-Book/Code-Listings/blob/main/Appendix%20B/Listing%20B.4_Geospatial_data_TBS.ipynb
What datasets are highlighted for neural combinatorial optimization?Besides classic benchmark instances, Listing B.5 includes: Convex Hull (point sets): https://tinyurl.com/2vb476hn; TSP datasets for Pointer Networks: https://tinyurl.com/2vb476hn; NYC TLC Trip Record Data (taxi trips): https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page. Link to listing: https://github.com/Optimization-Algorithms-Book/Code-Listings/blob/main/Appendix%20B/Listing%20B.5_ML_datasets.ipynb
How should I choose benchmarks and datasets to evaluate an optimization algorithm?Match benchmarks to your problem type and the algorithm’s assumptions: use test functions with known properties (convexity, separability, multimodality) to stress specific capabilities; select combinatorial instances (e.g., TSP, VRP, JSS) from recognized repositories for comparability; include realistic geospatial or domain datasets when relevant; and, for neural approaches, use standard NCO datasets from Listing B.5 alongside classic benchmarks from Listing B.2.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Optimization Algorithms ebook for free
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Optimization Algorithms ebook for free
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Optimization Algorithms ebook for free