3 Testing and Evaluating Hypotheses

This chapter shows how to move from broad questions to testable, evaluable hypotheses and why early alignment on the question, hypothesis, and deliverable prevents downstream rework. It adopts a probabilistic mindset, emphasizing that analysts gather evidence rather than “proof,” and that clarity about assumptions, uncertainty, and decisions is central to rigorous, business-relevant analysis. The material blends research-methods fundamentals with applied, real-world examples to frame testing and evaluation in organizations where data, context, and constraints are imperfect.

Readers learn to inform hypotheses through a three-pronged approach—targeted research, stakeholder interviews, and exploring available organizational data—often coupled with opportunity sizing to estimate impact and prioritize work. The chapter formalizes hypothesis construction (clear independent and dependent variables, concise declarative expectations, and, when possible, directional predictions) and encourages quantification of expected effects to set concrete evaluation criteria, borrowing intuition from Bayesian thinking about plausible effect sizes. A running case study follows an analyst at a non-profit who synthesizes literature, elicits stakeholder goals, mines internal evidence, and translates all of this into explicit, measurable hypotheses about volunteer registration and downstream outcomes.

Methods of gathering evidence are organized into descriptions, associations, and experiments, with practical guidance and cautions: descriptive statistics inform but do not imply generalization; correlations aid planning yet do not establish causation; experiments (including quasi-experiments) enable stronger causal claims when design conditions are met. The chapter outlines common study designs—between-subjects, within-subjects (repeated measures), cohort, and longitudinal—and connects them to typical organizational programs: basic/applied research, scaled A/B testing, and program evaluation. Throughout, it stresses disciplined communication (“evidence,” not “proof”), stakeholder guidance on interpretation, and iterative follow-up, culminating in a complete evaluation plan that integrates descriptive trends, associations, hypothesis tests, and longitudinal tracking to inform decisions and improve outcomes over time.

Many frustrating situations can be avoided by aligning on the question, hypothesis, and expected deliverable.

Distributions showing a hypothetical group difference between the treatment and control group.

Time series plot of daily reports of rat sightings in NYC.

Time series plot of total monthly rat sightings in NYC.

Time series plot of mean and median monthly rat sightings in NYC.

Pearson’s Correlations between the number of daily rat sightings and weather parameters.

Scatterplot of daily high temperatures vs. daily rat sightings.

A hypothesized causal relationship between two independent and one dependent variable.

Quasi-experiments have a similar design to controlled experiments without random group assignment.

Between-subjects comparisons evaluate differences between mutually exclusive characteristics or assignments.

Within-subjects comparisons evaluate differences before and after treatment.

Cohort comparisons evaluate meaningful cohort groups over time.

Longitudinal comparisons evaluate participants over months, years, or even lifetimes.

Example of an A/B/C group comparison with minor changes to a website.

• Developing a data-informed and quantifiable hypothesis involves synthesizing peer-reviewed and public research, gathering stakeholder information, and conducting foundational analyses of available data at your organization.
• Research the process of investigating a topic as a study or experiment to gain new information about that topic. The goal is to accumulate information over time, enabling you to build expertise and better inform decisions, practice, and policy.
• Data can be evaluated as one of the following:
• Descriptive information, which seeks to describe only existing information in a dataset without making inferences, predictions, or quantifying the relationships between variables.
• Correlations, which measure a non-causal linear association between two variables. Correlations range from -1 to 1, with values further from 0 indicating a stronger relationship.
• Causal/predictive analyses aims to determine if one phenomenon (your independent variable) causes something else to happen (your dependent variable). This is usually tested as an experiment under controlled settings to be able to confidently isolate an independent variable as the cause of the dependent variable.
• Experimental designs comparing differences between groups are often set up as between-subjects comparisons (e.g., random assignment between an experiment and control group) or within-subjects designs (e.g., a single experiment group compared before and after treatment). These standard designs allow for comparison across vast arrays of research.
• Cohort comparisons (e.g., participants grouped by age and tracked for the duration of an experiment) and longitudinal studies (tracking participants over long periods) are less common due but highly valuable in many domains of study, research, and work.
• Individual study designs usually become part of more extensive research programs within an organization.
• Basic and applied research programs seek to contribute to the overall knowledge base about a specific topic. These are common in academic settings.
• A/B testing programs seek to continually improve on a product and/or business outcome through a continuous program of concurrent or successive experiments run on large numbers of users for short periods of time. These programs are common in marketing and product analytics, and are primarily run in business settings.
• Program evaluations are typical in non-profit and government environments. These seek to evaluate the efficacy of new and ongoing programs for the populations they serve, continually improving their ability to achieve a specific goal (e.g., reducing rates of cancer).

FAQ