6 Are you measuring what you think you’re measuring?

This chapter explains how to turn abstract concepts into dependable measurements and challenges common shortcuts, using Net Promoter Score (NPS) as a cautionary example. It frames measurement through principles from measurement theory: agree on precise definitions, map concepts to observable indicators, pick appropriate scales or proxies, and recognize constraints. The analyst’s role is to align stakeholders on what matters, select feasible data collection strategies, and combine multiple indicators so a metric reflects the intended construct rather than convenience.

It compares self-report and behavioral measures: self-report is fast, flexible, and accesses internal states, but suffers from self-selection, wording and scale effects, and indirectness; behavioral data is direct, time-stamped, and integrable, but can be intrusive, costly, and bound by legal/ethical rules. The chapter lays out practical guidance for building higher-quality self-reports: craft unbiased questions (avoid ambiguity, leading or double-barreled framing, emotive language, jargon; manage order effects) and design response scales thoughtfully (balanced Likert options, meaningful midpoints or opt-outs, 5–7 points, consistent labeling and order). It also shows how to audit existing organizational data—understand sources and refresh cadence, meaning of timestamps, whether rows capture objects or actions, levels of aggregation, and data quality controls—illustrated by a case study blending survey and system data.

Finally, it equips readers to test whether a measure is dependable and truly about the intended construct. Reliability is checked via test–retest consistency over time, agreement between raters when judgments are subjective, and internal consistency when aggregating items (for example, using correlations and Cronbach’s alpha, refining items as needed). Validity ensures you are measuring what you think you’re measuring: content/face validity through expert and literature alignment; construct validity via convergent and discriminant relationships; and criterion validity via concurrent or predictive correlations with external outcomes. The overarching message is to triangulate across multiple measures, resist overreliance on a single number like NPS, and iteratively refine definitions, instruments, and datasets so metrics remain interpretable, comparable, and decision-ready.

The Net Promoter Score is used across industries and products.

NPS responses are categorized into three groups: promoters, passives, and detractors

Steps to defining and refining the concepts you want to measure

Can you measure the “best” in subjective areas like creative art?

The quality of data from self-report measures depends on numerous factors outside of the control of the researcher or analyst.

Varying methods of presenting the same response format

Examples of different question wording. How do you think these formats might impact the responses you receive?

Assessment of the NPS question on the appropriate Likert scale criteria

Each row in this dataset represents a record of a visit to a doctor’s office.

Each row in this dataset contains information about visitors and the pages they viewed. What level of aggregation exists in this dataset?

A few simple queries can help determine if there are issues with the quality of your data.

Test-retest reliability correlations are expected to be quite high across time periods (e.g., T1, T2, T3), often with an r value above 0.7.

Inter-rater reliability involves systematically evaluating the agreement between individuals where your measurement includes a subjective judgment.

Steps to calculating Cronbach’s alpha

• Self-report measures are collected directly from individuals, usually through questionnaires or surveys. These measures capture information about perceptions, attitudes, beliefs, or experiences. In analytics, self-report measures provide valuable insights into subjective experiences and allow for assessing factors that may not otherwise be observable.
• Designing accurate self-report measures (e.g., questionnaires) requires careful wording of questions and assessment items. Leading language, double-barreled statements, and negative wording can all reduce the quality of responses you receive.
• There are many ways to structure the response format to questions (e.g. Likert scales) that can impact your results. A scale should be balanced, and have a moderate number of response items (usually 5 or 7) to maximize the quality of responses.
• Behavioral measures are records of behaviors or actions captured by direct observation or recorded in a data warehouse as an interaction within an application or tool. These measures can offer direct insights into user and participant experiences without the bias associated with self-report measures.
• Reliability refers to the consistency, stability, and repeatability of a measurement. A reliable measure is expected to produce similar results under consistent conditions. Ensuring reliability in analytics is essential to produce accurate and trustworthy results that can inform decision-making and support evidence-based practices.
• Analysts typically assess three common forms of reliability: (1) test-retest reliability, which is an assessment of consistency in measurement responses over time, (2) inter-rater reliability, which is a measure of consistency between raters for qualitative evaluations, and (3) internal consistency, which is an assessment of the degree to which different variables measure the same underlying construct.
• Validity refers to the degree to which a measure accurately assesses the concept or process it’s designed to measure. Ensuring that your measures are valid is critical to building confidence in your stakeholders that your results are meaningful, accurate, and reflective of the true phenomenon you are investigating.
• Three forms of validity are essential to assess as part of your measurement: (1) content/face validity, which is an assessment of how comprehensively your measure captures the concept or phenomenon of interest, (2) construct validity, which is an assessment of whether you are measuring your intended theoretical construct, and (3) criterion validity, which assesses whether your measure is associated with other measures of similar constructs.

FAQ