1 Why we need a new way to test AI opportunities
Artificial intelligence offers major opportunities to create predictions, recommendations, decisions, and content that improve business outcomes, but the chapter argues that most organizations still struggle to turn AI ambition into durable impact. The core problem is not usually a lack of technology, data, or enthusiasm; it is the absence of a disciplined way to test whether an AI idea truly fits a business problem, user need, organizational reality, and data environment before significant investment begins. The authors introduce the AI Road Test as a structured, evidence-based process designed to help teams evaluate AI opportunities early, avoid predictable failure modes, and make better go, no-go, or pivot decisions.
The chapter illustrates these risks through Zillow Offers, where a strong data asset and successful valuation algorithm were extended into a business model that required different predictions, stronger data governance, and greater operational readiness. Zillow’s failure is presented less as a failure of AI itself and more as a failure to translate analytics into a viable business outcome: biased seller inputs, volatile market conditions, growth pressure, weak guardrails, and a mismatch between model design and business need all contributed to large losses. From this and other examples, the chapter identifies seven recurring causes of AI project failure, including weak business relevance, poor fit with users, lack of stakeholder support, insufficient human resources, inadequate infrastructure, poor-quality or biased data, and the wrong analytical approach.
To manage these risks, the chapter organizes AI Road Testing around three lenses: the user, the enterprise, and data analytics. It emphasizes that testing should happen before a proof of concept because a PoC may validate technical feasibility while leaving unresolved questions about problem framing, user adoption, economics, scalability, interpretability, buy-versus-build choices, and whether the model addresses correlation or causality. The authors also argue that early testing is economically attractive: although it requires time upfront, it prevents wasteful development, speeds execution by clarifying the target state, and improves the odds that AI initiatives reach sustained management and value creation. A hybrid of stage-gate discipline and agile iteration is recommended to combine governance, adaptability, and impact.
The three lenses—user, enterprise, and data analytics—capture all seven major reasons AI projects fail
The AI Road Test process combines critical steps (e.g. investigating user needs), checklists (e.g. problem statement template) and decision rules (analytical complexity no more than one level above current maturity level).
The AI Road Test sits between use case prioritization and development in the AI project lifecycle.
details the percentage of time spent by the project team during the phases of a typical AI project.
Summary
- AI can go spectacularly wrong if unproperly tested. Zillow’s failed use of AI-generated property valuations in its homebuying division, known as Zillow Offers, resulted in losses exceeding $300 million and 2,000 jobs.
- As AI investments and adoption soar, more than 80% of analytics projects fail to deliver value, and the situation appears to be getting worse.
- The main causes of failure are well known and can be analyzed through three lenses: user attractiveness, enterprise viability, and data and analytics feasibility.
- We have a proven methodology for each of these three lenses to guide the assessment: Design Thinking for the user lens, hypothesis-driven problem solving for the enterprise lens, and CRISP-DM—or a similar AI project management methodology—for the data and analytics lens.
- The best way to avoid these risks is not to rely on intuition, but to use a structured process that counteracts human bias. Our proposed process is the AI Road Test.
- Identifying and mitigating flaws in AI projects should occur before any proof of concept (PoC); PoCs cannot correct issues related to scope, approach, or the sunk cost fallacy.
- The small investment of time required for the AI Road Test can generate significant returns through higher success rates, and faster time to value.
FAQ
Why do so many AI projects fail to deliver lasting impact?
AI projects often fail because they do not bridge the gap between opportunity and execution. Common reasons include solving problems that are not business-relevant, ignoring user and stakeholder needs, exceeding the organization’s analytics maturity or infrastructure capabilities, relying on poor-quality or biased data, and choosing an inadequate analytical approach.
What is the AI Road Test?
The AI Road Test is a structured, evidence-based process for evaluating AI opportunities before committing significant resources to development, procurement, or a proof of concept. It helps teams assess whether an AI idea is desirable for users, viable for the enterprise, and feasible from a data and analytics perspective.
Why should AI ideas be tested before building a proof of concept?
A proof of concept mainly tests technical feasibility, but it does not necessarily validate whether the problem is worth solving, whether users will adopt the solution, whether the organization can scale it, or whether the economics make sense. Testing before a PoC helps avoid wasting resources on ideas that are technically impressive but commercially or operationally weak.
What lessons does the Zillow Offers failure provide for AI projects?
Zillow Offers shows that strong data and a successful algorithm do not automatically create a successful business outcome. Zillow’s model was good at estimating current home values, but the business required forecasting future resale values under volatile market conditions. The case also highlights the dangers of biased input data, weak governance, aggressive growth incentives, and insufficient holistic testing before scaling.
What are the seven main causes of failure in AI projects?
The chapter identifies seven recurring causes: the AI solution does not solve a business-relevant problem; it does not match user needs or AI literacy; stakeholders do not support the project; the solution cannot be maintained or scaled due to lack of human resources; the technical infrastructure cannot scale sustainably; the data is low-quality or biased; and the analytical approach is inappropriate for the problem or data.
What are the three lenses of AI Road Testing?
The three lenses are User, Enterprise, and Data Analytics. The User lens checks whether the solution fits real user needs and adoption requirements. The Enterprise lens checks business value, stakeholder support, resources, maturity, and scalability. The Data Analytics lens checks data quality, feasibility, and whether the analytical approach is appropriate.
How does the AI Road Test help reduce cognitive bias?
The AI Road Test reduces bias by using structured decision rules, checklists, reflection points, and predefined GO / NO GO / PIVOT criteria. This helps teams avoid traps such as overconfidence, confirmation bias, sunk-cost fallacy, and misaligned incentives, especially when projects become complex, expensive, or politically sensitive.
What are the main stages of the AI Road Test process?
The process includes moments of synthesis and decision-making around the AI Statement, Problem Statement, Approach Statement, and Business Case. Between these decision points, teams investigate business and user needs, identify feasible solutions, and test whether the proposed approach is attractive, viable, and worth pursuing.
Where does the AI Road Test fit in the AI project lifecycle?
The AI Road Test sits after use case prioritization and before development, acquisition, rental, or proof-of-concept work. Its purpose is to assess and plan an AI opportunity in enough detail to decide whether to proceed, pivot, or stop before larger investments are made.
What is the economic value of testing AI opportunities early?
Early testing requires time, but the chapter argues it produces strong returns by preventing unnecessary development work on weak ideas. Based on the authors’ assumptions, every dollar invested in AI Road Testing can generate more than three dollars in short-term return, mainly by avoiding costly PoCs and development efforts for projects unlikely to succeed.
Look Before You Leap ebook for free
