Overview

5 Selecting and evaluating AI tools

This chapter offers a practical roadmap for choosing AI tools and rigorously evaluating how well they work for your task. It introduces clear distinctions among AI options and pairs them with decision criteria grounded in goals, constraints, and capabilities. It then outlines a disciplined evaluation protocol and the most common performance measures, emphasizing how to avoid overestimating results and how to align evaluation with business objectives.

On tool selection, it contrasts proprietary and open-source AI: proprietary tools emphasize ease, support, and fast starts (often via apps or usage-based APIs), while open-source options offer transparency, self-hosting, and deep customization—albeit with higher infrastructure burden, potential security risks, and licensing limits. The chapter also weighs off-the-shelf use (prompt engineering, few-shot prompting, and RAG as first resorts) against fine-tuning for highly tailored outputs or to make smaller, self-hosted models competitive; it flags overfitting as the main risk and notes common controls such as early stopping, limited parameter updates (e.g., LoRA), regularization, and dropout. Finally, it distinguishes customer-facing apps (task-focused, end-user friendly) from foundation models (general-purpose building blocks accessed via API), advising the former for direct task assistance and the latter when engineering custom solutions.

On evaluation, the chapter prescribes a three-dataset protocol: train on a training set, choose among models using a validation set, and reserve a separate, one-time test set for a final, unbiased check. It cautions against data leakage and “cheating,” especially with LLMs whose training data may contain public answers, and recommends creating validation/test data that truly reflect unseen cases. For measurement, it covers accuracy for correctness; precision and recall for retrieval tasks, noting their trade-off and encouraging business-aligned targets (e.g., fix minimum recall, compare precision); and MAE/RMSE for numeric prediction error. The overarching advice is to benchmark with care, separate modeling and assessment decisions, and select both tools and metrics to match real-world needs.

Summary

Proprietary AI is a good choice when you need an easy-to-use, done-for-you solution.
Open-source AI is a good choice when you need to self-host or customize models.
If AI isn’t working quite the way you expect, or you need to customize it, it’s usually recommended to still use off-the-shelf models and enhance your prompts. If that doesn’t work, you may want to fine-tune a model to your own data. Fine-tuning is also a good option when you prefer to use a smaller model.
Customer-facing AI apps are designed to be friendly and useful to end users. They’re powered by foundation models behind the scenes, which are large, general-purpose AI models which you can use to build your own AI-based apps.
Make sure to use a validation set (with data not present in the training set) to compare and select models. You should also perform a sanity check afterward using a separate test set, once you’ve selected your favorite model. Do not use the test set twice.
The accuracy of a model measures how often it performs a task correctly. Measures like precision and recall are used for information retrieval (such as fetching relevant legal cases according to a query from a much larger pool of legal cases). Precision and recall cannot be used by themselves; they must be combined in a way that matches business preferences about their relative importance. You can use the mean absolute error (MAE) or the root mean squared error (RMSE) to evaluate the performance of a model at predicting a number (such as the temperature).

FAQ

What’s the difference between proprietary and open‑source AI?

Proprietary AI keeps code and model parameters private and is typically accessed via consumer apps (e.g., ChatGPT) or usage‑billed APIs. Open‑source AI publishes model code and weights, usually allowing modification and optional self‑hosting. Note that many “open” models still don’t disclose their training data.

When should I choose proprietary AI vs open‑source AI?

Choose proprietary AI for done‑for‑you solutions, fast start‑up, minimal ML/DevOps needs, and small‑scale economics. Choose open‑source AI for self‑hosting (governance and data control), large‑scale cost efficiency, and deep customization such as fine‑tuning. The quality gap is narrowing and many open models are highly capable.

What licensing, transparency, and security issues should I expect with open‑source models?

Licenses can restrict use (e.g., user caps, no using outputs to train competing LLMs). Training datasets and RLHF data are often undisclosed. Self‑hosting adds infra cost and operational risk, and malicious models have appeared in the wild—so audit sources and artifacts. The upside is greater code visibility and data governance.

What’s the difference between off‑the‑shelf prompting and fine‑tuning?

Off‑the‑shelf uses prompt engineering (e.g., few‑shot examples, RAG for context, style cues) without changing model weights. Fine‑tuning changes or augments model parameters with your data to specialize behavior. Open‑source models are easiest to fine‑tune; some proprietary providers offer fine‑tuning via API with less control and higher cost.

How do I decide between prompt engineering and fine‑tuning?

Start with prompt engineering—it’s fastest and often sufficient, especially with large context windows. Fine‑tune when you need highly specific outputs, when prompts still fall short, or to make smaller/self‑hosted models perform well. Fine‑tuning requires ML expertise, compute, and careful evaluation.

How can I prevent overfitting when fine‑tuning?

Use early stopping based on a separate validation set; limit updates (e.g., LoRA) instead of changing all weights; add regularization; apply dropout. Compare variants on a validation set and confirm the final choice on a separate, never‑used test set.

What’s the difference between customer‑facing AI apps and foundation models?

Customer‑facing apps (e.g., ChatGPT, GitHub Copilot, Cursor) are tailored tools for end users. Foundation models (e.g., GPT‑4o, Llama) are general building blocks accessed directly or via API to power your own applications. Use apps to accomplish tasks quickly; use foundation models when engineering your own solutions.

What are training, validation, and test sets—and why do they matter?

The training set teaches the model. The validation set compares models and tunes prompts/hyperparameters; it must not overlap with training data. The test set is a one‑time final check after selection. With LLMs, guard against contamination from public data (e.g., ensure items post‑date training or are novel).

Which performance measures should I use to evaluate AI?

Pick metrics aligned to your business goal. Common ones: accuracy for classification or task success; precision and recall for retrieval/detection (consider their trade‑off; F‑measure is an option but may not reflect business priorities); MAE/RMSE for numeric predictions. Often, set a minimum recall and compare precision across models.

Why is AI performance often overestimated, and how can I avoid it?

Leakage between training and validation, model selection bias on the validation set, and LLM memorization of public data all inflate results. Use clean splits, build validation/test sets unlikely to appear in training data (or after cutoff dates), and reserve a never‑touched test set for the final sanity check.

pro $24.99 per month

lite $19.99 per month

team

pro

team

pro

team

pro

team