This chapter shows how to balance cost and quality when analyzing data with large language models by combining smart model selection, tuning, prompt engineering, and—only when justified—fine-tuning. Using a review-sentiment classification task as a running example, it emphasizes measuring token usage and fees, experimenting on small samples, and picking the smallest model that meets quality needs. The cost gap between large and small models can be dramatic (e.g., orders of magnitude over a year’s workload), so careful, empirical optimization pays off before scaling to full datasets.
The chapter starts with an untuned GPT‑3.5 baseline that correctly labels 6/10 reviews and then incrementally improves it. Practical tuning includes prioritizing label tokens via logit_bias and, critically, capping output with max_tokens to avoid runaway generation; together these yield 7/10 with fewer wasted tokens. Swapping to a larger model (GPT‑4) lifts accuracy to 8/10 but at vastly higher per‑token cost, so it is not always economical. Prompt engineering—specifically few-shot examples embedded in the prompt—raises GPT‑3.5 to 8/10 as well, roughly doubling prompt tokens yet remaining far cheaper than GPT‑4. A tunable classifier script illustrates how to systematically vary model, output length, prioritized token IDs, and number of in-prompt examples to benchmark quality and cost on a sample set.
Finally, the chapter covers fine-tuning: preparing JSONL training data as user/assistant message pairs, launching and monitoring a job, and using the resulting specialized model. Fine-tuning can boost accuracy and lets you shorten prompts (omitting instructions and examples), but it entails an upfront training cost and higher ongoing per-token fees. In the case study, the fine-tuned model handled bare-review prompts and achieved 7/10 with slightly fewer tokens than the tuned baseline, yet its per-token price was about six times higher—offsetting savings. The overarching guidance is to start with the cheapest adequate model, constrain decoding and outputs, design concise prompts (optionally with a few targeted examples), always cap output length, and reserve fine-tuning for scenarios where measured quality and total cost of ownership clearly justify it.
GPT tokenizer at https://platform.openai.com/tokenizer: enter text to learn the associated token IDs.
Summary
Tuning parameter settings can influence model performance and cost.
Consider limiting output length and introducing token logit bias.
Do not always use the largest available model as it increases cost.
Identify the best model for your task by evaluating it on samples.
The design of the prompt can have a significant impact on the performance.
Include samples of correctly solved tasks into the prompt for few-shot learning.
Fine-tuning allows you to specialize base models to the tasks you care about.
Fine-tuning may allow you to reduce prompt size due to specialization.
Fine-tuning incurs overheads proportional to the amount of data trained on.
Fine-tuning increases the cost per token when using the resulting model.
FAQ
What are the main ways to optimize both cost and quality when analyzing data with LLMs?Use four levers: (1) model selection—pick the smallest/cheapest model that still performs well; (2) model tuning—adjust parameters like temperature, max_tokens, and logit_bias; (3) prompt engineering—write better prompts and add few-shot examples; and (4) fine-tuning—train a customized model on a small task-specific dataset when other methods aren’t enough. Always test combinations on a small sample first to find the best trade-off.How do I estimate the cost of running a large-scale analysis?Costs scale with total tokens processed. Use: total_cost ≈ (tokens_processed × price_per_1k_tokens) / 1000. Example: 10,000 reviews/day × 100 tokens/review ≈ 1,000,000 tokens/day → ~365,000,000 tokens/year. At $0.06/1K tokens (GPT-4), that’s ≈ $21,900/year; at ~$0.0005/1K tokens (GPT-3.5 Turbo), ≈ $182.50/year.How can I prevent the model from outputting invalid class labels like “negative” instead of “neg”?Bias the logits toward the exact label tokens using logit_bias (e.g., token IDs for “pos” and “neg” are 981 and 29875). Important: also constrain output length (e.g., max_tokens=1) or include stop tokens. Without an end condition, the model can loop, emitting endless sequences of only the biased tokens, inflating cost and breaking the label format.Why set temperature to zero for classification tasks?Temperature=0 reduces randomness, making outputs deterministic and repeatable—ideal for classification where consistent, exact labels (“pos”/“neg”) matter. It also simplifies evaluation during tuning because results don’t fluctuate across runs.How do I find the token IDs for specific strings like “pos” and “neg”?Use OpenAI’s tokenizer tool at https://platform.openai.com/tokenizer. Select the correct tokenizer (e.g., “GPT 3.5 & GPT-4”), enter the string, and view token IDs. In the chapter’s example, “pos” is 981 and “neg” is 29875.What is few-shot prompting and how does it affect cost and accuracy?Few-shot prompting embeds one or more solved examples in the prompt to guide the model. In the example, adding one labeled review increased accuracy from 7/10 to 8/10 using GPT-3.5 Turbo, but roughly doubled token usage since each prompt now includes the sample plus the target review. It’s usually still far cheaper than switching to a much pricier model.When should I choose GPT-4 over GPT-3.5 Turbo for classification?Only when the added accuracy is worth the higher price per token. In the chapter, GPT-4 improved accuracy from 7/10 to 8/10 on the sample but costs roughly 120× more per token than GPT-3.5 Turbo. Evaluate whether the incremental gains (often on borderline cases) justify the cost for your application.What parameters should a “tunable classifier” expose?Expose: model (e.g., gpt-3.5-turbo or gpt-4), max_tokens (to cap output length), out_tokens (comma-separated token IDs to prioritize via logit_bias), nr_samples (how many examples to embed), and sample_path (CSV of labeled examples). Keep temperature at 0 for stability. This setup lets you test many configurations without rewriting code.What does fine-tuning involve, and how do I prepare data?Fine-tuning specializes a base model (e.g., GPT-3.5 Turbo) on your task using labeled examples, typically 50 to a few thousand. Prepare data as JSON Lines where each line is a message history: user contains the review text and assistant contains the target label (“pos”/“neg”). After fine-tuning, prompts can be shorter—often just the review—because task instructions and label semantics are learned.Is fine-tuning cost-effective compared to prompt and parameter tuning?It depends. In the example, the fine-tuned model achieved similar accuracy to a tuned base model while using slightly fewer tokens per request, but its per-token price was higher (e.g., 0.3 vs 0.05 cents per 1K tokens), so total cost didn’t drop. Factor in one-time training costs, per-token usage fees, prompt length savings, and accuracy gains; run experiments and compute totals before committing.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Data Analysis with LLMs ebook for free
