4 Zero-Shot probabilistic forecasting with Lag-Llama

This chapter presents Lag-Llama, an open-source, zero-shot probabilistic forecasting model positioned as a research- and proof-of-concept-friendly alternative to paid, API-based solutions like TimeGPT. Unlike point-forecasting methods, Lag-Llama outputs full predictive distributions, providing uncertainty estimates via inherent prediction intervals. At the time of writing, it is accessed by cloning its repository (no packaged API), supports univariate forecasting only (no exogenous features), and is best suited for experimentation rather than production. Its pretraining spans diverse domains but focuses on daily to minute-level frequencies with forecast horizons of 24–60 steps, implying weaker zero-shot performance for weekly, monthly, or longer horizons unless fine-tuned.

Architecturally, Lag-Llama adapts a decoder-only Transformer backbone inspired by LLaMA and builds tokens from lagged values and rich date-time features across granularities, enabling the model to infer series frequency implicitly. This tokenization yields a relatively large context window, which can slow inference but enriches temporal structure learning. Forecasts are generated autoregressively, and a distribution head outputs the parameters (mean, scale, degrees of freedom) of a Student’s t-distribution to capture uncertainty robustly. Practically, inference runs through GluonTS interfaces, requiring data in compatible formats and 32-bit floats; users obtain sample paths and derive summary statistics (e.g., medians and confidence intervals) for decision-making.

The chapter walks through end-to-end usage: cloning the repo, installing dependencies, loading pretrained weights, formatting data, and producing zero-shot forecasts on weekly Walmart sales. It highlights the importance of the context length hyperparameter—longer or shorter input windows can materially affect accuracy, with an example where a length of 64 performed best in cross-validation. Fine-tuning on a GPU (e.g., in Colab) further improved metrics over zero-shot, though prediction intervals remained relatively wide under the assumed t-distribution. Overall, Lag-Llama’s strengths are openness and immediate probabilistic outputs; its trade-offs are setup friction, hardware-dependent speed, univariate scope, and frequency/horizon limitations tied to pretraining—making it ideal for research, prototyping, and scenarios where local fine-tuning is feasible.

General architecture of Lag-Llama. It features only the decoder portion of the Transformer, and the distribution head is responsible for generating the distribution of future values.

Tokenization process in Lag-Llama. Here, we see that the token of the current time step includes lagged values of the series over a certain context length, as well as date-time features, like second-of-minute, min-of-hour, all the way to quarter-of-year.

Student’s t-distribution with varying scale and degrees of freedom. We can see that more degrees of freedom results in a lower peak, while the scale, also called the standard deviation, widens the peak and its value increases.

Zero-shot predictions from Lag-Llama for the next eight weeks of sales of four Walmart stores. The shaded area represents the 95% prediction interval.

MAE and sMAPE for each context length. Lower values are better. Here, a context length of 64 time steps generated the best results.

Plotting the best performing forecasts from Lag-Llama using a context length of 64 time steps. We can see the forecasts mostly overlap the true values.

• Lag-Llama is an open-source probabilistic foundation model
• It uses a decoder-only architecture, and it constructs features using past values and date-time features, like second-of-minute, minute-of-hour, all the way to quarter-of-year.
• Lag-Llama was pretrained on frequencies of every day, hour, 15 minutes, and minute, and on forecast horizons ranging from 24 to 60 time steps.
• Since the model must be downloaded locally, using a GPU will result in faster inference time and fine-tuning.

FAQ