2 Building a foundation model

This chapter provides a hands-on walkthrough of building a small-scale time series foundation model to illustrate the core ideas behind pretraining, transfer learning, and fine-tuning. Instead of starting with complex Transformers, it adopts the lightweight and generalizable N-BEATS architecture to make the process accessible and fast while still showcasing how foundation models learn broad patterns and adapt to new tasks. Along the way, it highlights practical considerations—like data frequency, forecast horizons, evaluation metrics, and compute/data constraints—that shape how robust foundation models are developed and deployed.

The chapter first demystifies N-BEATS, emphasizing neural basis expansion and a simple, fully connected design organized into blocks and stacks. Each block generates a forecast and a backcast, enabling sequential residual learning across stacks to capture information missed earlier. With this foundation, the model is pretrained on the monthly M3 dataset (learning from diverse series and domains), then saved and reused via transfer learning. Applied to a new monthly series, three approaches are compared—zero-shot forecasting with the pretrained model, fine-tuning that model on the target series, and training a model from scratch. Evaluated with MAE and sMAPE, the zero-shot pretrained model performs best in this monthly setting, underscoring the power of generalization from broad pretraining.

The chapter then tests robustness across frequencies by applying the monthly-pretrained model to a daily temperature series. Here, zero-shot performance degrades relative to a model trained specifically on the daily data, demonstrating that frequency shifts introduce different temporal patterns the pretrained model did not learn. This motivates practical lessons: data frequency matters, horizon limits can be restrictive (suggesting generative approaches for longer forecasts), fine-tuning depth and steps must be chosen carefully, and real foundation models demand substantial, high-variety datasets and significant compute. The chapter closes by reflecting on these challenges and setting up the exploration of larger, purpose-built foundation models in subsequent chapters.

Visualizing the impact of basis expansion. On the left, without basis expansion, we are stuck with using a linear model. On the right, after a second-degree polynomial basis expansion, we can fit a quadratic line and better fit our data.

Architecture of N-BEATS. N-BEATS is made of stacks, which are made of blocks, where each block outputs a forecast and a backcast.

Plotting the first four series of the monthly M3 dataset. The title refers to a label for each series. Note that each series is part of the M3 dataset.

Monthly volume of antidiabetic drug prescription in Australia, from 1991 to 2008.

Plotting the predictions against the actual values. We can see that the predictions mostly overlap with the actual values but coming from the pretrained model with zero-shot forecasting seem to be the closest.

Predictions on daily data using the pretrained model and a data-specific model. Here, we see that the pretrained model fails to make accurate predictions, since the dashed line (zero-shot forecasts) never overlaps the solid line (actual values). However, the forecasts from the trained model (dotted lines) is closer to the actual values.

• A pretrained model is trained on large amounts of data such that it can be used in another scenario.
• Transfer learning is when we use a pretrained model on a dataset that was not previously seen by that model.
• Pretrained models can generate zero-shot predictions or be fine-tuned. With zero-shot predictions, the model never trains on the new dataset, while fine-tuning allows the model to specialize in the task at hand by training for a few steps.
• We can easily build a pretrained model, but it will not perform well on other frequencies, and we are limited in the forecast horizon.
• Building foundation models is hard. They require massive amounts of data and expensive resources that are not available to the vast majority of practitioners.

FAQ