6 Moirai: A universal forecasting transformer

This chapter introduces Moirai, an open-source foundation model designed as a universal forecaster that natively supports exogenous features and produces probabilistic predictions. Positioned alongside TimeGPT, Lag-Llama, and Chronos, Moirai stands out by handling multivariate inputs and external drivers, enabling zero-shot forecasting and anomaly detection without task-specific training. The chapter blends a conceptual tour of the model with hands-on guidance, showing how to install and use it in Python, and evaluating its behavior on sales forecasting and anomaly detection tasks.

Moirai adopts an encoder-only Transformer with non-overlapping patching that tokenizes time series at lengths tied to data frequency, then flattens multivariate inputs into a single sequence and uses a binary attention bias to distinguish within- and cross-variate relations—an “any-variate” attention scheme. Its output layer predicts parameters of a mixture distribution (Student’s t, log-normal, negative binomial, and a low-variance normal), enabling flexible, potentially asymmetric uncertainty. Pretraining leverages LOTSA, a 27-billion-point, multi-domain, multi-frequency archive; balanced subsampling and varied input lengths (2–512 steps) teach horizons up to 256 steps. The chapter also presents Moirai-MoE, a successor that switches to a decoder-only architecture with a sparse, 32-expert feedforward layer and a centroid-based gating mechanism derived from clustered token embeddings; it standardizes patching with a single input projection and achieves higher accuracy with faster inference due to sparse expert activation despite having more parameters.

Practically, Moirai is accessed via a Python package (uni2ts) and GluonTS formats: data are wrapped in a PandasDataset, a predictor generates samples, and point forecasts come from the median with configurable prediction intervals. In a weekly sales example, zero-shot cross-validation shows partial overlap with actuals but underestimates seasonal variation when using the small model; adding exogenous features (holiday as future-known, temperature/fuel/CPI/unemployment as past-only) improves accuracy modestly (MAE from 84,140 to 82,041; sMAPE from 2.64% to 2.57%). For anomaly detection on NYC taxi data, rolling forecasts with a 99% interval yield high recall and the best F1 among tested models (0.53), albeit with many false positives. The chapter closes with guidance on model size versus hardware trade-offs and highlights Moirai’s strengths when exogenous covariates are available and horizons are within its pretrained limit.

Architecture of Moirai. Here, the series is patched before entering the encoder-only Transformer. Then, a mixture of distributions generates a flexible output distribution of future values.

Illustrating the process of patching. Typically, each data point (circle) would be a token and sent to the embedding layer. With patching, we group data points together and each patch becomes a token that is sent to the embedding.

To handle an arbitrary number of variates, Moirai flattens the data into a single series.

Illustrating a mixture distribution made of a Student’s T-distribution and a log-normal distribution. For this plot, points have the same chance of being drawn from each distribution, and the result is a combination of both functions, as shown by the bars.

Architecture of Moirai-MoE. Unlike the first version of Moirai, it now uses a decoder-only transformer. Inside the decoder-only transformer, we notice the addition of a gating function and a layer of mixture of experts. The gating function learns which expert to activate depending on the data, with a maximum of two experts being activated at once. The activation of an expert is indicated by a solid arrow.

Zero-shot forecasts of the weekly sales of four stores eight weeks into the future. The shaded area represents the 95% prediction interval.

Predictions from running cross-validation with Moirai. The shaded area represents the 95% prediction interval. There is some overlap, but Moirai seems to not forecast the seasonal variations correctly.

Predictions obtained from running cross-validation with Moirai and using exogenous features. The shaded area represents the 95% prediction interval. Despite using features, the model still cannot predict the cyclical variations.

Both performance metrics improved when including exogenous features, indicating a boost in performance. Lower is better in this plot.

Anomalies detected by Moirai. Here, all anomalies were identified, but many points were mislabeled as anomalies. Therefore, its recall is high, but the precision is likely very low due to the high number of false positives.

F1 sore of foundation models used for anomaly detection. Here, Moirai achieves the best overall result. Higher is better.

• Moirai is a large time model developed by Salesforce researchers. It leverages patching and an encoder-only Transformer for probabilistic forecasting.
• The model is open-source and can be installed as a Python library.
• Using the largest model usually leads to better results, although it also requires better hardware to have reasonable inference times.
• Researchers also released LOTSA which represents the largest collection of open-source time series data, with more than 27 billion data points. This is an important contribution for the future development of large time models.

FAQ