We gave a standard forecasting method 52 months of unit sales and hid the last 12 months from it. Then we asked it, and three plain rules, to guess those 12 months. On the average percent miss, the method finished last. Its miss was 25.19 percent. The rule "repeat the same month from a year ago" missed by 15.44 percent.

That is one series and one test, not a verdict on forecasting. But it shows why you check the plain rule before you trust a forecast.

The question and the data

The question: how much will we sell next month? We used one anonymised business-services company from the Conjura dataset: the series the dataset labels US, in USD. We counted units sold, adding the daily rows up into complete calendar months. That gave 52 months, February 2020 to May 2024. The partial first and last months were dropped so no month is short.

Basis for every figure here: one series, units per month, 52 months, 40 to fit and 12 to test.

How the test works

Think of it like a pop quiz with the answers hidden. We showed each method the first 40 months and kept the last 12 months back. Each method then wrote down its guess for those 12 months, and we compared the guesses to what really happened.

The method is Holt-Winters exponential smoothing. It tracks three things: a level (about where sales sit now), a trend (up or down over time), and a seasonal pattern that repeats each year. The three plain rules need no model at all:

  1. Last value: every test month equals the last month it saw.
  2. Seasonal naive: every test month equals the same month one year earlier.
  3. Three-month average: every test month equals the average of the last three months it saw.

All four were scored on the same 12 months, in the same way. We wrote down this plan, including which series and which split, before we ran anything.

The scores

Two scores are used. MAPE is the average percent miss. MAE is the average miss in units. For both, lower is better.

MethodMAPE (percent)MAE (units per month)
Seasonal naive15.441,320.75
Three-month average22.392,358.89
Last value23.912,555.33
Holt-Winters tool25.192,215.38

Basis: the 12 test months, one series, units per month.

By MAPE the tool is last, so it loses to all three plain rules. By MAE it beats two of them (the three-month average and last value) and still loses to seasonal naive. Read the two scores together: the tool never beat the rule that copies last year's month. The tool's own label for its forecast is "low-confidence".

Why can the two scores disagree? MAPE divides each miss by that month's real sales, so a miss in a small month counts for more. MAE counts every unit the same. A method can look better on one and worse on the other.

A guess at why the plain rule won

This series swings hard with the seasons. The seasonal swing the tool found is about 13,158 units, and November is the largest effect, at about 11,332 units. A rule that copies last year's month gets that pattern for free. The tool has to learn it from only 40 months of history. Take this as a reading, not as proof: our test did not check the cause.

The forecast, with a warning label

We then fit the tool on all 52 months. For June 2024 it gives about 6,139 units, with a 95 percent interval of 2,759 to 9,519 units. Over 13 months it projects a fall of about 20.9 percent from the last real month (6,495 units, May 2024).

Because this tool lost its test, read that as a projection to look at with care, not a number to plan on. The interval is wide, and the last real month (6,495 units) sits near its middle.

How this compares

Each method believes something different about the future.

  • Last value is right when sales are steady and have little seasonal shape. It believes next month looks like this month.
  • Seasonal naive is right when a yearly shape repeats. It believes next month looks like the same month last year.
  • Three-month average is right for a noisy series without a trend. It believes recent months are the best guide and the noise cancels out.
  • Holt-Winters is right when a trend and a seasonal shape both continue and you have enough history to learn them. It believes the past level, slope and yearly pattern keep going.

None of these is "best". On this one series and this one test block, the plain rule that copies last year did better than the model.

What this does not tell you

  • It is one series and one 12-month test block. One test is not a track record.
  • It does not say anything about other series, other companies or other forecasting methods.
  • 2020 and 2021 sit inside the pandemic. The tool has no notion of it.
  • A forecast is a guess from history, not a promise. Nothing here says to stop forecasting.

Questions analysts ask

Why hide the last 12 months?

A method usually looks better on data it has already seen. Hiding months lets you test it on data it has not seen.

Does the loss mean Holt-Winters is a bad method?

No. It means that on this series, in this test, it did worse than three plain rules. Another series could go the other way.

Which score should I trust, MAPE or MAE?

Neither alone. Report both and say what each one counts.

Why only one test block?

The tool backtests one block of the most recent months. One block is one test.

Sources

Data: Conjura Multi-Region MMM Dataset (Anderson 2024), figshare. CC BY 4.0. Changes: one anonymised series the dataset labels US, in USD, daily rows summed to complete calendar months, units as the outcome.

Method: Holt, C. C. (1957) and Winters, P. R. (1960), exponential smoothing with trend and seasonality. Numbers come from stations/registries/concepts/forecast-academic/results.txt.