Executive Summary
Does Forecast actually beat doing nothing clever?
The short answer
Your forecast beats the seasonal-naive baseline decisively: a MASE of 0.488 means it is less than half as bad as repeating the same month 12 months earlier. A Diebold-Mariano test confirms this gap is larger than sampling noise (p < 0.001), not a chance finding.
The detail
Across 60 monthly periods, the forecast's MAE is 3.09 against the seasonal-naive baseline's 6.32, a MASE of 0.488. The Diebold-Mariano test statistic is -8.869 with p < 0.001, so the forecast is significantly more accurate than seasonal-naive. The process under-forecast by 0.16 units per month on average (mean error). MAPE reads 2.74%. Accuracy varies across time: 2021 was the worst stretch (MAE 4.12), 2022 the best (MAE 2.35). Error decays with lead time: the 6-month horizon carries 5.72 times the error of the 1-month horizon.
What this can't tell you
MAPE is asymmetric and penalizes over-forecasting more heavily, so it is not the row to trust most; MASE is. Longer horizons (3 and 6 months) fail to beat the seasonal-naive baseline, meaning the forecasting effort does not pay for itself at those lead times.
Analysis Overview
Backtest of 60 monthly forecasts of Actual.
The short answer
Three error metrics are needed because they answer different questions: MAE and RMSE measure forecast accuracy in your data's units, percentage metrics (MAPE, sMAPE) let you compare across different series, and MASE answers the only question that matters operationally — whether the forecasting effort beats doing nothing. This analysis scores 60 monthly forecasts already made, comparing them to two free baselines: repeating the last month and repeating the same month 12 months earlier.
The detail
The backtest covers 60 monthly periods with no rows dropped and no missing months. MAE (3.09) and RMSE (3.80) are in units of Actual; MAPE (2.74%) and sMAPE (2.76%) express error as percentages; MASE compares the forecast's error to what a seasonal-naive baseline would have scored. The companion forecasting tool produces the forecast; this tool grades it. Three metrics are needed because no single number captures whether a forecast is worth its cost — that is what MASE does, by dividing the forecast's error by the baseline's error.
What this can't tell you
This analysis grades forecasts already made; it does not explain what produced them or how to improve them. The MASE scaling factor is computed from the observed series rather than a held-out test window, because a forecast log carries no train-test split.
Data Quality
How the forecast log became a set of scored periods.
The short answer
All 60 periods loaded cleanly with no dropped rows, no duplicates, and no missing months in the span. The data cadence was inferred as monthly. Horizon Months carries three distinct levels; headline figures cover the most frequent level (1-month horizon), and the by-horizon card reports the rest.
The detail
180 rows loaded; 0 rows dropped for unreadable dates, missing forecasts, or missing actuals. 0 rows collapsed as duplicates. The cadence of Target Month was read as monthly from the median gap between periods. Each scored month appeared exactly once. Horizon Months carries 3 distinct levels; the headline figures cover the level with the most observations (1), and the by-horizon card covers the rest. The monthly series is complete — no month in the span is missing. Final scored set: 60 months.
What this can't tell you
The data quality assessment does not validate whether the forecast values themselves are plausible or whether the forecasting process was correctly specified — only that the log is complete and readable.
Actual vs Forecast
Actual and Forecast plotted over the same periods.
The short answer
The forecast tracks the actual line closely without lagging turns or flattening seasonal swings — exactly the pattern that earns a MASE below 1. The forecast ran high in 48.33% of periods and low in 51.67%, with an average gap of 0.16 units, showing no persistent one-sided drift.
The detail
Over 60 months from 2020-01-01 through 2024-12-01, the forecast and actual lines follow the same seasonal cycle and turning points. The forecast is above the actual in 48.33% of periods (over-forecast) and below in 51.67% (under-forecast), with a mean error of 0.16 units of Actual — the removable part of the miss. A forecast line that lags the actual's turns or flattens its seasonal swings is what produces a MASE at or above 1; this one does neither.
What this can't tell you
The visual match between forecast and actual does not diagnose the source of the forecast (which method, which inputs, which tuning) — only that the output tracks the signal without systematic lag or over-smoothing.
Accuracy Scorecard
Every headline error metric with what it does and does not tell you.
| Metric | Value | Interpretation |
|---|---|---|
| Periods evaluated | 60 | Distinct months of 'Forecast' scored against 'Actual'. |
| MAE (mean absolute error) | 3.086 | Average miss, in units of 'Actual', ignoring direction. |
| RMSE (root mean squared error) | 3.797 | Average miss with large misses weighted more heavily; RMSE is 1.23 times MAE here, so the misses are fairly evenly sized. |
| Mean error (bias) | 0.1607 | Actual minus forecast on average: the process under-forecast by 0.16 units of 'Actual' per month. |
| Periods under-forecast (%) | 51.67 | 31 of 60 periods came in above the forecast; 29 came in below and 0 landed exactly. |
| MAPE (%) | 2.745 | Computed on all 60 periods; no zero or near-zero actuals were present. |
| sMAPE (%) | 2.758 | Symmetric percentage error: the denominator uses both the actual and the forecast, so it is bounded at 200 percent and does not blow up as the actual approaches zero. |
| MASE vs naive | 0.4513 | MAE divided by the naive baseline's average absolute change (6.84). Below 1 means the forecast beats repeating the last month. |
| MASE vs seasonal naive (12 months) | 0.4882 | MAE divided by the seasonal-naive baseline's average absolute change (6.32). Below 1 means the forecast beats repeating the same month 12 months earlier. |
The short answer
Your forecast is consistently accurate across the 60 periods, with no systematic bias and evenly distributed misses. The MAE of 3.0857 and RMSE of 3.7972 show that large errors are rare; the mean error of 0.1607 is negligible. The MASE of 0.4882 against the seasonal-naive baseline is the decisive metric—it confirms your forecast is worth the effort.
The detail
MAE is 3.0857 units; RMSE is 3.7972, which is 1.23 times MAE, indicating misses are fairly evenly sized rather than concentrated in a few catastrophic months. Mean error (bias) is 0.1607—the process under-forecast by about 0.16 units per month on average, a removable offset. Of 60 periods, 31 came in above the forecast (51.67 percent under-forecast). MAPE is 2.745 percent across all 60 periods. The key row: MASE vs seasonal naive is 0.4882, meaning your forecast's error is 48.82 percent of what the free baseline would have scored.
What this can't tell you
The MAPE asymmetry (2.65 percent when forecast runs low, 2.84 percent when it runs high) reflects the metric's structure, not a process bias. Your actual mean error of 0.1607 is small enough that bias correction would not materially improve performance.
Forecast vs Naive Baselines
Mean absolute error of the forecast next to the two free alternatives.
The short answer
Your forecast's MAE of 2.97 is the shortest bar: it is 53% of the seasonal-naive baseline's 6.32. The forecast beats both free alternatives decisively.
The detail
Over 48 common periods (where both baselines are defined), the forecast's MAE is 2.97 against the seasonal-naive baseline's 6.32 and the naive baseline's 6.77. The forecast is 53.00% of the seasonal-naive baseline's error, meaning it removes 47% of the work a rule that costs nothing would have left undone. Both baselines are not straw men: repeating the last month (naive) and repeating the same month 12 months earlier (seasonal naive) are the two forecasts anyone can produce for free. A process that does not clear both of them is spending effort for nothing.
What this can't tell you
This comparison covers only the 48 periods where all three methods have defined values; the headline MASE uses all 60 periods, so the figures here are not identical to the all-period summary.
Error Distribution
How the per-month misses are spread, and whether they centre on zero.
The short answer
Errors centre close to zero (median 0.04), showing noise rather than systematic bias. The middle 90% of periods fall between -5.39 and 6.99, with a slightly longer tail on the high side. It is the tail, not the average, that breaks a plan built on the forecast.
The detail
The median error is 0.04 and the middle 90% of periods (5th to 95th percentile) fall between -5.39 and 6.99. The distribution is centred close to zero, so the misses are noise rather than a one-sided offset that a bias correction would remove. The tails are roughly symmetric: the worst under-forecast is 8.95 and the worst over-forecast is -7.11, so neither side dominates. The distribution's shape matters more than its mean because it is the tail — the worst-case scenario — that breaks a plan built on the forecast.
What this can't tell you
This histogram does not reveal whether the largest errors cluster in a particular time period or horizon; the period_accuracy and horizon_accuracy cards address that.
Accuracy Over Time
Mean absolute error by stretch of the forecast log.
The short answer
Accuracy varies substantially across time: 2021 was the worst stretch (MAE 4.12), 2022 the best (MAE 2.35), a gap of 1.75 times. The headline MAE of 3.09 averages over genuinely different regimes, so it is not a stable number.
The detail
Accuracy split across five annual stretches. 2020 (MAE 3.55, mean error 0.77), 2021 (MAE 4.12, mean error -0.81), 2022 (MAE 2.35, mean error 0.59), 2023 (MAE 3.02, mean error 0.31), and 2024 (MAE 2.39, mean error -0.05). The worst stretch (2021, MAE 4.12) is 1.75 times as bad as the best (2022, MAE 2.35). The mean error column separates bias from volatility: 2021's bad performance is both bad and one-sided (mean error -0.81), pointing to a bias problem in that year, while 2022's good performance is centred (mean error 0.59), showing noise rather than systematic drift.
What this can't tell you
This breakdown does not explain what caused 2021's degradation or 2022's improvement — only that they happened. A rising trend would signal the forecasting process is degrading; here the trend is U-shaped, with recovery in 2023-24.
Accuracy by Horizon
Error by lead time, read from Horizon Months.
| Horizon | N Periods | MAE | RMSE | Mean Error | MAPE | Mase | Verdict |
|---|---|---|---|---|---|---|---|
| 1 | 60 | 3.086 | 3.797 | 0.1607 | 2.745 | 0.4882 | beats seasonal naive |
| 3 | 60 | 9.195 | 11.16 | 1.595 | 8.261 | 1.455 | does not beat seasonal naive |
| 6 | 60 | 17.64 | 21.01 | 1.563 | 15.49 | 2.791 | does not beat seasonal naive |
The short answer
The forecast beats the seasonal-naive baseline at 1 month (MASE 0.488) but fails at 3 and 6 months (MASE 1.45 and 2.79). Accuracy decays with lead time by a factor of 5.72 from 1 to 6 months, exactly as it should — but longer horizons are not worth the effort.
The detail
At 1-month horizon (60 periods), MAE is 3.09 and MASE is 0.488 — beats seasonal naive. At 3-month horizon (60 periods), MAE is 9.20 and MASE is 1.45 — does not beat seasonal naive. At 6-month horizon (60 periods), MAE is 17.64 and MASE is 2.79 — does not beat seasonal naive. The error multiplier from 1 to 6 months is 5.72. Two of three horizons fail to beat the seasonal-naive baseline, meaning the forecasting effort is not paying for itself at those lead times.
What this can't tell you
This analysis does not diagnose why longer horizons fail or whether a different forecasting method would recover them; it only scores the process as it is. The mean error grows with horizon (0.16 at 1 month, 1.59 at 3 and 6 months), suggesting systematic under-forecasting at longer leads.
Methods & Disclosure
The formulas, the baselines, and where each metric breaks down.
| Item | Detail |
|---|---|
| Error convention | Every period's error is 'Actual' minus 'Forecast', so a positive error means the outcome came in above the forecast (an under-forecast). |
| Scale-dependent error | MAE is the mean of the absolute errors (3.09) and RMSE the square root of the mean squared error (3.80), both in units of 'Actual'. RMSE exceeds MAE whenever the misses are uneven in size. |
| Percentage error | MAPE is the mean of the absolute errors divided by the actual, computed on 60 of 60 periods (0 excluded for a zero or near-zero 'Actual', where the ratio is undefined or explodes). MAPE is also bounded at 100 percent when the forecast runs low but unbounded when it runs high, so it penalises over-forecasting more heavily and a process tuned to minimise it will drift low. Measured here: mean absolute percentage error of 2.65 in the periods that ran high versus 2.84 in the periods that ran low. |
| Symmetric percentage error | sMAPE (2.76 percent) divides twice the absolute error by the sum of the absolute actual and the absolute forecast, which bounds it at 200 percent and keeps it finite when the actual is small. |
| Scaling baselines | The naive baseline repeats the previous month's actual; its average absolute change over the observed series is 6.84. The seasonal-naive baseline repeats the same month 12 months earlier; its average absolute change is 6.32. Both are computed on a regular monthly calendar grid, so gaps in the data never shift the lag. |
| MASE | MASE is MAE divided by a baseline's average absolute change: 0.451 against naive and 0.488 against seasonal naive. The scaling factor is taken from the whole observed series rather than a separate training window, because a forecast log carries no train and test split; that makes these values comparable across series but not identical to a MASE computed from a held-out fit. |
| Diebold-Mariano test | Against the naive baseline: 59 paired periods, mean loss differential -3.703 (negative means the forecast's absolute error is smaller), Newey-West lag 3, statistic -6.859, p < 0.001. Against the seasonal-naive baseline: 48 paired periods, mean loss differential -3.350 (negative means the forecast's absolute error is smaller), Newey-West lag 3, statistic -8.869, p < 0.001. The test is implemented directly here: the per-period loss differential under absolute-error loss is divided by a Newey-West long-run standard error with Bartlett weights, given the Harvey-Leybourne-Newbold small-sample correction, and referred to a t distribution. |
| Calendar handling | The cadence of 'Target Month' was read as monthly from the median gap between periods. 0 of the 60 calendar months spanned by the data carry no actual (0.00 percent). Baseline lags line up with the calendar. |
| What this cannot tell you | This is a scorecard of a forecasting process, not a diagnosis of it. It measures how far 'Forecast' landed from 'Actual' and whether that beats a naive rule; it cannot say why the misses happened, and it cannot rule out that the forecasts were revised with knowledge the baselines did not have. |
The short answer
Error is computed as Actual minus Forecast (positive = under-forecast). MASE divides the forecast's MAE by a baseline's average absolute change (6.32 for seasonal naive), making it the only metric that answers whether the forecasting effort adds value. The Diebold-Mariano test uses absolute-error loss with Newey-West long-run variance and Harvey-Leybourne-Newbold small-sample correction.
The detail
Error convention: Actual minus Forecast, so positive error means under-forecast. Scale-dependent error: MAE (3.09) and RMSE (3.80) in units of Actual. Percentage error: MAPE (2.74%) computed on all 60 periods; it is asymmetric (capped at 100% below, unbounded above), penalizing over-forecasting more heavily. sMAPE (2.76%) is symmetric and bounded at 200%. Scaling baselines: naive baseline's average absolute change is 6.84; seasonal-naive is 6.32. MASE is MAE divided by baseline: 0.488 against seasonal naive. The MASE scaling factor is taken from the whole observed series, not a held-out window, because a forecast log carries no train-test split. Diebold-Mariano test against seasonal naive: 48 paired periods, mean loss differential -3.350, Newey-West lag 3, statistic -8.869, p < 0.001. Calendar: monthly cadence, 0 missing months (0.00%).
What this can't tell you
MASE values computed from the observed series rather than a held-out test window are comparable across series but not identical to a MASE from a separate fit. This tool scores the forecast without diagnosing what produced it or how to improve it.
Methodology
Statistical methodology and diagnostics for Forecast Accuracy Backtest
Statistical Method
Standard-library analysis: how good are the forecasts you already made? Map the period, the forecast, and the actual outcome — and optionally the horizon each forecast was made at — and get the full scorecard: MAE and RMSE in your own units, MAPE with its zero and asymmetry traps handled explicitly, sMAPE, MASE against both the naive and the seasonal-naive baseline, bias as a mean error plus the share of periods over- and under-forecast, a Diebold-Mariano test of whether the forecast really beats a baseline rather than just printing two numbers next to each other, actual against forecast over time, the error distribution, and accuracy broken out by calendar period and by forecast horizon.
- Each row is one forecast for one dated period, with the actual outcome for that same period
- The forecast and the actual are in the same units
- The forecast was made before the outcome was known — the report cannot detect a forecast revised after the fact
- Rows sharing a period at the same horizon are averaged into a single figure for that period
- The naive and seasonal-naive baselines are meaningful for this series, which requires the periods to be evenly spaced on a calendar
- This is a scorecard of a forecasting process, not a diagnosis of it — it measures the misses but cannot say why they happened
- MAPE is undefined at a zero actual and unstable near zero; those periods are excluded and counted, so MAPE may rest on a subset
- MAPE is asymmetric: unbounded when the forecast runs high but capped at 100 percent when it runs low, so minimising it biases a process low
- The MASE scaling factor is computed from the whole observed actual series rather than a separate training window, because a forecast log carries no train and test split; values stay comparable across series but are not identical to a MASE from a held-out fit
Analysis Code
Complete R source code for this analysis
Forecast Accuracy Backtest — Are Your Forecasts Actually Any Good?
Grades a log of forecasts against what actually happened. Scale-dependent error (MAE, RMSE), percentage error (MAPE with its zero and asymmetry traps handled, plus sMAPE), scale-free error against the naive and seasonal-naive baselines (MASE), and bias (mean error plus the share of periods over- and under-forecast). A Diebold-Mariano test decides whether the forecast's loss is really lower than a baseline's, rather than just printing two numbers side by side. Accuracy is then broken out by calendar period and, when a horizon column is mapped, by forecast horizon.
Why This Method?
A forecast is only worth its cost if it beats the cheapest thing you could have done instead. The cheapest things are the naive forecast ("next period equals this period") and the seasonal naive forecast ("next period equals the same period one cycle ago"). MASE expresses the forecast's error as a multiple of that baseline's error, so a value at or above 1 means the forecasting process is not adding value. The Diebold-Mariano test then asks whether the difference in loss is larger than the sampling noise.
What This Analysis Covers
- Actual against forecast over time (chart)
- MAE, RMSE, MAPE, sMAPE, MASE, and bias
- Naive and seasonal-naive baselines with a Diebold-Mariano test
- Error distribution, accuracy by period, accuracy by horizon
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {date, forecast, actual, horizon}. All narrative is derived from the user's own column names and computed values. The sibling tool standard_forecasting PRODUCES a forecast from history; this tool GRADES forecasts you already made.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
Step 1: Resolve the mapped columns (humanized for every sentence)
initial_rows <- nrow(df)
date_h <- humanize_semantic("date", col_map)[1]
forecast_h <- humanize_semantic("forecast", col_map)[1]
actual_h <- humanize_semantic("actual", col_map)[1]
horizon_h <- humanize_semantic("horizon", col_map)[1]
if (!all(c("date", "forecast", "actual") %in% names(df))) {
stop(sprintf(
"A forecast backtest needs three columns mapped: the period the forecast was for ('%s'), the forecast itself ('%s'), and the actual outcome ('%s').",
date_h, forecast_h, actual_h))
}
if (initial_rows < MIN_PERIODS) {
stop(sprintf(
"Only %d rows were supplied — grading '%s' against '%s' needs at least %d dated periods.",
initial_rows, forecast_h, actual_h, MIN_PERIODS))
}Step 2: Parse dates; coerce both numbers with the 95% rule
raw_date <- df$date
non_blank <- !(is.na(raw_date) | !nzchar(trimws(as.character(raw_date))))
dts <- parse_dates_robust(raw_date)
n_nonblank <- sum(non_blank)
if (n_nonblank == 0) {
stop(sprintf("The '%s' column is empty — there are no periods to score forecasts against.", date_h))
}
n_unparsed <- sum(non_blank & is.na(dts))
if (n_unparsed / n_nonblank > 0.05) {
stop(sprintf(
"%d of %d values in '%s' could not be read as dates. Please use a recognizable date format such as 2024-01-31 or 01/31/2024.",
n_unparsed, n_nonblank, date_h))
}
coerce_num <- function(v, label_h, role) {
if (is.numeric(v)) return(as.numeric(v))
ch <- as.character(v)
nb <- !is.na(ch) & trimws(ch) != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(nb) == 0 || sum(!is.na(conv[nb])) < 0.95 * sum(nb)) {
stop(sprintf(
"The column '%s' was mapped as the %s but fewer than 95%% of its values parse as numbers. Map a numeric column instead.",
label_h, role))
}
conv
}
fcv <- coerce_num(df$forecast, forecast_h, "forecast")
acv <- coerce_num(df$actual, actual_h, "actual outcome")
keep <- !is.na(dts) & !is.na(fcv) & !is.na(acv)
n_dropped <- sum(!keep)
work <- data.frame(date = dts[keep], forecast = fcv[keep], actual = acv[keep],
stringsAsFactors = FALSE)
has_horizon <- "horizon" %in% names(df)
if (has_horizon) work$horizon_raw <- df$horizon[keep]
n_rows <- nrow(work)
if (n_rows < MIN_PERIODS) {
stop(sprintf(
"Only %d complete rows remained after dropping rows with an unreadable '%s' or a missing '%s' or '%s' — at least %d are needed.",
n_rows, date_h, forecast_h, actual_h, MIN_PERIODS))
}
work <- work[order(work$date), , drop = FALSE]Step 3: Horizon column — numeric, binned, or lumped categorical
single_horizon <- FALSE; primary_label <- NA_character_
n_horizons <- 0L; horizon_note <- ""
horizon_usable <- FALSE
if (has_horizon) {
hv <- work$horizon_raw
ch <- trimws(as.character(hv))
nb <- !is.na(ch) & nzchar(ch)
hnum <- suppressWarnings(as.numeric(ch))
numeric_ok <- sum(nb) > 0 && sum(!is.na(hnum[nb])) >= 0.95 * sum(nb)
if (numeric_ok) {
hnum[!nb] <- NA_real_
uq <- sort(unique(hnum[!is.na(hnum)]))
if (length(uq) > 12) {
brks <- unique(stats::quantile(hnum, probs = seq(0, 1, length.out = 7), na.rm = TRUE))
if (length(brks) >= 3) {
cutf <- cut(hnum, breaks = brks, include.lowest = TRUE, dig.lab = 6)
work$horizon_label <- as.character(cutf)
work$horizon_order <- as.numeric(cutf)
horizon_note <- sprintf("'%s' held %d distinct values, so they were grouped into %d ranges.",
horizon_h, length(uq), length(levels(cutf)))
} else {
work$horizon_label <- as.character(hnum)
work$horizon_order <- hnum
}
} else {
work$horizon_label <- ifelse(is.na(hnum), "Missing", format(hnum, trim = TRUE, scientific = FALSE))
work$horizon_order <- ifelse(is.na(hnum), Inf, hnum)
}
} else {
lab <- ch
lab[!nb] <- "Missing"
tb <- sort(table(lab), decreasing = TRUE)
keep_lv <- names(tb)[seq_len(min(8L, length(tb)))]
n_lumped <- sum(!(lab %in% keep_lv))
lab[!(lab %in% keep_lv)] <- "Other"
work$horizon_label <- lab
cnt <- table(lab)
work$horizon_order <- as.numeric(-cnt[lab])
horizon_note <- sprintf("'%s' is not numeric, so it was treated as a label%s.",
horizon_h,
if (n_lumped > 0) sprintf(" and %d rows outside the eight most common levels were grouped as \"Other\"", n_lumped) else "")
}
lv_tab <- table(work$horizon_label)
n_horizons <- length(lv_tab)
horizon_usable <- n_horizons >= 1
single_horizon <- n_horizons == 1
cand <- names(lv_tab)[lv_tab == max(lv_tab)]
ord <- sapply(cand, function(k) min(work$horizon_order[work$horizon_label == k]))
primary_label <- cand[order(ord, cand)][1]
}Step 4: The actual series (all rows), its cadence, and its calendar grid
ser <- stats::aggregate(actual ~ date, data = work, FUN = mean)
ser <- ser[order(ser$date), , drop = FALSE]
if (nrow(ser) < 4) {
stop(sprintf("Only %d distinct periods appear in '%s' — a backtest needs the forecasts spread over time.",
nrow(ser), date_h))
}
med_gap <- stats::median(as.numeric(diff(ser$date)))
if (!is.finite(med_gap) || med_gap <= 0) med_gap <- 1
if (med_gap <= 1.5) {
freq_label <- "daily"; step_days <- 1; m <- 7L; period_word <- "day"; cycle_word <- "week"
} else if (med_gap >= 5.5 && med_gap <= 8.5) {
freq_label <- "weekly"; step_days <- 7; m <- 52L; period_word <- "week"; cycle_word <- "year"
} else if (med_gap >= 26 && med_gap <= 35) {
freq_label <- "monthly"; step_days <- 30; m <- 12L; period_word <- "month"; cycle_word <- "year"
} else if (med_gap >= 85 && med_gap <= 95) {
freq_label <- "quarterly"; step_days <- 91; m <- 4L; period_word <- "quarter"; cycle_word <- "year"
} else if (med_gap >= 350 && med_gap <= 380) {
freq_label <- "yearly"; step_days <- 365; m <- 1L; period_word <- "year"; cycle_word <- "period"
} else {
freq_label <- "irregular"; step_days <- max(1, round(med_gap)); m <- 1L
period_word <- "period"; cycle_word <- "cycle"
}
origin <- ser$date[1]
if (freq_label == "irregular") {
gidx_ser <- seq_len(nrow(ser))
n_grid <- nrow(ser)
} else {
gidx_ser <- grid_index(ser$date, origin, freq_label, step_days)
gidx_ser[gidx_ser < 1] <- 1L
n_grid <- max(gidx_ser)
}
gv <- rep(NA_real_, n_grid)
gv[gidx_ser] <- ser$actual
n_grid_missing <- sum(is.na(gv))
pct_grid_missing <- 100 * n_grid_missing / n_gridStep 5: Naive and seasonal-naive baselines on that grid
g_naive <- c(NA_real_, gv[-n_grid])
seasonal_available <- m > 1L && n_grid > m + 2L
g_seas <- if (seasonal_available) c(rep(NA_real_, m), gv[seq_len(n_grid - m)]) else rep(NA_real_, n_grid)
d_naive <- abs(gv - g_naive)
d_seas <- abs(gv - g_seas)
q_naive <- if (any(!is.na(d_naive))) mean(d_naive, na.rm = TRUE) else NA_real_
q_seas <- if (any(!is.na(d_seas))) mean(d_seas, na.rm = TRUE) else NA_real_
if (!is.na(q_naive) && q_naive <= 0) q_naive <- NA_real_
if (!is.na(q_seas) && q_seas <= 0) q_seas <- NA_real_
if (is.na(q_seas)) seasonal_available <- FALSE
seasonal_phrase <- sprintf("the same %s %d %ss earlier", period_word, m, period_word)Step 6: Headline rows — the primary horizon if one is mapped,
collapsed to one forecast per period so every metric is per-period.
if (horizon_usable && !is.na(primary_label)) {
hl <- work[work$horizon_label == primary_label, , drop = FALSE]
} else {
hl <- work
}
hl_agg <- stats::aggregate(cbind(forecast, actual) ~ date, data = hl, FUN = mean)
hl_agg <- hl_agg[order(hl_agg$date), , drop = FALSE]
n_dup_rows <- nrow(hl) - nrow(hl_agg)
n_eval <- nrow(hl_agg)
if (n_eval < MIN_PERIODS) {
stop(sprintf(
"Only %d distinct periods are available to score%s — at least %d are needed to grade '%s'.",
n_eval,
if (horizon_usable) sprintf(" at the level of '%s' being scored (%s)", horizon_h, primary_label) else "",
MIN_PERIODS, forecast_h))
}
final_rows <- n_eval
rows_removed <- initial_rows - final_rowsStep 7: Error metrics on the evaluated periods
Convention: error = actual minus forecast, so a POSITIVE error is an under-forecast (the outcome beat the plan).
err <- hl_agg$actual - hl_agg$forecast
mae <- mean(abs(err))
rmse <- sqrt(mean(err^2))
me <- mean(err)
n_under <- sum(err > 0); n_over <- sum(err < 0); n_exact <- sum(err == 0)
pct_under <- 100 * n_under / n_eval
pct_over <- 100 * n_over / n_eval
pct_exact <- 100 * n_exact / n_evalMAPE is undefined at a zero actual and unstable at a near-zero one, so both are excluded and counted rather than silently dropped.
abs_a <- abs(hl_agg$actual)
pos_a <- abs_a[abs_a > 1e-9]
med_a <- if (length(pos_a) > 0) stats::median(pos_a) else NA_real_
zero_idx <- abs_a <= 1e-9
near_idx <- if (is.na(med_a)) rep(FALSE, n_eval) else (!zero_idx & abs_a < 0.01 * med_a)
mape_ok <- !zero_idx & !near_idx
n_zero <- sum(zero_idx); n_near_zero <- sum(near_idx)
n_mape_excl <- n_zero + n_near_zero
n_mape_used <- sum(mape_ok)
mape <- if (n_mape_used >= 3) 100 * mean(abs(err[mape_ok] / hl_agg$actual[mape_ok])) else NA_real_Measured asymmetry: mean absolute percentage error among the periods the forecast ran high versus those it ran low.
ape <- if (n_mape_used > 0) 100 * abs(err[mape_ok] / hl_agg$actual[mape_ok]) else numeric(0)
err_ok <- err[mape_ok]
ape_over <- if (sum(err_ok < 0) >= 3) mean(ape[err_ok < 0]) else NA_real_
ape_under <- if (sum(err_ok > 0) >= 3) mean(ape[err_ok > 0]) else NA_real_sMAPE — symmetric denominator, bounded at 200 percent.
sden <- abs(hl_agg$actual) + abs(hl_agg$forecast)
sok <- sden > 1e-9
smape <- if (sum(sok) >= 3) 100 * mean(2 * abs(err[sok]) / sden[sok]) else NA_real_
mape_reliable <- !is.na(mape) && n_mape_excl == 0 && (is.na(med_a) || min(pos_a) > 0.05 * med_a)
mase_naive <- if (!is.na(q_naive)) mae / q_naive else NA_real_
mase_seas <- if (!is.na(q_seas)) mae / q_seas else NA_real_
beats_naive <- !is.na(mase_naive) && mase_naive < 1
beats_seasonal <- !is.na(mase_seas) && mase_seas < 1Step 8: Paired baseline errors for the evaluated periods
if (freq_label == "irregular") {
gidx_hl <- match(hl_agg$date, ser$date)
} else {
gidx_hl <- grid_index(hl_agg$date, origin, freq_label, step_days)
}
gidx_hl[gidx_hl < 1 | gidx_hl > n_grid] <- NA_integer_
base_naive_fc <- g_naive[gidx_hl]
base_seas_fc <- g_seas[gidx_hl]
e_naive <- hl_agg$actual - base_naive_fc
e_seas <- hl_agg$actual - base_seas_fc
dm_naive <- dm_test(err, e_naive, h = 1)
dm_seas <- if (seasonal_available) dm_test(err, e_seas, h = 1) else NULLStep 9: Like-for-like baseline comparison on one common subset
bench_idx <- if (seasonal_available) !is.na(e_seas) else !is.na(e_naive)
bench_n <- sum(bench_idx)
bench_kind <- if (seasonal_available) "seasonal-naive" else "naive"
bench_fc_mae <- if (bench_n > 0) mean(abs(err[bench_idx])) else NA_real_
bench_naive_mae <- if (bench_n > 0) mean(abs(e_naive[bench_idx]), na.rm = TRUE) else NA_real_
bench_seas_mae <- if (bench_n > 0 && seasonal_available) mean(abs(e_seas[bench_idx]), na.rm = TRUE) else NA_real_
bench_rows <- list(data.frame(
method = sprintf("Your forecast(%s)", forecast_h),
mae = round(bench_fc_mae, 4),
n_periods = bench_n,
note = sprintf("Mean absolute error of '%s' against '%s' over the %d periods where every baseline is defined.",
forecast_h, actual_h, bench_n),
stringsAsFactors = FALSE))
if (is.finite(bench_naive_mae)) {
bench_rows[[length(bench_rows) + 1]] <- data.frame(
method = sprintf("Naive(last %s)", period_word),
mae = round(bench_naive_mae, 4), n_periods = bench_n,
note = sprintf("Repeat the previous %s's actual as the forecast — the cheapest thing you could have done.", period_word),
stringsAsFactors = FALSE)
}
if (is.finite(bench_seas_mae)) {
bench_rows[[length(bench_rows) + 1]] <- data.frame(
method = sprintf("Seasonal naive(%d %ss back)", m, period_word),
mae = round(bench_seas_mae, 4), n_periods = bench_n,
note = sprintf("Repeat %s as the forecast — the cheapest thing that respects the seasonal cycle.", seasonal_phrase),
stringsAsFactors = FALSE)
}
baseline_comparison_df <- do.call(rbind, bench_rows)
rownames(baseline_comparison_df) <- NULLStep 10: Chart datasets (both capped well under the row limit)
keep_n <- min(n_eval, 700L)
tail_idx <- (n_eval - keep_n + 1):n_eval
series_points_df <- rbind(
data.frame(period = format(hl_agg$date[tail_idx], "%Y-%m-%d"),
value = round(hl_agg$actual[tail_idx], 4),
series = actual_h, stringsAsFactors = FALSE),
data.frame(period = format(hl_agg$date[tail_idx], "%Y-%m-%d"),
value = round(hl_agg$forecast[tail_idx], 4),
series = forecast_h, stringsAsFactors = FALSE)
)
rownames(series_points_df) <- NULL
set.seed(42)
eidx <- if (n_eval > 2000) sample(n_eval, 2000) else seq_len(n_eval)
error_distribution_df <- data.frame(error = round(err[eidx], 4), stringsAsFactors = FALSE)
error_distribution_df <- error_distribution_df[order(error_distribution_df$error), , drop = FALSE]
rownames(error_distribution_df) <- NULLStep 11: Accuracy by calendar period
span_days <- as.numeric(max(hl_agg$date) - min(hl_agg$date))
if (span_days >= 3 * 365) {
bucket <- format(hl_agg$date, "%Y")
} else if (span_days >= 200) {
bucket <- paste0(format(hl_agg$date, "%Y"), " Q",
(as.integer(format(hl_agg$date, "%m")) - 1L) %/% 3L + 1L)
} else {
bucket <- format(hl_agg$date, "%Y-%m")
}
ub <- unique(bucket)
if (length(ub) > 12) {
per <- ceiling(length(ub) / 12)
grp <- ceiling(seq_along(ub) / per)
newlab <- tapply(ub, grp, function(x) if (length(x) == 1) x[1] else paste0(x[1], " to ", x[length(x)]))
bucket <- unname(newlab[as.character(grp[match(bucket, ub)])])
}
ape_all <- rep(NA_real_, n_eval)
ape_all[mape_ok] <- 100 * abs(err[mape_ok] / hl_agg$actual[mape_ok])
pa <- lapply(unique(bucket), function(b) {
ix <- bucket == b
data.frame(period_bucket = b,
n_periods = sum(ix),
mae = round(mean(abs(err[ix])), 4),
mean_error = round(mean(err[ix]), 4),
mape = if (any(mape_ok & ix)) round(mean(ape_all[ix], na.rm = TRUE), 3) else NA_real_,
stringsAsFactors = FALSE)
})
period_accuracy_df <- do.call(rbind, pa)
rownames(period_accuracy_df) <- NULL
pa_ok <- which(!is.na(period_accuracy_df$mae))
worst_bucket <- if (length(pa_ok) > 0) period_accuracy_df$period_bucket[pa_ok[which.max(period_accuracy_df$mae[pa_ok])]] else NA_character_
best_bucket <- if (length(pa_ok) > 0) period_accuracy_df$period_bucket[pa_ok[which.min(period_accuracy_df$mae[pa_ok])]] else NA_character_
worst_mae <- if (length(pa_ok) > 0) max(period_accuracy_df$mae[pa_ok]) else NA_real_
best_mae <- if (length(pa_ok) > 0) min(period_accuracy_df$mae[pa_ok]) else NA_real_Step 12: Accuracy by horizon
scale_for_mase <- if (seasonal_available) q_seas else q_naive
scale_label <- if (seasonal_available) "seasonal naive" else "naive"
if (horizon_usable) {
lv <- unique(work$horizon_label)
ordv <- sapply(lv, function(k) min(work$horizon_order[work$horizon_label == k]))
lv <- lv[order(ordv, lv)]
hz <- lapply(lv, function(k) {
sub <- work[work$horizon_label == k, , drop = FALSE]
sub <- stats::aggregate(cbind(forecast, actual) ~ date, data = sub, FUN = mean)
e <- sub$actual - sub$forecast
aok <- abs(sub$actual) > 1e-9 & (is.na(med_a) | abs(sub$actual) >= 0.01 * med_a)
data.frame(
horizon = k,
n_periods = nrow(sub),
mae = round(mean(abs(e)), 4),
rmse = round(sqrt(mean(e^2)), 4),
mean_error = round(mean(e), 4),
mape = if (sum(aok) >= 3) round(100 * mean(abs(e[aok] / sub$actual[aok])), 3) else NA_real_,
mase = if (!is.na(scale_for_mase)) round(mean(abs(e)) / scale_for_mase, 4) else NA_real_,
verdict = if (is.na(scale_for_mase)) "no baseline available"
else if (mean(abs(e)) / scale_for_mase < 1) sprintf("beats %s", scale_label)
else sprintf("does not beat %s", scale_label),
stringsAsFactors = FALSE)
})
horizon_accuracy_df <- do.call(rbind, hz)
} else {
horizon_accuracy_df <- data.frame(
horizon = sprintf("All forecasts(no '%s' column mapped)", horizon_h),
n_periods = n_eval, mae = round(mae, 4), rmse = round(rmse, 4),
mean_error = round(me, 4),
mape = if (is.na(mape)) NA_real_ else round(mape, 3),
mase = if (is.na(scale_for_mase)) NA_real_ else round(mae / scale_for_mase, 4),
verdict = if (is.na(scale_for_mase)) "no baseline available"
else if (mae / scale_for_mase < 1) sprintf("beats %s", scale_label)
else sprintf("does not beat %s", scale_label),
stringsAsFactors = FALSE)
}
rownames(horizon_accuracy_df) <- NULL
hz_ok <- which(!is.na(horizon_accuracy_df$mae))
horizon_degrades <- horizon_usable && length(hz_ok) >= 2 &&
horizon_accuracy_df$mae[hz_ok[length(hz_ok)]] > horizon_accuracy_df$mae[hz_ok[1]]
hz_ratio <- if (horizon_usable && length(hz_ok) >= 2 && horizon_accuracy_df$mae[hz_ok[1]] > 0) {
horizon_accuracy_df$mae[hz_ok[length(hz_ok)]] / horizon_accuracy_df$mae[hz_ok[1]]
} else NA_real_Step 13: The headline metric table
bias_dir <- if (me > 0) "under-forecast" else if (me < 0) "over-forecast" else "unbiased"
acc_rows <- list(
data.frame(metric = "Periods evaluated", value = n_eval,
interpretation = sprintf("Distinct %ss of '%s' scored against '%s'.", period_word, forecast_h, actual_h),
stringsAsFactors = FALSE),
data.frame(metric = "MAE(mean absolute error)", value = round(mae, 4),
interpretation = sprintf("Average miss, in units of '%s', ignoring direction.", actual_h),
stringsAsFactors = FALSE),
data.frame(metric = "RMSE(root mean squared error)", value = round(rmse, 4),
interpretation = sprintf("Average miss with large misses weighted more heavily; RMSE is %s times MAE here, so %s.",
r2(if (mae > 0) rmse / mae else NA_real_),
if (mae > 0 && rmse / mae > 1.5) "a minority of periods carries most of the damage" else "the misses are fairly evenly sized"),
stringsAsFactors = FALSE),
data.frame(metric = "Mean error(bias)", value = round(me, 4),
interpretation = sprintf("Actual minus forecast on average: the process %s by %s units of '%s' per %s.",
bias_dir, r2(abs(me)), actual_h, period_word),
stringsAsFactors = FALSE),
data.frame(metric = "Periods under-forecast(%)", value = round(pct_under, 2),
interpretation = sprintf("%d of %d periods came in above the forecast; %d came in below and %d landed exactly.",
n_under, n_eval, n_over, n_exact),
stringsAsFactors = FALSE),
data.frame(metric = "MAPE(%)", value = if (is.na(mape)) NA_real_ else round(mape, 3),
interpretation = if (is.na(mape))
sprintf("Not computable — too few periods with a usable non-zero '%s'.", actual_h)
else if (n_mape_excl > 0)
sprintf("Computed on %d of %d periods; %d were excluded because '%s' was zero or near zero, where a percentage error is undefined or explodes.",
n_mape_used, n_eval, n_mape_excl, actual_h)
else
sprintf("Computed on all %d periods; no zero or near-zero actuals were present.", n_eval),
stringsAsFactors = FALSE),
data.frame(metric = "sMAPE(%)", value = if (is.na(smape)) NA_real_ else round(smape, 3),
interpretation = "Symmetric percentage error: the denominator uses both the actual and the forecast, so it is bounded at 200 percent and does not blow up as the actual approaches zero.",
stringsAsFactors = FALSE),
data.frame(metric = "MASE vs naive", value = if (is.na(mase_naive)) NA_real_ else round(mase_naive, 4),
interpretation = if (is.na(mase_naive))
sprintf("Not computable — '%s' does not change from one %s to the next.", actual_h, period_word)
else sprintf("MAE divided by the naive baseline's average absolute change (%s). Below 1 means the forecast beats repeating the last %s.",
r2(q_naive), period_word),
stringsAsFactors = FALSE),
data.frame(metric = sprintf("MASE vs seasonal naive(%d %ss)", m, period_word),
value = if (is.na(mase_seas)) NA_real_ else round(mase_seas, 4),
interpretation = if (is.na(mase_seas))
sprintf("Not computable — the history does not cover enough full %ss for a seasonal baseline.", cycle_word)
else sprintf("MAE divided by the seasonal-naive baseline's average absolute change (%s). Below 1 means the forecast beats repeating %s.",
r2(q_seas), seasonal_phrase),
stringsAsFactors = FALSE)
)
accuracy_metrics_df <- do.call(rbind, acc_rows)
rownames(accuracy_metrics_df) <- NULLStep 14: Methods and disclosure
dm_detail <- function(res, label) {
if (is.null(res)) return(sprintf("Not run against the %s baseline — too few paired periods, or the loss differential has no variation.", label))
sprintf("Against the %s baseline: %d paired periods, mean loss differential %s(%s means the forecast's absolute error is %s), Newey-West lag %d, statistic %s, %s.",
label, res$n, r3(res$mean_d),
if (res$mean_d < 0) "negative" else "positive",
if (res$mean_d < 0) "smaller" else "larger",
res$lag, r3(res$stat), fmt_pp(res$p))
}
methods_df <- data.frame(
item = c("Error convention", "Scale-dependent error", "Percentage error",
"Symmetric percentage error", "Scaling baselines", "MASE",
"Diebold-Mariano test", "Calendar handling", "What this cannot tell you"),
detail = c(
sprintf("Every period's error is '%s' minus '%s', so a positive error means the outcome came in above the forecast (an under-forecast).", actual_h, forecast_h),
sprintf("MAE is the mean of the absolute errors(%s) and RMSE the square root of the mean squared error(%s), both in units of '%s'. RMSE exceeds MAE whenever the misses are uneven in size.",
r2(mae), r2(rmse), actual_h),
if (is.na(mape))
sprintf("MAPE could not be computed: too few periods had a usable non-zero '%s'.", actual_h)
else sprintf("MAPE is the mean of the absolute errors divided by the actual, computed on %d of %d periods(%d excluded for a zero or near-zero '%s', where the ratio is undefined or explodes). MAPE is also bounded at 100 percent when the forecast runs low but unbounded when it runs high, so it penalises over-forecasting more heavily and a process tuned to minimise it will drift low. Measured here: mean absolute percentage error of %s in the periods that ran high versus %s in the periods that ran low.",
n_mape_used, n_eval, n_mape_excl, actual_h, r2(ape_over), r2(ape_under)),
if (is.na(smape)) "sMAPE could not be computed on this data."
else sprintf("sMAPE(%s percent) divides twice the absolute error by the sum of the absolute actual and the absolute forecast, which bounds it at 200 percent and keeps it finite when the actual is small.", r2(smape)),
sprintf("The naive baseline repeats the previous %s's actual; its average absolute change over the observed series is %s. The seasonal-naive baseline repeats %s; %s. Both are computed on a regular %s calendar grid, so gaps in the data never shift the lag.",
period_word, r2(q_naive), seasonal_phrase,
if (is.na(q_seas)) sprintf("it is unavailable because the history does not cover enough full %ss", cycle_word)
else sprintf("its average absolute change is %s", r2(q_seas)),
freq_label),
sprintf("MASE is MAE divided by a baseline's average absolute change: %s against naive and %s against seasonal naive. The scaling factor is taken from the whole observed series rather than a separate training window, because a forecast log carries no train and test split; that makes these values comparable across series but not identical to a MASE computed from a held-out fit.",
r3(mase_naive), r3(mase_seas)),
paste(dm_detail(dm_naive, "naive"), dm_detail(dm_seas, "seasonal-naive"),
"The test is implemented directly here: the per-period loss differential under absolute-error loss is divided by a Newey-West long-run standard error with Bartlett weights, given the Harvey-Leybourne-Newbold small-sample correction, and referred to a t distribution.",
sep = " "),
sprintf("The cadence of '%s' was read as %s from the median gap between periods. %d of the %d calendar %ss spanned by the data carry no actual (%s percent). %s",
date_h, freq_label, n_grid_missing, n_grid, period_word, r2(pct_grid_missing),
if (pct_grid_missing > 10)
"With that many periods missing, the baselines are computed only where both ends of a lag are present, and they rest on fewer comparisons than the period count suggests."
else "Baseline lags line up with the calendar."),
sprintf("This is a scorecard of a forecasting process, not a diagnosis of it. It measures how far '%s' landed from '%s' and whether that beats a naive rule; it cannot say why the misses happened, and it cannot rule out that the forecasts were revised with knowledge the baselines did not have.",
forecast_h, actual_h)
),
stringsAsFactors = FALSE
)Step 15: Metrics and the one-paragraph answer
bench_verdict <- if (seasonal_available) {
if (beats_seasonal) "beats seasonal naive" else "does not beat seasonal naive"
} else if (!is.na(mase_naive)) {
if (beats_naive) "beats naive" else "does not beat naive"
} else "no baseline available"
metrics <- list(
`Periods Evaluated` = n_eval,
`MAE` = round(mae, 3),
`RMSE` = round(rmse, 3),
`Mean Error(Bias)` = round(me, 3),
`MAPE(%)` = if (is.na(mape)) NA_real_ else round(mape, 2),
`MASE vs Naive` = if (is.na(mase_naive)) NA_real_ else round(mase_naive, 3),
`MASE vs Seasonal Naive` = if (is.na(mase_seas)) NA_real_ else round(mase_seas, 3),
`Verdict` = bench_verdict
)
headline <- if (seasonal_available) {
if (beats_seasonal) {
sprintf("The forecast beats the seasonal-naive baseline: MAE of %s against the baseline's %s, a MASE of %s.",
r2(mae), r2(q_seas), r3(mase_seas))
} else {
sprintf("The forecast does not beat the seasonal-naive baseline: MAE of %s against the baseline's %s, a MASE of %s. A forecast that does not beat seasonal naive is not adding value — repeating %s would have been at least as accurate.",
r2(mae), r2(q_seas), r3(mase_seas), seasonal_phrase)
}
} else if (!is.na(mase_naive)) {
if (beats_naive) {
sprintf("No seasonal baseline was available, so the forecast is judged against the naive one, which it beats: MAE of %s against %s, a MASE of %s.",
r2(mae), r2(q_naive), r3(mase_naive))
} else {
sprintf("No seasonal baseline was available, so the forecast is judged against the naive one, which it does not beat: MAE of %s against %s, a MASE of %s. A forecast that does not beat the naive rule is not adding value.",
r2(mae), r2(q_naive), r3(mase_naive))
}
} else {
sprintf("Neither baseline could be computed, because '%s' does not change from one %s to the next; MAE of %s is reported without a scale-free comparison.",
actual_h, period_word, r2(mae))
}
dm_sentence <- function(res, label) {
if (is.null(res)) return("")
if (res$p < 0.05) {
sprintf(" A Diebold-Mariano test says that gap against the %s baseline is larger than sampling noise(statistic %s, %s), so the forecast is significantly %s than %s.",
label, r3(res$stat), fmt_pp(res$p),
if (res$mean_d < 0) "more accurate" else "less accurate", label)
} else {
sprintf(" A Diebold-Mariano test cannot separate the forecast from the %s baseline(statistic %s, %s), so on this evidence the two perform equivalently.",
label, r3(res$stat), fmt_pp(res$p))
}
}
dm_headline <- if (seasonal_available) dm_sentence(dm_seas, "seasonal-naive") else dm_sentence(dm_naive, "naive")
mape_clause <- if (is.na(mape)) {
sprintf(" MAPE is not reported: too few periods had a usable non-zero '%s', so MASE is the only reliable accuracy scale here.", actual_h)
} else if (n_mape_excl > 0) {
sprintf(" MAPE reads %s percent but rests on only %s of %s periods — %s were excluded because '%s' was zero or near zero — so lead with MASE, not MAPE.",
r2(mape), format(n_mape_used, big.mark = ","), format(n_eval, big.mark = ","),
format(n_mape_excl, big.mark = ","), actual_h)
} else {
sprintf(" MAPE reads %s percent across all %s periods.", r2(mape), format(n_eval, big.mark = ","))
}
bias_clause <- sprintf(" On average the process %s by %s units of '%s' per %s, running high in %s percent of periods and low in %s percent.",
bias_dir, r2(abs(me)), actual_h, period_word,
r2(pct_over), r2(pct_under))
json_output <- list(
answer = paste0(
"Backtest of ", format(n_eval, big.mark = ","), " ", freq_label,
" forecasts of '", actual_h, "'",
if (horizon_usable && !single_horizon) sprintf(" at the level of '%s' with the most observations (%s)", horizon_h, primary_label) else "",
": ", headline, dm_headline, bias_clause, mape_clause
),
cards = lapply(
c("tldr", "overview", "preprocessing", "actual_vs_forecast",
"accuracy_metrics", "baseline_comparison", "error_distribution",
"period_accuracy", "horizon_accuracy", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
date_h = date_h, forecast_h = forecast_h, actual_h = actual_h, horizon_h = horizon_h,
n_rows = n_rows, n_dropped = n_dropped, n_dup_rows = n_dup_rows,
n_unparsed = n_unparsed, n_eval = n_eval,
freq_label = freq_label, period_word = period_word, cycle_word = cycle_word,
m = m, seasonal_phrase = seasonal_phrase,
n_grid = n_grid, n_grid_missing = n_grid_missing, pct_grid_missing = pct_grid_missing,
mae = mae, rmse = rmse, me = me, mape = mape, smape = smape,
mape_reliable = mape_reliable,
n_zero = n_zero, n_near_zero = n_near_zero,
n_mape_excl = n_mape_excl, n_mape_used = n_mape_used,
ape_over = ape_over, ape_under = ape_under,
n_under = n_under, n_over = n_over, n_exact = n_exact,
pct_under = pct_under, pct_over = pct_over, pct_exact = pct_exact,
bias_dir = bias_dir,
q_naive = q_naive, q_seas = q_seas,
mase_naive = mase_naive, mase_seas = mase_seas,
beats_naive = beats_naive, beats_seasonal = beats_seasonal,
seasonal_available = seasonal_available, bench_verdict = bench_verdict,
bench_n = bench_n, bench_kind = bench_kind, bench_fc_mae = bench_fc_mae,
bench_naive_mae = bench_naive_mae, bench_seas_mae = bench_seas_mae,
dm_naive = dm_naive, dm_seas = dm_seas, dm_headline = dm_headline,
headline = headline, mape_clause = mape_clause, bias_clause = bias_clause,
has_horizon = has_horizon, horizon_usable = horizon_usable,
single_horizon = single_horizon, primary_label = primary_label,
n_horizons = n_horizons, horizon_note = horizon_note,
horizon_degrades = horizon_degrades, hz_ratio = hz_ratio,
worst_bucket = worst_bucket, best_bucket = best_bucket,
worst_mae = worst_mae, best_mae = best_mae,
series_points_df = series_points_df,
accuracy_metrics_df = accuracy_metrics_df,
baseline_comparison_df = baseline_comparison_df,
error_distribution_df = error_distribution_df,
period_accuracy_df = period_accuracy_df,
horizon_accuracy_df = horizon_accuracy_df,
methods_df = methods_df,
metrics = metrics, json_output = json_output
)
}