Executive Summary
What is associated with Weekly Orders across 600 observations
Marketing Spend is the strongest driver of order counts: each additional unit multiplies expected Weekly Orders by 1.54 (54% increase, very strong evidence, p < 0.001). Three of four drivers are statistically significant at p < 0.05. The negative binomial model is the headline because overdispersion = 2.14, well above the 1.0 a plain Poisson model assumes; Poisson significance tests would be overconfident. The model explains 40.7% of deviance in Weekly Orders across 600 observations.
Analysis Overview
Negative binomial regression of Weekly Orders on 4 drivers across 600 observations.
Order counts range from 0 to 23 across 600 observations, with a mean of 2.88 and median of 2. The distribution is right-skewed and zero-inflated (27.3% of rows at zero), making a count-specific model essential. A negative binomial regression with dispersion = 2.14 captures the variance structure that ordinary linear regression would mishandle. The model's log-link respects the non-negative integer nature of counts and interprets effects as multiplicative rate ratios rather than additive shifts. Four drivers were modeled; the analysis accounts for 40.7% of deviance in Weekly Orders.
Data Quality
Count validation, driver typing, and exclusions.
All 600 rows passed validation as non-negative integer counts, the requirement for an honest count model. Numeric drivers had missing values imputed with column medians; categorical drivers were grouped into 'Other' for rare levels and 'Missing' for blanks. Weekly Orders ranges from 0 to 23 with a median of 2 and mean of 2.88. No rows were removed during preprocessing. All four mapped drivers (Marketing Spend, Distance to Store, Plan Type, Random Noise) were usable in the final model, confirming complete data coverage for estimation.
Rate Ratios
Each driver's multiplicative effect on the expected Weekly Orders with 95% confidence intervals and plain-language significance.
| Term | Rate Ratio | CI Low | CI High | P Value | Significance |
|---|---|---|---|---|---|
| Baseline (intercept) | 1.633 | 1.279 | 2.084 | < 0.001 | very strong evidence |
| Marketing Spend | 1.539 | 1.457 | 1.626 | < 0.001 | very strong evidence |
| Distance to Store | 0.732 | 0.7 | 0.766 | < 0.001 | very strong evidence |
| Plan Type: Enterprise | 1.79 | 1.473 | 2.174 | < 0.001 | very strong evidence |
| Plan Type: Pro | 1.276 | 1.073 | 1.517 | 0.00592 | strong evidence |
| Random Noise | 0.997 | 0.97 | 1.023 | 0.795 | not significant |
Marketing Spend shows the strongest multiplicative effect: rate ratio = 1.54 (95% CI: 1.457–1.626, p < 0.001). Distance to Store is negatively associated with orders: rate ratio = 0.732 (CI: 0.70–0.766, p < 0.001), meaning each unit increase multiplies expected orders by 0.732. Plan Type: Enterprise has rate ratio = 1.79 (CI: 1.473–2.174, p < 0.001) versus baseline; Plan Type: Pro has rate ratio = 1.276 (CI: 1.073–1.517, p = 0.00592). Random Noise shows no reliable association: rate ratio = 0.997 (CI: 0.97–1.023, p = 0.795, not significant). The baseline intercept is 1.633 (CI: 1.279–2.084).
Driver Ranking
Relative statistical influence of each driver, scaled 0-100.
Marketing Spend dominates statistical influence with importance score 100, raising order counts with very strong evidence. Distance to Store ranks second at importance 87, lowering counts with very strong evidence. Plan Type ranks third at importance 37.9, with categorical effects of very strong evidence. Random Noise scores only 1.7 and is not significant, indistinguishable from noise in this dataset. Importance is scaled relative to Marketing Spend's z-statistic, providing a scale-free ranking of each driver's influence on Weekly Orders.
Count Distribution
How the observed Weekly Orders values are distributed.
Weekly Orders are right-skewed with mean 2.88 exceeding median 2, reflecting a tail of high-count observations. The range is 0 to 23. Zero-inflation is substantial: 27.3% of rows have exactly zero orders. This shape—whole numbers, a floor at zero, and a long right tail—violates the assumptions of ordinary linear regression, which treats counts as unbounded continuous values. The negative binomial model respects these structural features and produces valid confidence intervals and predictions constrained to non-negative integers.
Model Quality
Overdispersion verdict, model choice, and fit statistics for the negative binomial model.
| Metric | Value | Interpretation |
|---|---|---|
| Model used | Negative binomial | Chosen because dispersion = 2.14 — well above the 1.0 a Poisson model assumes, so a negative binomial model that absorbs the extra variance is the headline. |
| Overdispersion (Pearson chi-squared / df) | 2.14 | Above 1.5: counts vary more than a plain Poisson model allows, so Poisson p-values would be overconfident. |
| Pseudo R-squared (explained deviance) | 0.407 | The drivers account for 40.7% of the deviance in Weekly Orders. |
| Predicted vs observed correlation | 0.63 | Predicted counts capture the broad pattern; substantial row-level noise remains. |
| Observations modeled | 600 | Rows with a valid Weekly Orders value used in the model. |
The negative binomial model was selected because overdispersion = 2.14, substantially above the 1.0 a plain Poisson model assumes. This overdispersion means counts vary far more than Poisson allows; Poisson significance tests would be overconfident. The model accounts for 40.7% of deviance (pseudo R-squared = 0.407), explaining substantial systematic variation in Weekly Orders. Predicted versus observed counts correlate at 0.63, indicating the drivers capture the broad pattern while substantial row-level noise remains. The rate ratio confidence intervals are the measure of driver effect reliability, not individual row predictions.
Methodology
Statistical methodology and diagnostics for Count Outcomes — Poisson & Negative Binomial
Statistical Method
Standard-library analysis: model event counts (orders, tickets, defects, visits) on the drivers you choose — the statistically correct way. Fits a Poisson regression, tests for overdispersion, and automatically upgrades to a negative binomial (or quasi-Poisson) model when the data demands it. Every effect is reported as a rate ratio — 'each unit of X multiplies the expected count by Y' — with confidence intervals and significance, plus a driver ranking and a model-quality readout. Works on any dataset: map a count column and 1-8 drivers.
- The outcome is a true count: non-negative whole numbers of events per row
- Drivers act multiplicatively on the expected count (log link)
- Observations are independent
- For Poisson results: variance roughly equals the mean (checked via the dispersion statistic; the model upgrades automatically when violated)
- Rate ratios describe association, not causation
- Very heavy zero-inflation beyond what a negative binomial absorbs may need a dedicated zero-inflated model
- Counts from unequal exposure windows (e.g. stores open different numbers of days) are not offset-adjusted in this tool
Analysis Code
Complete R source code for this analysis
Count Outcomes — Poisson & Negative Binomial
Models an event count (orders, tickets, defects, visits) on one or more driver columns chosen by the user. Fits a Poisson regression, tests for overdispersion via the Pearson dispersion statistic, and upgrades to a negative binomial model (quasi-Poisson fallback) when the data demands it. Reports every effect as a rate ratio with confidence intervals and significance, ranks drivers by statistical influence, and surfaces model quality.
Why This Method?
Counts are non-negative integers, usually right-skewed with many zeros, and their variance grows with the mean — every one of which violates ordinary linear regression. A log-link count model respects all of it, and its exponentiated coefficients read directly as business English: "each unit of X multiplies the expected count by Y."
What This Analysis Covers
- Count-column validation (non-negative whole numbers) and distribution
- Automatic overdispersion test and model selection (Poisson vs NB)
- Rate ratio table with 95% confidence intervals and significance
- Driver ranking by statistical influence
- Model quality: dispersion, pseudo R-squared, predicted-vs-actual fit
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {count, driver_1..driver_N}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Step 1: Row accounting + semantic column discovery
initial_rows <- nrow(df)
if (!"count" %in% names(df)) {
stop("column_mapping must map a 'count' column (the event count to explain)")
}
driver_cols <- grep("^driver_[0-9]+$", names(df), value = TRUE)
driver_cols <- driver_cols[order(as.integer(sub("^driver_", "", driver_cols)))]
if (length(driver_cols) == 0) {
stop("column_mapping must map at least one driver column(driver_1)")
}
count_name <- humanize_semantic("count", col_map)
driver_names <- setNames(humanize_semantic(driver_cols, col_map), driver_cols)Step 2: Validate the count column — non-negative near-integers
A count model is only honest on true event counts. Require >= 95% of non-blank values to be numeric, non-negative, and whole (tolerance 1e-6); otherwise stop with a humanized, actionable message.
raw_chr <- as.character(df$count)
nonblank <- !is.na(df$count) & trimws(raw_chr) != ""
n_nonblank <- sum(nonblank)
if (n_nonblank < 30) {
stop(sprintf(
"Only %d rows have a value in the count column('%s'). At least 30 are required for a count regression.",
n_nonblank, count_name))
}
conv <- suppressWarnings(as.numeric(raw_chr))
ok <- nonblank & !is.na(conv) & conv >= 0 & abs(conv - round(conv)) <= 1e-6
if (sum(ok) < 0.95 * n_nonblank) {
stop(sprintf(
paste0("The column mapped as the count('%s') must contain non-negative whole ",
"numbers — events per row, like orders, tickets, or visits. %s of %s ",
"values are fractional, negative, or non-numeric, so a count model ",
"cannot be fit. If this outcome is a continuous measure(revenue, ",
"duration, a rate), use the linear regression tool instead."),
count_name,
format(n_nonblank - sum(ok), big.mark = ","),
format(n_nonblank, big.mark = ",")))
}
df$count <- round(conv)
df$count[!ok] <- NA_real_
df <- df[!is.na(df$count), , drop = FALSE]
if (nrow(df) < 30) {
stop(sprintf(
"Only %d rows have a usable value in the count column('%s') after validation. At least 30 are required.",
nrow(df), count_name))
}Step 3: Type each driver — numeric if >=95% of values convert, else factor
dropped_drivers <- character(0)
for (dc in driver_cols) {
v <- df[[dc]]
if (!is.numeric(v)) {
conv_d <- suppressWarnings(as.numeric(as.character(v)))
n_orig <- sum(!is.na(v) & as.character(v) != "")
if (n_orig > 0 && sum(!is.na(conv_d)) >= 0.95 * n_orig) {
df[[dc]] <- conv_d
}
}
v <- df[[dc]]
if (is.numeric(v)) {Numeric: impute NA with median
med <- median(v, na.rm = TRUE)
if (is.na(med)) { dropped_drivers <- c(dropped_drivers, dc); next }
v[is.na(v)] <- med
df[[dc]] <- v
} else {Categorical: blank/NA -> "Missing"; lump beyond 12 levels into "Other"
v <- as.character(v)
v[is.na(v) | trimws(v) == ""] <- "Missing"
if (length(unique(v)) > nrow(df) / 2) {Near-unique text column (an ID, not a driver) — exclude BEFORE lumping, or the lump step disguises identifiers as 12 categories.
dropped_drivers <- c(dropped_drivers, dc)
next
}
tab <- sort(table(v), decreasing = TRUE)
if (length(tab) > 12) {
keep <- names(tab)[1:12]
v[!(v %in% keep)] <- "Other"
}
df[[dc]] <- factor(v)
}
}Step 4: Drop zero-variance drivers
for (dc in setdiff(driver_cols, dropped_drivers)) {
v <- df[[dc]]
zero_var <- if (is.numeric(v)) {
isTRUE(var(v, na.rm = TRUE) == 0) || is.na(var(v, na.rm = TRUE))
} else {
length(unique(v)) <= 1
}
if (zero_var) dropped_drivers <- c(dropped_drivers, dc)
}
model_drivers <- setdiff(driver_cols, dropped_drivers)
if (length(model_drivers) == 0) {
stop("No usable driver columns remained after cleaning(all were constant, empty, or identifier-like).")
}
df_clean <- df[, c(model_drivers, "count"), drop = FALSE]
final_rows <- nrow(df_clean)
rows_removed <- initial_rows - final_rowsStep 5: Guard — need clearly more rows than model terms
n_terms_est <- sum(sapply(model_drivers, function(dc) {
if (is.factor(df_clean[[dc]])) nlevels(df_clean[[dc]]) - 1 else 1
}))
while (n_terms_est >= final_rows - 5 && length(model_drivers) > 1) {
drop_dc <- model_drivers[length(model_drivers)]
dropped_drivers <- c(dropped_drivers, drop_dc)
model_drivers <- model_drivers[-length(model_drivers)]
df_clean <- df_clean[, c(model_drivers, "count"), drop = FALSE]
n_terms_est <- sum(sapply(model_drivers, function(dc) {
if (is.factor(df_clean[[dc]])) nlevels(df_clean[[dc]]) - 1 else 1
}))
}Step 6: Fit Poisson; test overdispersion; upgrade if needed
Dispersion = Pearson chi-squared / residual df. Equidispersed Poisson data sits near 1. Above 1.5 we refit as negative binomial (MASS ships with base R distributions but availability is verified defensively); if MASS is unavailable or glm.nb fails to converge, quasi-Poisson keeps the point estimates and corrects the standard errors.
fit_pois <- glm(count ~ ., family = poisson(), data = df_clean)
df_res <- df.residual(fit_pois)
dispersion <- if (df_res > 0) {
sum(residuals(fit_pois, type = "pearson")^2) / df_res
} else NA_real_
overdispersed <- is.finite(dispersion) && dispersion > 1.5
model <- fit_pois
model_label <- "Poisson"
model_reason <- paste0(
"dispersion = ", round(dispersion, 2),
", close enough to the 1.0 a Poisson model assumes")
if (overdispersed) {
nb_fit <- NULL
if (requireNamespace("MASS", quietly = TRUE)) {
nb_fit <- tryCatch(
suppressWarnings(MASS::glm.nb(count ~ ., data = df_clean)),
error = function(e) NULL)
}
if (!is.null(nb_fit)) {
model <- nb_fit
model_label <- "Negative binomial"
model_reason <- paste0(
"dispersion = ", round(dispersion, 2),
" — well above the 1.0 a Poisson model assumes, so a negative ",
"binomial model that absorbs the extra variance is the headline")
} else {
model <- glm(count ~ ., family = quasipoisson(), data = df_clean)
model_label <- "Quasi-Poisson"
model_reason <- paste0(
"dispersion = ", round(dispersion, 2),
" — well above the 1.0 a Poisson model assumes; a negative binomial ",
"fit was unavailable, so quasi-Poisson corrects the standard errors")
}
}
pseudo_r2 <- 1 - model$deviance / model$null.deviance
if (!is.finite(pseudo_r2)) pseudo_r2 <- NA_real_Step 7: Rate ratios — exp(coef) with 95% CIs (Wald, tryCatch-guarded)
tidy_df <- broom::tidy(model)
ci <- tryCatch(suppressMessages(confint.default(model)),
error = function(e) NULL)
if (!is.null(ci) && all(tidy_df$term %in% rownames(ci))) {
ci_lo_raw <- ci[tidy_df$term, 1]
ci_hi_raw <- ci[tidy_df$term, 2]
} else {
ci_lo_raw <- tidy_df$estimate - 1.96 * tidy_df$std.error
ci_hi_raw <- tidy_df$estimate + 1.96 * tidy_df$std.error
}
safe_exp <- function(x) {
out <- exp(x)
out[!is.finite(out)] <- NA_real_
round(out, 3)
}
rate_ratios_df <- data.frame(
term = humanize_term(tidy_df$term, model_drivers, col_map),
rate_ratio = safe_exp(tidy_df$estimate),
ci_low = safe_exp(ci_lo_raw),
ci_high = safe_exp(ci_hi_raw),
p_value = format_p(tidy_df$p.value),
significance = sig_word(tidy_df$p.value),
stringsAsFactors = FALSE
)
# broom/coef carry term rownames; reset so the renderer adds no "Row" column.
rownames(rate_ratios_df) <- NULLStep 8: Driver importance — max |z| per driver, NA-filtered, 0-100
term_raw <- tidy_df$term[tidy_df$term != "(Intercept)"]
term_z <- tidy_df$statistic[tidy_df$term != "(Intercept)"]
term_est <- tidy_df$estimate[tidy_df$term != "(Intercept)"]
term_p <- tidy_df$p.value[tidy_df$term != "(Intercept)"]
imp_rows <- lapply(model_drivers, function(dc) {
idx <- which(startsWith(term_raw, dc))Aliased (perfectly collinear) driver terms carry NA statistics — no estimable independent effect. Filter NA BEFORE which.max so an all-NA driver is excluded instead of crashing (LAT-1445 class).
idx <- idx[!is.na(term_z[idx])]
if (length(idx) == 0) return(NULL)
best <- idx[which.max(abs(term_z[idx]))]
data.frame(
semantic = dc,
driver = driver_names[[dc]],
abs_z = abs(term_z[best]),
direction = if (is.numeric(df_clean[[dc]])) {
if (term_est[best] > 0) "positive" else "negative"
} else "categorical",
p_raw = term_p[best],
best_term = term_raw[best],
best_est = term_est[best],
stringsAsFactors = FALSE
)
})
importance_df <- do.call(rbind, Filter(Negate(is.null), imp_rows))
if (!is.null(importance_df) && nrow(importance_df) > 0) {
importance_df <- importance_df[order(-importance_df$abs_z), , drop = FALSE]
max_z <- max(importance_df$abs_z, na.rm = TRUE)
importance_df$importance_score <- if (is.finite(max_z) && max_z > 0) {
round(100 * importance_df$abs_z / max_z, 1)
} else 0
importance_df$significance <- sig_word(importance_df$p_raw)
rownames(importance_df) <- NULL
} else {
importance_df <- data.frame(
semantic = character(0), driver = character(0), abs_z = numeric(0),
direction = character(0), p_raw = numeric(0), best_term = character(0),
best_est = numeric(0), importance_score = numeric(0),
significance = character(0), stringsAsFactors = FALSE)
}
top_driver_name <- if (nrow(importance_df) > 0) importance_df$driver[1] else "Unknown"
top_term_rr <- if (nrow(importance_df) > 0) round(exp(importance_df$best_est[1]), 2) else NA_real_
top_term_p <- if (nrow(importance_df) > 0) importance_df$p_raw[1] else NA_real_
top_is_numeric <- if (nrow(importance_df) > 0) importance_df$direction[1] != "categorical" else FALSE
top_term_label <- if (nrow(importance_df) > 0) {
humanize_term(importance_df$best_term[1], model_drivers, col_map)
} else "Unknown"Step 9: Count distribution — <=1500 sample + zero share
set.seed(42)
didx <- if (final_rows > 1500) sample(final_rows, 1500) else seq_len(final_rows)
count_distribution_df <- data.frame(
observed_count = as.integer(df_clean$count[didx]),
stringsAsFactors = FALSE
)
pct_zero <- round(100 * mean(df_clean$count == 0), 1)
count_mean <- mean(df_clean$count)
count_med <- median(df_clean$count)
count_max <- max(df_clean$count)Step 10: Predicted vs actual counts — fit check on the response scale
predicted <- as.numeric(fitted(model))
pred_actual_cor <- suppressWarnings(
cor(predicted, df_clean$count, use = "complete.obs"))Step 11: Significant drivers + KPI metrics
n_sig <- sum(importance_df$p_raw < 0.05, na.rm = TRUE)
metrics <- list(
`Observations` = final_rows,
`Model Used` = model_label,
`Dispersion` = round(dispersion, 2),
`Pseudo R Squared` = round(pseudo_r2, 3),
`Significant Drivers` = as.integer(n_sig),
`Top Driver` = top_driver_name
)Step 12: Model quality table — all values as strings (no e-notation)
model_quality_df <- data.frame(
metric = c("Model used",
"Overdispersion(Pearson chi-squared / df)",
"Pseudo R-squared(explained deviance)",
"Predicted vs observed correlation",
"Observations modeled"),
value = c(model_label,
formatC(dispersion, digits = 2, format = "f"),
formatC(pseudo_r2, digits = 3, format = "f"),
formatC(pred_actual_cor, digits = 2, format = "f"),
format(final_rows, big.mark = ",")),
interpretation = c(
paste0("Chosen because ", model_reason, "."),
if (overdispersed) {
"Above 1.5: counts vary more than a plain Poisson model allows, so Poisson p-values would be overconfident."
} else {
"Near 1: the variance tracks the mean, so the plain Poisson model is adequate."
},
paste0("The drivers account for ", round(100 * pseudo_r2, 1),
"% of the deviance in ", count_name, "."),
if (is.finite(pred_actual_cor) && pred_actual_cor >= 0.7) {
"Predicted counts track observed counts closely."
} else if (is.finite(pred_actual_cor) && pred_actual_cor >= 0.4) {
"Predicted counts capture the broad pattern; substantial row-level noise remains."
} else {
"Predicted counts explain only part of the row-level variation — counts are individually noisy."
},
paste0("Rows with a valid ", count_name, " value used in the model.")),
stringsAsFactors = FALSE
)Step 13: json_output machine channel
n_terms_fit <- nrow(rate_ratios_df) - 1
json_output <- list(
answer = paste0(
model_label, " regression of ", count_name, " on ",
length(model_drivers),
if (length(model_drivers) == 1) " driver" else " drivers",
" across ", format(final_rows, big.mark = ","), " rows ",
"(overdispersion statistic = ", round(dispersion, 2),
if (overdispersed) ", so plain Poisson was rejected" else "", "). ",
top_driver_name, " is the strongest driver: ",
if (top_is_numeric) {
paste0("each additional unit multiplies the expected ", count_name,
" by ", top_term_rr)
} else {
paste0(top_term_label, " multiplies the expected ", count_name,
" by ", top_term_rr, " versus the baseline")
},
" (", sig_word(top_term_p), "). ",
n_sig, " of ", nrow(importance_df),
if (nrow(importance_df) == 1) " driver is" else " drivers are",
" significant at p < 0.05; pseudo R-squared = ", round(pseudo_r2, 3), "."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "rate_ratios_table",
"driver_importance", "count_distribution", "model_quality"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows,
final_rows = final_rows,
rows_removed = rows_removed,
count_name = count_name,
driver_names = driver_names,
model_drivers = model_drivers,
dropped_drivers = dropped_drivers,
df_clean = df_clean,
model = model,
model_label = model_label,
model_reason = model_reason,
dispersion = dispersion,
overdispersed = overdispersed,
pseudo_r2 = pseudo_r2,
rate_ratios_df = rate_ratios_df,
importance_df = importance_df,
top_driver_name = top_driver_name,
top_term_rr = top_term_rr,
top_term_p = top_term_p,
top_is_numeric = top_is_numeric,
top_term_label = top_term_label,
count_distribution_df = count_distribution_df,
pct_zero = pct_zero,
count_mean = count_mean,
count_med = count_med,
count_max = count_max,
pred_actual_cor = pred_actual_cor,
model_quality_df = model_quality_df,
n_sig = n_sig,
metrics = metrics,
json_output = json_output
)
}Compute shared resources
shared <- compute_shared(df, params, col_map)