Executive Summary
Anomaly scan across 4 columns
8 of 500 transactions (1.6%) are flagged as anomalous at the 99.9% threshold. The single most extreme row is Row 468 with an anomaly score of 171.16—9.3 times the threshold. It stands out because of Account Age: its value of -185.854 sits far below the typical value of 974.382, placing it 7.34 robust standard deviations away. This is a data-quality issue rather than a behavioral outlier—a negative account age is impossible and suggests a data entry or system error that warrants immediate investigation.
Analysis Overview
Multivariate anomaly scan across 4 columns and 500 observations.
Each transaction is scored by its squared Mahalanobis distance from the multivariate center across 4 columns: Transaction Amount, Session Duration, Items Purchased, and Account Age. This distance captures unusual combinations of values even when individual columns appear normal in isolation. The method is scale-free and accounts for how columns co-vary. Rows exceeding the 99.9% chi-square threshold of 18.47 are flagged as anomalies. Of 500 observations, 8 rows (1.6%) exceeded this threshold. The dominant driving feature for each anomaly is identified using robust z-scores (median/MAD), which are resistant to distortion by extreme values themselves.
Data Quality
Column typing, imputation, and exclusions.
All 500 rows were retained; no rows were removed during preprocessing. Missing values were imputed using each column's median, preserving sample size and avoiding deletion bias. All mapped columns were usable; no constant columns were present in the scanned feature set. The dataset required no rows to be dropped, and median imputation ensured complete data for covariance estimation. This approach maintains the integrity of the multivariate distance calculation while handling missing data conservatively.
Anomaly Score Distribution
Distribution of per-row anomaly scores with the flag threshold.
The distribution of anomaly scores shows a dense concentration at the left (normal behavior) with a long right tail extending to 171.16. The bulk of rows cluster below 10 in anomaly score; only 8 rows breach the 18.47 threshold. The maximum score of 171.16 represents an extreme departure, while the next-highest flagged anomalies (Row 133 at 168.5, Row 260 at 145.27, Row 77 at 131.08) remain substantially elevated. This pattern indicates that the 8 flagged transactions are genuine statistical outliers rather than a continuous gradation—they form a distinct tail separated from normal transaction behavior.
Top Anomalies
The most anomalous rows with the feature driving each one.
| Row ID | Anomaly Score | Dominant Feature | Dominant Value | Typical Value | Direction |
|---|---|---|---|---|---|
| Row 468 | 171.2 | Account Age | -185.9 | 974.4 | below typical |
| Row 133 | 168.5 | Items Purchased | 49.23 | 20.49 | above typical |
| Row 260 | 145.3 | Transaction Amount | -33.52 | 51.14 | below typical |
| Row 77 | 131.1 | Session Duration | -29.69 | 202.3 | below typical |
| Row 25 | 128.2 | Transaction Amount | 112.6 | 51.14 | above typical |
| Row 402 | 108.9 | Items Purchased | -5.057 | 20.49 | below typical |
| Row 199 | 85.14 | Account Age | 2028 | 974.4 | above typical |
| Row 314 | 65.85 | Session Duration | 408.8 | 202.3 | above typical |
| Row 18 | 9.01 | Account Age | 1456 | 974.4 | above typical |
| Row 230 | 7.63 | Session Duration | 312.4 | 202.3 | above typical |
| Row 473 | 7.23 | Session Duration | 96.11 | 202.3 | below typical |
| Row 152 | 7.06 | Items Purchased | 10.04 | 20.49 | below typical |
| Row 466 | 7.03 | Session Duration | 100.9 | 202.3 | below typical |
| Row 437 | 6.74 | Session Duration | 297.6 | 202.3 | above typical |
| Row 14 | 6.72 | Items Purchased | 9.802 | 20.49 | below typical |
The 15 most anomalous rows reveal diverse drivers: Session Duration dominates 6 of the 15 most unusual rows, while Account Age and Items Purchased each appear as dominant features in multiple anomalies. Among the 8 flagged anomalies (score > 18.47), Account Age is most common (2 of 8). Row 468 (Account Age: -185.854 vs. typical 974.382) and Row 133 (Items Purchased: 49.233 vs. typical 20.486) rank as the two most extreme. Deviations run in both directions: Row 468 and Row 260 sit far below typical values, while Row 25 (Transaction Amount: 112.642 vs. typical 51.137) and Row 199 (Account Age: 2027.74 vs. typical 974.382) sit far above. This diversity suggests independent issues rather than a single systematic problem.
Anomaly Map
The dataset in 2D with anomalies highlighted.
The 8 flagged anomalies scatter across different directions when projected onto the dataset's two principal components (PC1: 83.5% variance; PC2: 6.5% variance), appearing as isolated one-off outliers rather than clustering in a single systematic direction. No subset of anomalies groups together, and they do not align along a single axis of variation. This pattern indicates that the unusual transactions arise from independent, unrelated deviations rather than a common root cause (e.g., a systematic data collection error or fraud ring). Each anomaly represents a distinct type of irregularity that requires separate investigation.
Which Columns Drive Anomalies
Per-column maximum robust deviation across the anomalous rows.
Transaction Amount drives the most extreme deviations among anomalous rows, with a maximum robust z-score of 8.08—the largest deviation across all columns. Items Purchased and Account Age tie at 7.34 robust standard deviations, while Session Duration reaches 5.94. Every scanned column contributes at least one extreme value among the 8 flagged anomalies, indicating that unusual transactions are not confined to a single feature. Transaction Amount warrants the closest scrutiny for data-quality and fraud-detection purposes, but the breadth of involvement across all four features suggests that comprehensive validation across the full transaction profile is necessary.
Methodology
Statistical methodology and diagnostics for Anomaly Detection — Outlier Finder
Statistical Method
Standard-library analysis: find the unusual rows in your dataset across several numeric columns. Robust per-column z-scores (median/MAD) combined with multivariate Mahalanobis distance flag the rows that don't fit the overall pattern — with the driving column named for every anomaly, a score histogram, a 2D anomaly map, and the columns that cause the most trouble. Works on any dataset: map 2 or more numeric columns.
- The bulk of rows represent normal behavior (anomalies are a small minority)
- Columns are numeric or cleanly convertible
- Mahalanobis distance assumes roughly elliptical (correlated-normal) structure in the normal bulk
- Flags statistical outliers — whether they are errors, fraud, or genuine rare events needs domain judgment
- Strongly clustered or multi-modal data can make normal rows in small clusters look anomalous
- Extreme anomalies can inflate the covariance estimate and partially mask each other
Analysis Code
Complete R source code for this analysis
Anomaly Detection — Outlier Finder
Finds the unusual rows in a dataset across several numeric columns. Every row is scored by its Mahalanobis distance from the multivariate center of the mapped columns; rows beyond the 99.9% chi-square threshold are flagged, and each anomaly is explained by its dominant feature — the column with the largest robust z-score (median/MAD).
Why This Method?
Mahalanobis distance is scale-free and correlation-aware: a row can be flagged for an unusual COMBINATION of values even when no single column looks extreme. Robust per-column z-scores (median/MAD) resist the very outliers they are meant to find, and give each anomaly a plain-language explanation: which column, how far, and in which direction.
What This Analysis Covers
- Anomaly score per row + 99.9% threshold flags (and the top 1% by score)
- Top anomalies ranked and explained (dominant feature, value vs typical)
- Score distribution histogram and a 2D PCA anomaly map
- Per-column deviation profile across the anomalous rows
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {feature_1..feature_N}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
compute_shared <- function(df, params, col_map = list()) {
# === SHARED EXPORTS ===
# initial_rows/final_rows/rows_removed $ row accounting
# feature_names $ named character — semantic -> humanized names
# used_features $ character — semantic names used
# dropped_features $ character — excluded columns
# scores $ numeric — squared Mahalanobis distance per row
# score_method $ "mahalanobis" | "mahalanobis_ridge" | "max_robust_z"
# threshold $ numeric — 99.9% flag threshold on the score
# flag $ logical — anomaly flags per row
# n_anom / n_top1 $ counts: flagged rows, top-1%-by-score rows
# dom_idx $ integer — per-row dominant feature index (NA-safe)
# Z $ matrix — robust z per row x feature
# top_anomalies_df $ data.frame(row_id, anomaly_score, dominant_feature,
# dominant_value, typical_value, direction) — top 15
# score_distribution_df $ data.frame(anomaly_score) — <=5000 sample
# anomaly_map_df $ data.frame(pc1, pc2, status) — <=1000, all anomalies
# pc_var $ numeric(2) — % variance of PC1/PC2
# feature_deviations_df $ data.frame(feature, max_robust_z)
# extreme $ list — most extreme row (id, score, feature, ...)
# metrics / json_output
# === /SHARED EXPORTS ===Step 1: Discover mapped features
initial_rows <- nrow(df)
feat_cols <- grep("^feature_[0-9]+$", names(df), value = TRUE)
feat_cols <- feat_cols[order(as.integer(sub("^feature_", "", feat_cols)))]
if (length(feat_cols) < 2) {
stop("column_mapping must map at least two feature columns(feature_1, feature_2)")
}
feature_names <- setNames(humanize_semantic(feat_cols, col_map), feat_cols)Step 2: Coerce numeric (95% rule); impute median; drop unusable
dropped_features <- character(0)
for (fc in feat_cols) {
v <- df[[fc]]
if (!is.numeric(v)) {
conv <- suppressWarnings(as.numeric(as.character(v)))
n_orig <- sum(!is.na(v) & as.character(v) != "")
if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
df[[fc]] <- conv
} else {
dropped_features <- c(dropped_features, fc); next
}
}
v <- df[[fc]]
med <- median(v, na.rm = TRUE)
if (is.na(med)) { dropped_features <- c(dropped_features, fc); next }
v[is.na(v)] <- med
df[[fc]] <- v
if (isTRUE(var(v) == 0) || is.na(var(v))) {
dropped_features <- c(dropped_features, fc)
}
}
used_features <- setdiff(feat_cols, dropped_features)
if (length(used_features) < 2) {
stop(paste0("Anomaly detection needs at least two usable numeric columns; only ",
length(used_features), " remained after cleaning(",
paste(feature_names[used_features], collapse = ", "), ")."))
}
X <- as.matrix(df[, used_features, drop = FALSE])
final_rows <- nrow(X)
rows_removed <- initial_rows - final_rows
if (final_rows < 20) {
stop(sprintf("Only %d usable rows — anomaly detection needs at least 20.", final_rows))
}
k <- length(used_features)
hn <- unname(feature_names[used_features])Step 3: Robust z per column — (x - median) / (1.4826 * MAD),
MAD == 0 -> sd fallback, both 0 -> z = 0
robust_z <- function(v) {
med <- median(v)
s <- 1.4826 * median(abs(v - med))
if (!is.finite(s) || s == 0) s <- sd(v)
if (!is.finite(s) || s == 0) return(rep(0, length(v)))
(v - med) / s
}
Z <- vapply(seq_len(k), function(j) robust_z(X[, j]), numeric(final_rows))
if (is.null(dim(Z))) Z <- matrix(Z, nrow = final_rows)
colnames(Z) <- used_features
absZ <- abs(Z)Step 4: Anomaly score — squared Mahalanobis distance from the
multivariate center (ridge-regularized covariance if singular; fallback to per-row max |robust z| if both fail)
ctr <- colMeans(X)
S <- stats::cov(X)
score_method <- "mahalanobis"
scores <- tryCatch(
stats::mahalanobis(X, center = ctr, cov = S),
error = function(e) NULL
)
if (is.null(scores)) {
score_method <- "mahalanobis_ridge"
S_r <- S + diag(1e-6 * mean(diag(S)), k)
scores <- tryCatch(
stats::mahalanobis(X, center = ctr, cov = S_r),
error = function(e) NULL
)
}
if (is.null(scores)) {
score_method <- "max_robust_z"
scores <- apply(absZ, 1, function(r) {
ok <- r[is.finite(r)]
if (length(ok) == 0) 0 else max(ok)
})
}
scores[!is.finite(scores)] <- 0Step 5: Threshold flags — 99.9% chi-square on the squared distance
(5-robust-sigma when on the fallback score) + top 1% by score
threshold <- if (score_method == "max_robust_z") 5 else qchisq(0.999, df = k)
flag <- scores > threshold
n_anom <- sum(flag)
n_top1 <- max(1L, as.integer(ceiling(0.01 * final_rows)))
ord <- order(-scores)
top1_cutoff <- scores[ord[n_top1]]Step 6: Per-row dominant feature — max |robust z|, all-NA guarded
dom_idx <- vapply(seq_len(final_rows), function(i) {
r <- absZ[i, ]
ok <- which(is.finite(r))
if (length(ok) == 0) return(NA_integer_)
ok[which.max(r[ok])]
}, integer(1))Step 7: Top anomalies table — top 15 by score, explained
col_medians <- apply(X, 2, median)
top_idx <- head(ord, 15)
top_anomalies_df <- do.call(rbind, lapply(top_idx, function(i) {
j <- dom_idx[i]
if (is.na(j)) {
data.frame(row_id = paste0("Row ", i),
anomaly_score = round(scores[i], 2),
dominant_feature = "(not determinable)",
dominant_value = NA_real_, typical_value = NA_real_,
direction = "", stringsAsFactors = FALSE)
} else {
data.frame(
row_id = paste0("Row ", i),
anomaly_score = round(scores[i], 2),
dominant_feature = hn[j],
dominant_value = round(X[i, j], 3),
typical_value = round(col_medians[j], 3),
direction = if (X[i, j] >= col_medians[j]) "above typical" else "below typical",
stringsAsFactors = FALSE
)
}
}))
rownames(top_anomalies_df) <- NULLStep 8: Score distribution — all scores, <=5000 sampled
set.seed(42)
sd_idx <- if (final_rows > 5000) sample(final_rows, 5000) else seq_len(final_rows)
score_distribution_df <- data.frame(anomaly_score = round(scores[sd_idx], 3),
stringsAsFactors = FALSE)Step 9: Anomaly map — PCA projection to 2D (as standard_pca does),
<=1000 rows ALWAYS including every flagged anomaly
pca <- prcomp(X, center = TRUE, scale. = TRUE)
eig <- pca$sdev^2
pc_var <- round(100 * eig / sum(eig), 1)
if (length(pc_var) < 2) pc_var <- c(pc_var, 0)
anom_rows <- which(flag)
if (length(anom_rows) > 1000) anom_rows <- ord[seq_len(1000)][flag[ord[seq_len(1000)]]]
norm_rows <- setdiff(seq_len(final_rows), anom_rows)
budget <- max(0, 1000 - length(anom_rows))
set.seed(42)
if (length(norm_rows) > budget) norm_rows <- sample(norm_rows, budget)
map_rows <- sort(c(anom_rows, norm_rows))
anomaly_map_df <- data.frame(
pc1 = round(pca$x[map_rows, 1], 3),
pc2 = if (ncol(pca$x) >= 2) round(pca$x[map_rows, 2], 3) else 0,
status = ifelse(flag[map_rows], "anomaly", "normal"),
stringsAsFactors = FALSE
)
rownames(anomaly_map_df) <- NULLStep 10: Feature deviations — per-feature max |robust z| across the
anomalous rows (top 1% by score when nothing crosses the threshold)
dev_rows <- if (n_anom > 0) which(flag) else head(ord, n_top1)
feature_deviations_df <- data.frame(
feature = hn,
max_robust_z = vapply(seq_len(k), function(j) {
r <- absZ[dev_rows, j]
ok <- r[is.finite(r)]
if (length(ok) == 0) 0 else round(max(ok), 2)
}, numeric(1)),
stringsAsFactors = FALSE
)
feature_deviations_df <- feature_deviations_df[
order(-feature_deviations_df$max_robust_z), , drop = FALSE]
rownames(feature_deviations_df) <- NULLStep 11: Most extreme row + narrative anchors
ex_i <- ord[1]
ex_j <- dom_idx[ex_i]
extreme <- list(
row_id = paste0("Row ", ex_i),
score = round(scores[ex_i], 2),
feature = if (is.na(ex_j)) "(not determinable)" else hn[ex_j],
value = if (is.na(ex_j)) NA_real_ else round(X[ex_i, ex_j], 3),
typical = if (is.na(ex_j)) NA_real_ else round(col_medians[ex_j], 3),
direction = if (is.na(ex_j)) "" else
if (X[ex_i, ex_j] >= col_medians[ex_j]) "above" else "below",
robust_z = if (is.na(ex_j)) NA_real_ else round(Z[ex_i, ex_j], 2)
)
metrics <- list(
`Observations` = final_rows,
`Columns Scanned` = k,
`Anomalies Flagged` = as.integer(n_anom),
`Anomaly Rate %` = round(100 * n_anom / final_rows, 2),
`Score Threshold` = round(threshold, 2),
`Max Anomaly Score` = round(max(scores), 2),
`Most Extreme Row` = extreme$row_id
)
json_output <- list(
answer = paste0(
"Multivariate anomaly scan of ", format(final_rows, big.mark = ","),
" rows across ", k, " columns: ", n_anom, " row(s) flagged beyond the ",
"99.9% threshold(score > ", round(threshold, 2), "). The most extreme is ",
extreme$row_id, " (score ", extreme$score, "), driven by ",
extreme$feature,
if (!is.na(extreme$value)) paste0(" = ", extreme$value, " — ",
extreme$direction, " its typical value of ", extreme$typical) else "",
". Scoring method: ",
if (score_method == "max_robust_z") "per-row max robust z(covariance singular)"
else "squared Mahalanobis distance", "."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "score_distribution",
"top_anomalies", "anomaly_map", "feature_deviations"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows,
rows_removed = rows_removed,
feature_names = feature_names, used_features = used_features,
dropped_features = dropped_features,
scores = scores, score_method = score_method,
threshold = threshold, flag = flag,
n_anom = n_anom, n_top1 = n_top1, top1_cutoff = top1_cutoff,
dom_idx = dom_idx, Z = Z,
top_anomalies_df = top_anomalies_df,
score_distribution_df = score_distribution_df,
anomaly_map_df = anomaly_map_df, pc_var = pc_var,
feature_deviations_df = feature_deviations_df,
extreme = extreme,
metrics = metrics, json_output = json_output
)
}Claims about ANOMALIES count flagged (above-threshold) rows only — the table also shows below-threshold rows, which are merely "most unusual".
flagged <- tdf[tdf$anomaly_score > shared$threshold, , drop = FALSE]
anomaly_note <- if (nrow(flagged) > 0) {
ftab <- sort(table(flagged$dominant_feature), decreasing = TRUE)
paste0(" Among the ", nrow(flagged), " flagged ",
if (nrow(flagged) == 1) "anomaly" else "anomalies",
", the most common dominant feature is ", names(ftab)[1],
" (", as.integer(ftab[1]), " of ", nrow(flagged), ").")
} else ""
text <- paste0(
"The ", nrow(tdf), " most anomalous rows, ranked by score. ",
n_flagged_shown, " of them exceed the 99.9% threshold(",
round(shared$threshold, 2), "). Each row is explained by its dominant ",
"feature — the column furthest from typical in robust terms — with its ",
"actual value against the column median. ",
names(dom_tab)[1], " appears as the dominant feature in ",
as.integer(dom_tab[1]), " of the ", nrow(tdf), " most unusual rows.",
anomaly_note
)
list(
title = "Top Anomalies",
description = "The most anomalous rows with the feature driving each one.",
text = text,
data = list(top_anomalies = tdf)
)
}
# Card: anomaly_map (scatter)
card_anomaly_map <- function(shared, df, params) {
amap <- shared$anomaly_map_df
n_anom_shown <- sum(amap$status == "anomaly")
anom_pts <- amap[amap$status == "anomaly", , drop = FALSE]
spread_note <- if (n_anom_shown >= 2) {
same_side_pc1 <- max(mean(anom_pts$pc1 > 0), mean(anom_pts$pc1 < 0))
if (same_side_pc1 >= 0.8)
"Most anomalies fall on the same side of the map — a systematic pattern rather than random noise."
else
"The anomalies scatter in different directions — they look like independent one-off outliers rather than one systematic issue."
} else if (n_anom_shown == 1) {
"The single flagged anomaly sits isolated from the main cloud."
} else {
"No rows crossed the threshold; the cloud below is the full(sampled) dataset."
}
text <- paste0(
"All rows projected onto the dataset's two main axes of variation ",
"(principal components), which together carry ",
round(shared$pc_var[1] + shared$pc_var[2], 1), "% of the variance. ",
"The main cloud is normal behavior; highlighted points are the ",
n_anom_shown, " flagged anomalies. ", spread_note,
" (Showing ", format(nrow(amap), big.mark = ","),
" rows; every flagged anomaly is included.)"
)
list(
title = "Anomaly Map",
description = "The dataset in 2D with anomalies highlighted.",
text = text,
chart_labels = list(
pc1 = paste0("PC1 — ", shared$pc_var[1], "% of variance"),
pc2 = paste0("PC2 — ", shared$pc_var[2], "% of variance")
),
data = list(anomaly_map = amap)
)
}
# Card: feature_deviations (horizontal_bar)
card_feature_deviations <- function(shared, df, params) {
fdf <- shared$feature_deviations_df
basis <- if (shared$n_anom > 0) {
paste0("the ", shared$n_anom, " flagged anomalous row(s)")
} else {
paste0("the top 1% of rows by score(nothing crossed the threshold)")
}
quiet <- fdf$feature[fdf$max_robust_z < 3]
text <- paste0(
"For each column, the largest robust z-score observed across ", basis,
" — how many robust standard deviations the worst value sits from that ",
"column's median. ", fdf$feature[1], " drives the most extreme ",
"deviations(max |z| = ", fdf$max_robust_z[1], ")",
if (nrow(fdf) > 1) paste0(", followed by ", fdf$feature[2],
" (", fdf$max_robust_z[2], ")") else "", ". ",
if (length(quiet) > 0)
paste0(paste(quiet, collapse = ", "),
ifelse(length(quiet) == 1, " stays", " stay"),
" below 3 robust sigmas even among the anomalies — ",
"the trouble is concentrated elsewhere.")
else
"Every scanned column contributes extreme values among the anomalies."
)
list(
title = "Which Columns Drive Anomalies",
description = "Per-column maximum robust deviation across the anomalous rows.",
text = text,
data = list(feature_deviations = fdf)
)
}