Standard Roc Analysis
Executive Summary

Executive Summary

How well Model Score separates 'Yes' from 'No'

Observations
600
AUC
0.874
AUC Quality
good
Youden Cutoff
0.6001
Sensitivity
0.85
Specificity
0.732
Across 600 rows, Model Score ranks 'Yes' cases of Disease above 'No' cases with an AUC of 0.874 (95% CI 0.843 to 0.905) — good discrimination. The positive class is 'Yes' (it matched a conventional positive label). The Youden-optimal cutoff is 0.6001 on Model Score: flagging rows at or above it catches 85.0% of true 'Yes' cases (sensitivity) while correctly clearing 73.2% of 'No' cases (specificity).
Interpretation

Across 600 observations, Model Score achieves an AUC of 0.874 (95% CI 0.843 to 0.905)—good discrimination. At the Youden-optimal cutoff of 0.6001, the model catches 85.0% of true 'Yes' cases (sensitivity) while correctly clearing 73.2% of 'No' cases (specificity), delivering 77.8% overall accuracy. Of rows flagged positive, 67.0% truly are 'Yes' (PPV); of rows cleared, 88.4% truly are 'No' (NPV). This represents a well-balanced operating point for practical deployment.

Overview

Analysis Overview

ROC evaluation of Model Score against Disease across 600 observations.

N Observations600
Auc0.874
Positive Rate0.39
Youden Cutoff0.6001
Interpretation

Your model achieves an AUC of 0.874, indicating good discrimination between the two classes. AUC represents the probability that a randomly selected 'Yes' case scores higher than a randomly selected 'No' case—a threshold-independent measure of ranking quality. Since 0.5 represents pure chance and 1.0 represents perfect separation, your score of 0.874 sits solidly in the good range. This metric isolates how well Model Score orders the two classes, independent of any specific decision rule or cutoff choice.

Data Preparation

Data Quality

Positive-class definition, dropped rows, and class balance.

Initial Rows600
Final Rows600
Rows Removed0
Positive Rate0.39
Interpretation

All 600 rows loaded with complete outcome and score data; no rows were removed. The dataset contains 234 positive rows (39.0%) and 366 negative rows (61.0%), with 'Yes' treated as the positive class. Model Score was read as a continuous value where higher scores indicate greater likelihood of the positive class. The class balance is reasonably healthy, with the positive class representing about two-fifths of the sample, reducing the risk of severe imbalance artifacts in performance estimation.

Visualization

ROC Curve

True positive rate vs false positive rate across every cutoff on Model Score.

Interpretation

The ROC curve bows substantially above the chance diagonal, reflecting the model's strong ranking ability. The Youden-optimal operating point sits at a false positive rate of 0.268 and true positive rate of 0.850 (cutoff 0.6001). The curve's pronounced upward bow toward the top-left corner, combined with the AUC of 0.874, demonstrates that Model Score separates the classes far better than random guessing. The curve's trajectory shows early gains in sensitivity at low false positive rates, indicating the score has useful discriminatory power across the operating range.

Data Table

Cutoff Performance

AUC, confidence interval, and every operating-point metric at the Youden cutoff.

MetricValueInterpretation
AUC0.874Probability a random 'Yes' row scores above a random 'No' row — good discrimination.
AUC 95% CI0.843 to 0.905Hanley-McNeil 95% confidence interval for the AUC.
Youden cutoff on Model Score0.6001Flag a row positive when Model Score is at or above this value (maximizes sensitivity + specificity).
Sensitivity85.0%Share of actual 'Yes' rows caught at this cutoff.
Specificity73.2%Share of actual 'No' rows correctly cleared.
PPV (precision)67.0%Of rows flagged positive, the share that truly are 'Yes'.
NPV88.4%Of rows cleared, the share that truly are 'No'.
Accuracy77.8%Overall share of rows classified correctly at this cutoff.
Interpretation

At the Youden cutoff of 0.6001, the model delivers 85.0% sensitivity and 73.2% specificity, with 77.8% overall accuracy across 600 rows. The confusion matrix shows 199 true positives, 98 false positives, 268 true negatives, and 35 false negatives. PPV is 67.0% and NPV is 88.4%, indicating strong negative predictive value—rows cleared as 'No' are correct 88.4% of the time. The AUC 95% confidence interval (0.843 to 0.905) is reasonably tight, suggesting stable discrimination across resampling.

Data Table

Threshold Sweep

Sensitivity, specificity, and Youden's J across candidate cutoffs.

ThresholdSensitivitySpecificityYouden J
0.67450.8250.7460.571
0.66880.8250.7430.568
0.6670.8250.740.565
0.66010.8290.740.569
0.64960.8290.7380.567
0.63960.8380.7350.573
0.63180.8420.7350.577
0.61490.8460.7350.581
0.60880.8460.7320.578
0.60010.850.7320.583
0.5820.850.730.58
0.57840.850.7270.577
0.5580.8550.7270.581
0.55370.8550.7240.579
0.5320.8550.7190.573
0.53130.8550.7160.571
0.53080.8550.7130.568
0.52810.8550.710.565
0.51210.8550.7080.562
Interpretation

The threshold sweep across 19 candidate cutoffs shows a clear trade-off between sensitivity and specificity. Youden's J peaks at 0.583 at the cutoff of 0.6001 (sensitivity 0.85, specificity 0.732). Lower thresholds (e.g., 0.5121) raise sensitivity to 0.855 but reduce specificity to 0.708; higher thresholds (e.g., 0.6745) hold sensitivity at 0.825 while lifting specificity to 0.746. The flat plateau around the optimal cutoff suggests modest flexibility—small threshold adjustments incur minimal performance loss, easing practical implementation.

Methodology

Methodology

Statistical methodology and diagnostics for ROC Curve & Classifier Cutoff

Statistical Method

ROC Curve & Classifier Cutoff

Standard-library analysis: evaluate a binary classifier or diagnostic score against the true yes/no outcome. Draws the full ROC curve, computes the AUC with a 95% confidence interval, and finds the best operating cutoff by Youden's J — with sensitivity, specificity, PPV, NPV, accuracy, and the confusion counts at that cutoff. Works on any dataset: map the true outcome and a numeric score where higher means more likely positive.

Data
N = 600 observations
Assumptions
  • Higher score means more likely to be in the positive class
  • The true outcome is genuinely binary (two classes)
Limitations
  • AUC measures ranking, not calibration — a well-ranked score can still be poorly calibrated
  • The best cutoff depends on the relative cost of false positives versus false negatives; Youden's J weights them equally
  • With few positives or negatives, the AUC confidence interval is wide
Software & Citation
MCP Analytics · mcpanalytics.ai
Code Appendix

Analysis Code

Complete R source code for this analysis

ROC Curve & Classifier Cutoff — How Good Is Your Score

Evaluates a binary classifier or diagnostic score against the true yes/no outcome. Treats one class as positive, sweeps every candidate cutoff on the score to trace the ROC curve, computes the AUC (with a Hanley-McNeil 95% confidence interval, cross-checked against the curve's own trapezoidal area), and recommends the operating cutoff that maximizes Youden's J — reporting sensitivity, specificity, PPV, NPV, accuracy, and the confusion counts at that cutoff.

Why This Method?

AUC is the single most honest summary of a score's ranking quality: the probability that a random positive scores above a random negative, independent of any threshold. The ROC curve shows the full sensitivity/specificity trade-off, and Youden's J turns it into a concrete decision cutoff.

What This Analysis Covers

  • The ROC curve (true positive rate vs false positive rate)
  • AUC with a 95% confidence interval and a quality band
  • The Youden-optimal cutoff with sensitivity, specificity, PPV, NPV, accuracy
  • A threshold sweep around the operating range

Standard Library

Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {actual, score}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Step 1: Row accounting + semantic column discovery

initial_rows <- nrow(df)
  if (!"actual" %in% names(df)) {
    stop("column_mapping must map an &#x27;actual' column (the true yes/no outcome).")
  }
  if (!"score" %in% names(df)) {
    stop("column_mapping must map a &#x27;score' column (the numeric model score).")
  }
  actual_name <- humanize_semantic("actual", col_map)
  score_name  <- humanize_semantic("score", col_map)

Step 2: Choose the positive class + binarize the outcome

v_raw <- df$actual
  if (is.logical(v_raw)) {
    keep_a <- !is.na(v_raw)
    df <- df[keep_a, , drop = FALSE]
    y <- as.integer(df$actual)
    positive_label <- "TRUE"
    negative_label <- "FALSE"
    positive_rule  <- "the boolean TRUE value"
  } else {
    vc <- trimws(as.character(v_raw))
    keep_a <- !is.na(v_raw) & !is.na(vc) & vc != ""
    df <- df[keep_a, , drop = FALSE]
    vc <- vc[keep_a]
    lv <- sort(unique(vc))
    if (length(lv) < 2) {
      stop(sprintf(
        "The outcome column(&#x27;%s') has only one value ('%s') — an ROC analysis needs both the positive and the negative class present.",
        actual_name, if (length(lv) == 1) lv[1] else "empty"))
    }

Positive class = the level matching a conventional positive label (case-insensitive); otherwise the alphabetically-last level.

pat <- "^(1|yes|true|positive|pos|disease|churn|churned|fraud|default|defaulted|converted)$"
    hits <- lv[grepl(pat, tolower(lv))]
    if (length(hits) >= 1) {
      positive_label <- hits[length(hits)]
      positive_rule  <- "it matched a conventional positive label"
    } else {
      positive_label <- lv[length(lv)]
      positive_rule  <- "no conventional positive label was found, so the alphabetically-last class was used"
    }
    neg_levels <- setdiff(lv, positive_label)
    negative_label <- if (length(neg_levels) == 1) neg_levels[1]
                      else paste0("not ", positive_label)
    if (length(lv) > 2) {
      positive_rule <- paste0(positive_rule,
        "; the outcome had more than two values, so this is a one-vs-rest evaluation")
    }
    y <- as.integer(vc == positive_label)
  }

Step 3: Coerce the score to numeric; drop NA (actual, score) pairs

sc <- df$score
  if (!is.numeric(sc)) sc <- suppressWarnings(as.numeric(as.character(sc)))
  keep_s <- !is.na(sc) & !is.na(y)
  sc <- sc[keep_s]
  y  <- y[keep_s]

  final_rows <- length(y)
  rows_removed <- initial_rows - final_rows
  if (final_rows < 20) {
    stop(sprintf(
      "Only %d rows have both a usable outcome(&#x27;%s') and a numeric score ('%s') — at least 20 are required.",
      final_rows, actual_name, score_name))
  }

Step 4: Guard — both classes present with at least 10 each

n1 <- sum(y == 1L)
  n0 <- sum(y == 0L)
  if (n1 < 10 || n0 < 10) {
    stop(sprintf(
      "An ROC analysis needs at least 10 of each class, but &#x27;%s' has %d '%s' and %d '%s' rows.",
      actual_name, n1, positive_label, n0, negative_label))
  }
  if (isTRUE(var(sc) == 0) || is.na(var(sc))) {
    stop(sprintf(
      "The score column(&#x27;%s') is constant — a single value cannot separate the two classes.",
      score_name))
  }

Step 5: Sweep thresholds (O(n log n)) — predict positive if score >= t

o <- order(sc, decreasing = TRUE)
  s_o <- sc[o]
  y_o <- y[o]
  cum_tp <- cumsum(y_o == 1L)
  cum_fp <- cumsum(y_o == 0L)

Keep the LAST index of each run of equal scores (descending sort): that index's cumulative counts are the counts for score >= that value.

run_end <- which(c(diff(s_o) != 0, TRUE))
  thr  <- s_o[run_end]
  tp_c <- cum_tp[run_end]
  fp_c <- cum_fp[run_end]
  tpr  <- tp_c / n1
  fpr  <- fp_c / n0
  sens_v <- tpr
  spec_v <- 1 - fpr
  j_v    <- sens_v + spec_v - 1

Step 6: AUC — rank formula (reported) + trapezoidal cross-check

Mann-Whitney rank AUC (ties handled by average ranks).

r <- rank(sc)
  auc <- (sum(r[y == 1L]) - as.numeric(n1) * (n1 + 1) / 2) /
         (as.numeric(n1) * as.numeric(n0))

Trapezoidal area under the ROC points (prepend the (0,0) origin; the sweep already ends at (1,1) via the lowest threshold).

roc_fpr <- c(0, fpr)
  roc_tpr <- c(0, tpr)
  auc_trap <- sum(diff(roc_fpr) * (head(roc_tpr, -1) + tail(roc_tpr, -1)) / 2)

Hanley-McNeil standard error + 95% CI (clamped to [0, 1]).

q1 <- auc / (2 - auc)
  q2 <- 2 * auc^2 / (1 + auc)
  se <- sqrt((auc * (1 - auc) +
              (n1 - 1) * (q1 - auc^2) +
              (n0 - 1) * (q2 - auc^2)) /
             (as.numeric(n1) * as.numeric(n0)))
  ci_low  <- max(0, auc - 1.96 * se)
  ci_high <- min(1, auc + 1.96 * se)
  band <- auc_band(auc)

Step 7: Operating cutoffs — Youden's J and closest-to-top-left

NA-filter before which.max/which.min (LAT-1445 crash class).

j_ok <- j_v
  j_ok[is.na(j_ok)] <- -Inf
  j_idx <- which.max(j_ok)
  youden_cutoff <- thr[j_idx]
  max_j <- j_v[j_idx]

  dist_tl <- sqrt(fpr^2 + (1 - tpr)^2)
  dist_tl[is.na(dist_tl)] <- Inf
  tl_idx <- which.min(dist_tl)
  topleft_cutoff <- thr[tl_idx]

Step 8: Confusion + operating metrics at the Youden cutoff

pred <- as.integer(sc >= youden_cutoff)
  tp <- sum(pred == 1L & y == 1L)
  fp <- sum(pred == 1L & y == 0L)
  tn <- sum(pred == 0L & y == 0L)
  fn <- sum(pred == 0L & y == 1L)
  sensitivity <- if ((tp + fn) > 0) tp / (tp + fn) else NA_real_
  specificity <- if ((tn + fp) > 0) tn / (tn + fp) else NA_real_
  ppv <- if ((tp + fp) > 0) tp / (tp + fp) else NA_real_
  npv <- if ((tn + fn) > 0) tn / (tn + fn) else NA_real_
  accuracy <- (tp + tn) / final_rows

Step 9: ROC points dataset (sorted; sampled to <= 1000 for the chart)

roc_points_df <- data.frame(
    false_positive_rate = round(roc_fpr, 4),
    true_positive_rate  = round(roc_tpr, 4),
    stringsAsFactors = FALSE
  )
  roc_points_df <- roc_points_df[order(roc_points_df$false_positive_rate,
                                       roc_points_df$true_positive_rate), ]
  rownames(roc_points_df) <- NULL
  if (nrow(roc_points_df) > 1000) {
    set.seed(42)
    idx <- sort(unique(c(1L,
                         as.integer(round(seq(1, nrow(roc_points_df),
                                              length.out = 1000))),
                         nrow(roc_points_df))))
    roc_points_df <- roc_points_df[idx, ]
    rownames(roc_points_df) <- NULL
  }

Step 10: Cutoff-metrics table (all values fixed-notation strings)

cutoff_metrics_df <- data.frame(
    metric = c("AUC", "AUC 95% CI", paste0("Youden cutoff on ", score_name),
               "Sensitivity", "Specificity", "PPV(precision)",
               "NPV", "Accuracy"),
    value = c(
      fnum(auc, 3),
      paste0(fnum(ci_low, 3), " to ", fnum(ci_high, 3)),
      fnum(youden_cutoff, 4),
      fpct(sensitivity), fpct(specificity), fpct(ppv), fpct(npv), fpct(accuracy)
    ),
    interpretation = c(
      paste0("Probability a random &#x27;", positive_label,
             "&#x27; row scores above a random '", negative_label, "' row — ",
             band, " discrimination."),
      "Hanley-McNeil 95% confidence interval for the AUC.",
      paste0("Flag a row positive when ", score_name,
             " is at or above this value(maximizes sensitivity + specificity)."),
      paste0("Share of actual &#x27;", positive_label, "' rows caught at this cutoff."),
      paste0("Share of actual &#x27;", negative_label, "' rows correctly cleared."),
      paste0("Of rows flagged positive, the share that truly are &#x27;",
             positive_label, "&#x27;."),
      paste0("Of rows cleared, the share that truly are &#x27;", negative_label, "'."),
      "Overall share of rows classified correctly at this cutoff."
    ),
    stringsAsFactors = FALSE
  )

Step 11: Threshold sweep table (<= 20 rows around the operating range)

sweep_full <- data.frame(
    threshold   = round(thr, 4),
    sensitivity = round(sens_v, 3),
    specificity = round(spec_v, 3),
    youden_j    = round(j_v, 3),
    stringsAsFactors = FALSE
  )
  n_sw <- nrow(sweep_full)
  if (n_sw <= 20) {
    sel <- seq_len(n_sw)
  } else {
    lo <- max(1L, j_idx - 10L)
    hi <- min(n_sw, j_idx + 10L)
    win <- lo:hi
    if (length(win) <= 20) {
      sel <- win
    } else {

19 evenly-spaced cutoffs across the window + the Youden row itself, guaranteeing the operating point is shown and total stays <= 20.

sel <- sort(unique(c(j_idx,
                           win[as.integer(round(seq(1, length(win),
                                                    length.out = 19L)))])))
    }
  }
  threshold_sweep_df <- sweep_full[sel, , drop = FALSE]
  rownames(threshold_sweep_df) <- NULL

Step 12: KPI metrics + machine channels

metrics <- list(
    `Observations`  = final_rows,
    `AUC`           = round(auc, 3),
    `AUC Quality`   = band,
    `Youden Cutoff` = round(youden_cutoff, 4),
    `Sensitivity`   = round(sensitivity, 3),
    `Specificity`   = round(specificity, 3)
  )

  roc_summary <- list(
    auc = auc, auc_trap = auc_trap, ci_low = ci_low, ci_high = ci_high,
    band = band, youden_cutoff = youden_cutoff, topleft_cutoff = topleft_cutoff,
    sensitivity = sensitivity, specificity = specificity, ppv = ppv, npv = npv,
    accuracy = accuracy, tp = tp, fp = fp, tn = tn, fn = fn,
    n1 = n1, n0 = n0, max_j = max_j,
    positive_label = positive_label, negative_label = negative_label
  )

  json_output <- list(
    answer = paste0(
      "ROC analysis of ", score_name, " against ", actual_name,
      " (positive = &#x27;", positive_label, "', ",
      fpct(n1 / final_rows), " of ", format(final_rows, big.mark = ","),
      " rows): AUC = ", fnum(auc, 3), " (95% CI ", fnum(ci_low, 3), " to ",
      fnum(ci_high, 3), ", ", band, "). The Youden-optimal cutoff is ",
      fnum(youden_cutoff, 4), " on ", score_name, ", giving sensitivity ",
      fpct(sensitivity), " and specificity ", fpct(specificity), "."
    ),
    cards = lapply(
      c("tldr", "overview", "preprocessing", "roc_curve",
        "cutoff_metrics", "threshold_table"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
    actual_name = actual_name, score_name = score_name,
    positive_label = positive_label, negative_label = negative_label,
    positive_rule = positive_rule, n1 = n1, n0 = n0,
    auc = auc, auc_trap = auc_trap, ci_low = ci_low, ci_high = ci_high, band = band,
    youden_cutoff = youden_cutoff, topleft_cutoff = topleft_cutoff,
    sensitivity = sensitivity, specificity = specificity, ppv = ppv, npv = npv,
    accuracy = accuracy, tp = tp, fp = fp, tn = tn, fn = fn, max_j = max_j,
    roc_points_df = roc_points_df, cutoff_metrics_df = cutoff_metrics_df,
    threshold_sweep_df = threshold_sweep_df,
    roc_summary = roc_summary, metrics = metrics, json_output = json_output
  )
}

This is the exact R code executed to produce the analysis above. Run it with the same dataset to reproduce these results.

Your data has more stories to tell. Run any analysis on your own data — 60+ validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No Signup Sign Up Free
Something wrong? Report it and we'll fix it — or get your credits back
Download PDF · Print · Share

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing