Brief

A B2B software company is deciding whether to build a requested feature. Product wants evidence of demand across the account base before committing a quarter of engineering.

What share of our accounts actually want this, and how confident can we be?

Two survey routes are available. Customer Success can reach the 150 accounts that have a named CSM — fast, warm, high response. Or the company can draw a random sample of accounts and chase them cold, which is slower and yields far fewer replies.

The instinct is that 150 answers beat 30. This review shows that instinct is wrong in a way no amount of extra sample can repair, and it can prove it, because the ground truth is known: this is a simulated account base where every account’s true preference is recorded.

The data

One row per account. The full population is on file, which is what makes this a teaching dataset — in a real survey you never get to see the answer you are estimating.

column meaning unit
account_id account reference id
segment enterprise / mid_market / smb factor
seats licensed seats on the account count
has_csm account has a named Customer Success Manager 0/1
wants_feature the account’s true position on the feature 0/1

Generation is deterministic (set.seed(20260824)); re-knitting reproduces data.csv byte-for-byte. Every figure below is computed from the written-out file.

set.seed(20260824)

segment <- c(rep("enterprise", 100), rep("mid_market", 140), rep("smb", 160))
want_n  <- c(enterprise = 92, mid_market = 104, smb = 83)   # true supporters per segment
csm_n   <- c(enterprise = 90, mid_market = 56,  smb = 4)    # who has a CSM

wants_feature <- has_csm <- integer(400)
for (s in names(want_n)) {
  idx <- which(segment == s)
  wants_feature[sample(idx, want_n[s])] <- 1L
  has_csm[sample(idx, csm_n[s])]        <- 1L
}

set.seed(1234)
seats <- ifelse(segment == "enterprise", round(rnorm(400, 420, 90)),
         ifelse(segment == "mid_market", round(rnorm(400, 95, 25)),
                round(rnorm(400, 18, 6))))
seats <- pmax(seats, 3)

pop <- data.frame(
  account_id    = sprintf("ACC-%04d", seq_len(400)),
  segment       = segment,
  seats         = seats,
  has_csm       = has_csm,
  wants_feature = wants_feature
)
write.csv(pop, "data.csv", row.names = FALSE)

TRUTH <- mean(pop$wants_feature)
sprintf("ground truth: %d of 400 accounts want it = %.4f%%", sum(pop$wants_feature), TRUTH * 100)
## [1] "ground truth: 279 of 400 accounts want it = 69.7500%"
by_seg <- do.call(rbind, lapply(c("enterprise", "mid_market", "smb"), function(s) {
  v <- pop[pop$segment == s, ]
  data.frame(segment = s, accounts = nrow(v),
             wants = sum(v$wants_feature),
             want_rate = round(100 * mean(v$wants_feature), 2),
             with_csm = sum(v$has_csm),
             csm_coverage = round(100 * mean(v$has_csm), 1))
}))
knitr::kable(by_seg, caption = "The account base has structure — and the CSM frame tracks it")
The account base has structure — and the CSM frame tracks it
segment accounts wants want_rate with_csm csm_coverage
enterprise 100 92 92.00 90 90.0
mid_market 140 104 74.29 56 40.0
smb 160 83 51.88 4 2.5

This table is the whole problem in advance. Want-rate falls from 92.0% in enterprise to 51.9% in SMB, and CSM coverage falls the same way, from 90% to 2%. Any survey routed through CSMs is a survey of enterprise opinion wearing the account base’s name.

Route one: the fair sample

Thirty accounts drawn at random from all 400.

set.seed(6)
fair_idx <- sample(400, 30)
k_fair   <- sum(pop$wants_feature[fair_idx])
sprintf("%d of 30 = %.2f%%", k_fair, 100 * k_fair / 30)
## [1] "21 of 30 = 70.00%"

A single proportion is never the answer — the answer is an interval. At n = 30 the choice of interval is a real decision, and the three standard options do not agree.

wilson <- function(k, n, conf = 0.95) {
  z <- qnorm(1 - (1 - conf) / 2); p <- k / n; d <- 1 + z^2 / n
  c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d,
    (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d)
}
wald <- function(k, n, conf = 0.95) {
  z <- qnorm(1 - (1 - conf) / 2); p <- k / n
  c(p - z * sqrt(p * (1 - p) / n), p + z * sqrt(p * (1 - p) / n))
}
cp <- binom.test(k_fair, 30)$conf.int

ints <- data.frame(
  interval = c("Wald (textbook)", "Wilson", "Clopper-Pearson (exact)"),
  lower = round(100 * c(wald(k_fair, 30)[1], wilson(k_fair, 30)[1], cp[1]), 2),
  upper = round(100 * c(wald(k_fair, 30)[2], wilson(k_fair, 30)[2], cp[2]), 2)
)
ints$width <- round(ints$upper - ints$lower, 2)
ints$covers_truth <- ints$lower <= TRUTH * 100 & TRUTH * 100 <= ints$upper
knitr::kable(ints, caption = sprintf("Three 95%% intervals on the same %d of 30", k_fair))
Three 95% intervals on the same 21 of 30
interval lower upper width covers_truth
Wald (textbook) 53.60 86.40 32.80 TRUE
Wilson 52.12 83.34 31.22 TRUE
Clopper-Pearson (exact) 50.60 85.27 34.67 TRUE

All three cover the truth. They differ in shape and in behaviour: Wald is the widest here and the worst behaved near 0 and 1 (it can run past 100%), Clopper-Pearson buys guaranteed coverage and pays in width, and Wilson sits between them and behaves well at small counts — which is why it is the sensible default and the one the platform tool reports.

op <- par(mar = c(4, 10, 3, 2))
plot(NA, xlim = c(40, 100), ylim = c(0.5, 3.5), yaxt = "n", ylab = "",
     xlab = "share of accounts wanting the feature (%)",
     main = sprintf("Fair sample: %d of 30", k_fair))
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
text(TRUTH * 100, 3.45, " truth", col = "#3fbf6f", pos = 4, cex = 0.85)
for (i in 1:3) {
  segments(ints$lower[i], i, ints$upper[i], i, lwd = 8, col = "#5fa9dd", lend = 1)
  points(100 * k_fair / 30, i, pch = 19, cex = 1.3)
}
axis(2, 1:3, ints$interval, las = 2, cex.axis = 0.85)

par(op)

Route two: the convenient sample

Every account with a CSM — five times the sample, no chasing.

conv    <- pop[pop$has_csm == 1, ]
k_conv  <- sum(conv$wants_feature)
n_conv  <- nrow(conv)
ci_conv <- wilson(k_conv, n_conv)
sprintf("%d of %d = %.2f%%  ·  Wilson %.2f%% to %.2f%%",
        k_conv, n_conv, 100 * k_conv / n_conv, 100 * ci_conv[1], 100 * ci_conv[2])
## [1] "129 of 150 = 86.00%  ·  Wilson 79.54% to 90.66%"
op <- par(mar = c(4, 11, 3, 2))
plot(NA, xlim = c(40, 100), ylim = c(0.5, 2.5), yaxt = "n", ylab = "",
     xlab = "share of accounts wanting the feature (%)",
     main = "A fair 30 against a convenient 150")
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
text(TRUTH * 100, 2.44, " truth 69.75%", col = "#3fbf6f", pos = 4, cex = 0.85)
ci_fair <- wilson(k_fair, 30)
segments(100 * ci_fair[1], 2, 100 * ci_fair[2], 2, lwd = 10, col = "#3fbf6f", lend = 1)
points(100 * k_fair / 30, 2, pch = 19, cex = 1.4)
segments(100 * ci_conv[1], 1, 100 * ci_conv[2], 1, lwd = 10, col = "#c0392b", lend = 1)
points(100 * k_conv / n_conv, 1, pch = 19, cex = 1.4)
axis(2, 1:2, c(sprintf("convenient n = %d", n_conv), "fair n = 30"), las = 2, cex.axis = 0.85)

par(op)

The convenient sample is narrower and wrong. Its interval (79.5% to 90.7%) does not contain the truth at all. More data made the answer more precise and no more correct — precision is a statement about the sample, not about the population it was supposed to represent.

It is not one unlucky draw

A single comparison proves nothing; either sample could have got lucky. Repeat both routes 10,000 times and count how often each interval actually contains the truth.

set.seed(2026)
fair_cov <- replicate(10000, {
  k <- sum(pop$wants_feature[sample(400, 30)])
  ci <- wilson(k, 30); ci[1] <= TRUTH && TRUTH <= ci[2]
})
conv_idx <- which(pop$has_csm == 1)
conv_cov <- replicate(10000, {
  k <- sum(pop$wants_feature[sample(conv_idx, 150, replace = TRUE)])
  ci <- wilson(k, 150); ci[1] <= TRUTH && TRUTH <= ci[2]
})
knitr::kable(data.frame(
  route = c("fair random sample, n = 30", "convenient CSM sample, n = 150"),
  advertised_coverage = c("95%", "95%"),
  actual_coverage = sprintf("%.2f%%", 100 * c(mean(fair_cov), mean(conv_cov)))
), caption = "How often the 95% interval actually contains the truth, over 10,000 repeats")
How often the 95% interval actually contains the truth, over 10,000 repeats
route advertised_coverage actual_coverage
fair random sample, n = 30 95% 95.68%
convenient CSM sample, n = 150 95% 0.16%

95.7% against 0.16%. The fair sample delivers what a 95% interval promises. The convenient sample, with five times the data, is essentially never right — and it never tells you so.

Why more of a biased sample cannot help

Sampling error shrinks with n. Bias does not — it is a property of the frame, and the frame does not improve when you fill it up.

set.seed(99)
lad <- do.call(rbind, lapply(c(20, 50, 100, 200), function(nn) {
  k <- sum(pop$wants_feature[sample(400, nn)]); ci <- wilson(k, nn)
  data.frame(n = nn, wants = k, estimate = round(100 * k / nn, 2),
             lower = round(100 * ci[1], 2), upper = round(100 * ci[2], 2),
             width_points = round(100 * (ci[2] - ci[1]), 2),
             covers_truth = ci[1] <= TRUTH && TRUTH <= ci[2])
}))
knitr::kable(lad, caption = "Fair samples: the interval narrows as n grows (n = 400 is a census — no sampling error at all)")
Fair samples: the interval narrows as n grows (n = 400 is a census — no sampling error at all)
n wants estimate lower upper width_points covers_truth
20 13 65.0 43.29 81.88 38.60 TRUE
50 33 66.0 52.15 77.56 25.41 TRUE
100 71 71.0 61.46 78.99 17.52 TRUE
200 127 63.5 56.63 69.86 13.23 TRUE 
op <- par(mar = c(4, 6, 3, 2))
plot(NA, xlim = c(40, 95), ylim = c(0.5, nrow(lad) + 0.5), yaxt = "n", ylab = "",
     xlab = "share wanting the feature (%)", main = "Fair sampling: precision is bought with n")
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
for (i in seq_len(nrow(lad))) {
  y <- nrow(lad) - i + 1
  segments(lad$lower[i], y, lad$upper[i], y, lwd = 8, col = "#5fa9dd", lend = 1)
  points(lad$estimate[i], y, pch = 19)
  text(lad$upper[i] + 1, y, sprintf("%.1f pts", lad$width_points[i]), pos = 4, cex = 0.8)
}
axis(2, nrow(lad):1, sprintf("n = %d", lad$n), las = 2, cex.axis = 0.85)

par(op)

The width falls from 38.6 points at n = 20 to 13.2 at n = 200. Going from 50 to 200 — four times the survey effort — bought roughly half the width. Precision is expensive, and it is the only thing sample size buys.

The structure that made the shortcut fail

tab <- table(pop$segment, pop$wants_feature)[, c("1", "0")]
chi <- chisq.test(tab)
ent_smb <- prop.test(c(sum(pop$wants_feature[pop$segment == "enterprise"]),
                       sum(pop$wants_feature[pop$segment == "smb"])),
                     c(100, 160))
knitr::kable(data.frame(
  test = c("segments differ (chi-square, 3 segments)", "enterprise vs smb (two-proportion)"),
  statistic = c(sprintf("X2 = %.4f, df = %d", chi$statistic, chi$parameter),
                sprintf("diff = %.2f pts", 100 * (0.92 - 83 / 160))),
  p = c(signif(chi$p.value, 4), signif(ent_smb$p.value, 4))
), caption = "The account base is not homogeneous, which is exactly why the frame mattered")
The account base is not homogeneous, which is exactly why the frame mattered
test statistic p
segments differ (chi-square, 3 segments) X2 = 49.0578, df = 2 0
enterprise vs smb (two-proportion) diff = 40.12 pts 0

If every segment wanted the feature at the same rate, the CSM shortcut would have been harmless. The segments differ at p = 2.22^{-11}, so a frame that over-samples enterprise must overstate demand.

What the data has to look like

standard_group_comparison takes one row per account with column_mapping {group: segment, outcome: wants_feature} — the group label and a numeric outcome, where a 0/1 answer is a perfectly good numeric outcome. Minimum 10 rows.

Two requirements decide whether this analysis is possible:

Decision

  1. Do not survey through the CSM list. It reports 86% against a true 69.8%, and its interval excludes the truth outright.
  2. A fair 30 beats a convenient 150. Report the interval, not the point: 70% with a 95% interval of 52% to 83%. It is a wide, honest answer, and wide is the correct feeling at n = 30.
  3. Use Wilson intervals at these counts, not the textbook Wald formula.
  4. If the decision needs a tighter answer, buy it with a bigger fair sample — n = 200 halves the width against n = 50 — but never with a bigger convenient one. Before trusting any survey, ask the one question that catches more bad answers than any formula: who could never have been in this sample?