. The data on this page is synthetic (simulated), made up to teach the method. It is not real customer data.

Brief

A B2B software company is deciding whether to build a requested feature. Product wants evidence of demand across the account base before committing a quarter of engineering.

What share of our accounts actually want this, and how confident can we be?

Two survey routes are available. Customer Success can reach the 150 accounts that have a named CSM: fast, warm, high response. Or the company can draw a random sample of accounts and chase them cold, which is slower and yields far fewer replies.

The instinct is that 150 answers beat 30. This review shows that instinct is wrong in a way no amount of extra sample can repair, and it can show it, because the ground truth is known: this is a simulated (synthetic) account base where every account’s true preference is recorded.

The data

One row per account. The full population is on file, which is what makes this a teaching dataset, in a real survey you never get to see the answer you are estimating.

column meaning unit
account_id account reference id
segment enterprise / mid_market / smb factor
seats licensed seats on the account count
has_csm account has a named Customer Success Manager 0/1
wants_feature the account’s true position on the feature 0/1

Generation is deterministic (set.seed(20260824)); re-knitting rebuilds the same 400 rows and rewrites sampling_responses.csv byte-for-byte. Every figure below is computed from those rows.

set.seed(20260824)

segment <- c(rep("enterprise", 100), rep("mid_market", 140), rep("smb", 160))
want_n  <- c(enterprise = 92, mid_market = 104, smb = 83)   # true supporters per segment
csm_n   <- c(enterprise = 90, mid_market = 56,  smb = 4)    # who has a CSM

wants_feature <- has_csm <- integer(400)
for (s in names(want_n)) {
  idx <- which(segment == s)
  wants_feature[sample(idx, want_n[s])] <- 1L
  has_csm[sample(idx, csm_n[s])]        <- 1L
}

set.seed(1234)
seats <- ifelse(segment == "enterprise", round(rnorm(400, 420, 90)),
         ifelse(segment == "mid_market", round(rnorm(400, 95, 25)),
                round(rnorm(400, 18, 6))))
seats <- pmax(seats, 3)

pop <- data.frame(
  account_id    = sprintf("ACC-%04d", seq_len(400)),
  segment       = segment,
  seats         = seats,
  has_csm       = has_csm,
  wants_feature = wants_feature
)
write.csv(pop, "sampling_responses.csv", row.names = FALSE)

TRUTH <- mean(pop$wants_feature)
sprintf("ground truth: %d of 400 accounts want it = %.4f%%", sum(pop$wants_feature), TRUTH * 100)
## [1] "ground truth: 279 of 400 accounts want it = 69.7500%"
by_seg <- do.call(rbind, lapply(c("enterprise", "mid_market", "smb"), function(s) {
  v <- pop[pop$segment == s, ]
  data.frame(segment = s, accounts = nrow(v),
             wants = sum(v$wants_feature),
             want_rate = round(100 * mean(v$wants_feature), 2),
             with_csm = sum(v$has_csm),
             csm_coverage = round(100 * mean(v$has_csm), 1))
}))
knitr::kable(by_seg, caption = "The account base has structure, and the CSM frame tracks it")
The account base has structure, and the CSM frame tracks it
segment accounts wants want_rate with_csm csm_coverage
enterprise 100 92 92.00 90 90.0
mid_market 140 104 74.29 56 40.0
smb 160 83 51.88 4 2.5

This table is the whole problem in advance. Want-rate falls from 92.0% in enterprise to 51.9% in SMB, and CSM coverage falls the same way, from 90% to 2.5%. Any survey routed through CSMs is a survey of enterprise opinion wearing the account base’s name.

Route one: the fair sample

Thirty accounts drawn at random from all 400.

set.seed(6)
fair_idx <- sample(400, 30)
k_fair   <- sum(pop$wants_feature[fair_idx])
sprintf("%d of 30 = %.2f%%", k_fair, 100 * k_fair / 30)
## [1] "21 of 30 = 70.00%"

A single proportion is never the answer. The answer is an interval. At n = 30 the choice of interval is a real decision, and the three standard options do not agree.

wilson <- function(k, n, conf = 0.95) {
  z <- qnorm(1 - (1 - conf) / 2); p <- k / n; d <- 1 + z^2 / n
  c((p + z^2 / (2 * n) - z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d,
    (p + z^2 / (2 * n) + z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2))) / d)
}
wald <- function(k, n, conf = 0.95) {
  z <- qnorm(1 - (1 - conf) / 2); p <- k / n
  c(p - z * sqrt(p * (1 - p) / n), p + z * sqrt(p * (1 - p) / n))
}
cp <- binom.test(k_fair, 30)$conf.int

ints <- data.frame(
  interval = c("Wald (textbook)", "Wilson", "Clopper-Pearson (exact)"),
  lower = round(100 * c(wald(k_fair, 30)[1], wilson(k_fair, 30)[1], cp[1]), 2),
  upper = round(100 * c(wald(k_fair, 30)[2], wilson(k_fair, 30)[2], cp[2]), 2)
)
ints$width <- round(ints$upper - ints$lower, 2)
ints$covers_truth <- ints$lower <= TRUTH * 100 & TRUTH * 100 <= ints$upper
knitr::kable(ints, caption = sprintf("Three 95%% intervals on the same %d of 30", k_fair))
Three 95% intervals on the same 21 of 30
interval lower upper width covers_truth
Wald (textbook) 53.60 86.40 32.80 TRUE
Wilson 52.12 83.34 31.22 TRUE
Clopper-Pearson (exact) 50.60 85.27 34.67 TRUE

All three cover the truth. They differ in shape and in behaviour: Wald is the worst behaved near 0 and 1 (it can run past 100%), Clopper-Pearson is built never to cover less than promised and pays for that here by being the widest, and Wilson is the narrowest here and behaves well at small counts, which is why it is a sensible default. This is one count (21 of 30), so do not read these widths as a rule.

op <- par(mar = c(4, 10, 3, 2))
plot(NA, xlim = c(40, 100), ylim = c(0.5, 3.5), yaxt = "n", ylab = "",
     xlab = "share of accounts wanting the feature (%)",
     main = sprintf("Fair sample: %d of 30", k_fair))
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
text(TRUTH * 100, 3.45, " truth", col = "#3fbf6f", pos = 4, cex = 0.85)
for (i in 1:3) {
  segments(ints$lower[i], i, ints$upper[i], i, lwd = 8, col = "#5fa9dd", lend = 1)
  points(100 * k_fair / 30, i, pch = 19, cex = 1.3)
}
axis(2, 1:3, ints$interval, las = 2, cex.axis = 0.85)

par(op)

Route two: the convenient sample

Every account with a CSM, five times the sample, no chasing.

conv    <- pop[pop$has_csm == 1, ]
k_conv  <- sum(conv$wants_feature)
n_conv  <- nrow(conv)
ci_conv <- wilson(k_conv, n_conv)
sprintf("%d of %d = %.2f%%  ·  Wilson %.2f%% to %.2f%%",
        k_conv, n_conv, 100 * k_conv / n_conv, 100 * ci_conv[1], 100 * ci_conv[2])
## [1] "129 of 150 = 86.00%  ·  Wilson 79.54% to 90.66%"
op <- par(mar = c(4, 11, 3, 2))
plot(NA, xlim = c(40, 100), ylim = c(0.5, 2.5), yaxt = "n", ylab = "",
     xlab = "share of accounts wanting the feature (%)",
     main = "A fair 30 against a convenient 150")
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
text(TRUTH * 100, 2.44, " truth 69.75%", col = "#3fbf6f", pos = 4, cex = 0.85)
ci_fair <- wilson(k_fair, 30)
segments(100 * ci_fair[1], 2, 100 * ci_fair[2], 2, lwd = 10, col = "#3fbf6f", lend = 1)
points(100 * k_fair / 30, 2, pch = 19, cex = 1.4)
segments(100 * ci_conv[1], 1, 100 * ci_conv[2], 1, lwd = 10, col = "#c0392b", lend = 1)
points(100 * k_conv / n_conv, 1, pch = 19, cex = 1.4)
axis(2, 1:2, c(sprintf("convenient n = %d", n_conv), "fair n = 30"), las = 2, cex.axis = 0.85)

par(op)

The convenient sample is narrower and wrong. Its interval (79.5% to 90.7%) does not contain the truth at all. More data made the answer more precise and no more correct, precision is a statement about the sample, not about the population it was supposed to represent.

It is not one unlucky draw

A single comparison shows little; either sample could have got lucky. Repeat both routes 10,000 times and count how often each interval actually contains the truth.

The convenient route needs one adjustment to be repeatable at all. As defined, it surveys every account with a CSM, so repeating it exactly returns the same 150 accounts and the same wrong interval every time: its coverage is exactly 0%. To give it every chance to get lucky, the simulation below runs a looser, hypothetical version of it: each repeat draws 150 responses at random, with replacement, from the CSM frame. That lets the answer wobble the way a sample does, while still drawing only from the convenient frame.

set.seed(2026)
fair_cov <- replicate(10000, {
  k <- sum(pop$wants_feature[sample(400, 30)])
  ci <- wilson(k, 30); ci[1] <= TRUTH && TRUTH <= ci[2]
})
conv_idx <- which(pop$has_csm == 1)
conv_cov <- replicate(10000, {
  k <- sum(pop$wants_feature[sample(conv_idx, 150, replace = TRUE)])
  ci <- wilson(k, 150); ci[1] <= TRUTH && TRUTH <= ci[2]
})
knitr::kable(data.frame(
  route = c("fair random sample, n = 30", "CSM frame, 150 random draws (looser variant)"),
  advertised_coverage = c("95%", "95%"),
  actual_coverage = sprintf("%.2f%%", 100 * c(mean(fair_cov), mean(conv_cov)))
), caption = "How often the 95% interval actually contains the truth, over 10,000 repeats")
How often the 95% interval actually contains the truth, over 10,000 repeats
route advertised_coverage actual_coverage
fair random sample, n = 30 95% 95.68%
CSM frame, 150 random draws (looser variant) 95% 0.16%

95.7% against 0.16%. The fair sample delivers what a 95% interval promises. Drawing from the convenient frame, even with five times the data and the randomness added back in, is almost never right, and it never tells you so. The route as actually defined does worse: its interval misses the truth every time.

Why more of a biased sample cannot help

Sampling error shrinks with n. Bias does not. It is a property of the frame, and the frame does not improve when you fill it up.

set.seed(99)
lad <- do.call(rbind, lapply(c(20, 50, 100, 200), function(nn) {
  k <- sum(pop$wants_feature[sample(400, nn)]); ci <- wilson(k, nn)
  data.frame(n = nn, wants = k, estimate = round(100 * k / nn, 2),
             lower = round(100 * ci[1], 2), upper = round(100 * ci[2], 2),
             width_points = round(100 * (ci[2] - ci[1]), 2),
             covers_truth = ci[1] <= TRUTH && TRUTH <= ci[2])
}))
knitr::kable(lad, caption = "Fair samples: the interval narrows as n grows (n = 400 is a census, no sampling error at all)")
Fair samples: the interval narrows as n grows (n = 400 is a census, no sampling error at all)
n wants estimate lower upper width_points covers_truth
20 13 65.0 43.29 81.88 38.60 TRUE
50 33 66.0 52.15 77.56 25.41 TRUE
100 71 71.0 61.46 78.99 17.52 TRUE
200 127 63.5 56.63 69.86 13.23 TRUE
op <- par(mar = c(4, 6, 3, 2))
plot(NA, xlim = c(40, 95), ylim = c(0.5, nrow(lad) + 0.5), yaxt = "n", ylab = "",
     xlab = "share wanting the feature (%)", main = "Fair sampling: precision is bought with n")
abline(v = TRUTH * 100, col = "#3fbf6f", lwd = 2, lty = 2)
for (i in seq_len(nrow(lad))) {
  y <- nrow(lad) - i + 1
  segments(lad$lower[i], y, lad$upper[i], y, lwd = 8, col = "#5fa9dd", lend = 1)
  points(lad$estimate[i], y, pch = 19)
  text(lad$upper[i] + 1, y, sprintf("%.1f pts", lad$width_points[i]), pos = 4, cex = 0.8)
}
axis(2, nrow(lad):1, sprintf("n = %d", lad$n), las = 2, cex.axis = 0.85)

par(op)

The width falls from 38.6 points at n = 20 to 13.2 at n = 200. Going from 50 to 200, four times the survey effort, bought roughly half the width. Precision is expensive, and it is the only thing sample size buys.

The structure that made the shortcut fail

tab <- table(pop$segment, pop$wants_feature)[, c("1", "0")]
chi <- chisq.test(tab)
ent_smb <- prop.test(c(sum(pop$wants_feature[pop$segment == "enterprise"]),
                       sum(pop$wants_feature[pop$segment == "smb"])),
                     c(100, 160))
knitr::kable(data.frame(
  test = c("segments differ (chi-square, 3 segments)", "enterprise vs smb (two-proportion)"),
  statistic = c(sprintf("X2 = %.4f, df = %d", chi$statistic, chi$parameter),
                sprintf("diff = %.2f pts", 100 * (0.92 - 83 / 160))),
  p = c(format(chi$p.value, digits = 4), format(ent_smb$p.value, digits = 4))
), caption = "The account base is not homogeneous, which is exactly why the frame mattered")
The account base is not homogeneous, which is exactly why the frame mattered
test statistic p
segments differ (chi-square, 3 segments) X2 = 49.0578, df = 2 2.224e-11
enterprise vs smb (two-proportion) diff = 40.12 pts 4.889e-11

If every segment wanted the feature at the same rate, the CSM shortcut would have been harmless. The segments differ (chi-square p = 2.22e-11), so a frame that over-samples enterprise must overstate demand.

What the data has to look like

standard_group_comparison takes one row per account with column_mapping {group: segment, outcome: wants_feature}: the group label and a numeric outcome, where a 0/1 answer is a perfectly good numeric outcome. Minimum 10 rows.

Two requirements decide whether this analysis is possible:

What this does not tell you

Try it on your own data. Take a survey export with one row per account, a column for who was asked, and a column for the answer, and compare the answer rate across your segments before you trust a headline number.

Decision

  1. Do not survey through the CSM list. It reports 86% against a true 69.8%, and its interval excludes the truth outright.
  2. A fair 30 beats a convenient 150. Report the interval, not the point: 70% with a 95% interval of 52% to 83%. It is a wide, honest answer, and wide is the correct feeling at n = 30.
  3. Use Wilson intervals at these counts, not the textbook Wald formula.
  4. If the decision needs a tighter answer, buy it with a bigger fair sample: n = 200 halves the width against n = 50, but never with a bigger convenient one. Before trusting any survey, ask the one question that catches more bad answers than any formula: who could never have been in this sample?

Sources

These two are cited only for who published the Wilson interval and the chi-square test, and when. They do not back any number on this page.

Your turn

Bring your own data and the question you actually need answered.

MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai