Which of glucose, BMI and age go with a diabetes flag, and how well can a simple model rank cases it has not seen? The way to learn what a method can find is to hand it data where you know the answer. This page does that. It builds 600 made-up patient records with three effects put in on purpose, fits a logistic regression, and shows what the fit recovered and what it missed.
The result is a lesson about odds ratios and about sample size. It is not a result about real patients.
Worked Example: Simulated Patient Records
Question. In made-up patient records, how do glucose, BMI and age link to a diabetes flag? How well does a simple model rank cases it did not learn from?
Inputs. The script makes up glucose, BMI, age and a yes-or-no result (seed 41, 600 rows). These are not real patient records. The script builds in an effect for each measure:
- Glucose: 0.035 log-odds per unit. With an SD of 20 that is 0.70 per SD, an odds ratio of about 2.0.
- BMI: 0.06 per unit. With an SD of 6 that is 0.36 per SD, an odds ratio of about 1.4.
- Age: 0.015 per unit. With an SD of 12 that is 0.18 per SD, an odds ratio of about 1.2.
Method. Fit logistic regression, a model for yes-or-no results, on a 420-row training set. Report odds ratios, which show how the odds change, for a one standard deviation step in each measure, with Wald intervals (a range of likely values). Score the 180 held-out rows, which the model did not learn from. Use a bootstrap, which re-scores resampled rows, to get an interval for the ROC AUC, a score of how well the model ranks cases.
Result. The run produced 35 events in 600 rows (5.8%). The glucose odds ratio was 2.269 per SD, with a 95% interval of 1.427 to 3.609. The BMI estimate was 1.001 [0.662, 1.515] and the age estimate was 1.292 [0.836, 1.996]. The BMI and age intervals include 1. The held-out AUC was 0.743, with a bootstrap 95% interval of 0.611 to 0.861.
The script built in an effect for BMI and for age. With only 35 events the run could not tell them from zero, which is low power, not no effect. The BMI estimate of 1.001 is also lower than the 1.4 that was built in, which is chance at this sample size.
Limit. The data and the result rule are made up, and the sample has only 35 events. The held-out set is 180 rows with few events, and the wide AUC interval says so. The run does not show what happens in real patients.
Complete script (run with python3; needs numpy, scipy and scikit-learn):
import numpy as np
from scipy.optimize import minimize
from scipy.special import expit
from sklearn.metrics import roc_auc_score
SEED = 41
rng = np.random.default_rng(SEED)
n = 600
x = np.column_stack([rng.normal(110, 20, n), rng.normal(30, 6, n),
rng.normal(45, 12, n)])
linear = (-3.0 + 0.035 * (x[:, 0] - 110) + 0.06 * (x[:, 1] - 30)
+ 0.015 * (x[:, 2] - 45))
y = rng.binomial(1, expit(linear))
order = rng.permutation(n)
split = int(0.7 * n)
train_idx, test_idx = order[:split], order[split:]
center = x[train_idx].mean(axis=0)
scale = x[train_idx].std(axis=0, ddof=0)
z = (x-center)/scale
X = np.column_stack([np.ones(n), z])
X_train, y_train = X[train_idx], y[train_idx]
X_test, y_test = X[test_idx], y[test_idx]
def nll(beta):
eta = X_train @ beta
return np.sum(np.logaddexp(0, eta) - y_train * eta)
fit = minimize(nll, np.zeros(X_train.shape[1]), method='BFGS')
beta = fit.x
p_train = expit(X_train @ beta)
info = X_train.T @ (X_train * (p_train * (1-p_train))[:, None])
cov = np.linalg.inv(info)
se = np.sqrt(np.diag(cov))
prob = expit(X_test @ beta)
auc = roc_auc_score(y_test, prob)
boot = np.random.default_rng(SEED+1)
boot_auc = []
for _ in range(2000):
take = boot.integers(0, len(y_test), len(y_test))
if np.unique(y_test[take]).size == 2:
boot_auc.append(roc_auc_score(y_test[take], prob[take]))
lo, hi = np.percentile(boot_auc, [2.5,97.5])
print(f'Simulated data, seed {SEED}')
print(f'Rows: {n}; events: {int(y.sum())}; non-events: {int(n-y.sum())}')
print(f'Split: train {len(train_idx)}; held-out test {len(test_idx)}')
print('Model: logistic regression; predictors glucose, BMI, age; '
'predictors centered and scaled on train data')
print('Term OR per SD 95% Wald CI')
for i,name in enumerate(['glucose','BMI','age'], start=1):
odds=np.exp(beta[i])
low,high=np.exp(beta[i]-1.96*se[i]),np.exp(beta[i]+1.96*se[i])
print(f'{name:<20} {odds:>7.3f} [{low:.3f}, {high:.3f}]')
print(f'Held-out ROC AUC: {auc:.3f}')
print('Bootstrap 95% CI for held-out AUC (2000 resamples): '
f'[{lo:.3f}, {hi:.3f}]')Printed output:
Simulated data, seed 41
Rows: 600; events: 35; non-events: 565
Split: train 420; held-out test 180
Model: logistic regression; predictors glucose, BMI, age; predictors centered and scaled on train data
Term OR per SD 95% Wald CI
glucose 2.269 [1.427, 3.609]
BMI 1.001 [0.662, 1.515]
age 1.292 [0.836, 1.996]
Held-out ROC AUC: 0.743
Bootstrap 95% CI for held-out AUC (2000 resamples): [0.611, 0.861]Why 35 Events Were Not Enough
A logistic regression learns from events, the rows with the outcome. With 5.8% of 600 rows flagged, the model had few events to learn from, and the intervals show it. Glucose has the largest built-in effect, and the fit found it. BMI and age have smaller built-in effects, and the fit could not separate them from zero.
That is the pattern to expect when the outcome is rare or the sample is small: large effects show up, moderate ones hide in wide intervals. Absence of a clear result is not evidence that a predictor has no effect. Before you collect data, plan the sample size from the effect size and outcome rate you expect, using a power calculation, and not from a rule of thumb.
Odds Ratios Are Not Risk Ratios
An odds ratio of 2 does not mean twice the risk. It means twice the odds. The two are close when the outcome is rare, and they drift apart as the outcome becomes common. If you need risk ratios for a clinical message, use a method built for them, such as Poisson regression with robust variance.
Discrimination and Calibration
The ROC curve plots the true positive rate against the false positive rate across all cutoffs. AUC is the chance that a random flagged case gets a higher predicted risk than a random unflagged case, so 0.5 is guessing and 1.0 is perfect ranking. In the example, the held-out AUC was 0.743 and the bootstrap interval ran from 0.611 to 0.861. That wide range is what 180 rows with few events looks like.
An AUC computed on the rows used to fit the model can be optimistic, so score held-out rows or use cross-validation. If you impute missing values first, do it without using the outcome. Filling gaps with medians computed within each outcome group leaks the label into the predictors.
Also remember that discrimination is not calibration. A model can rank cases well and still give probabilities that do not match observed rates. If you plan to use the probabilities, check a calibration plot as well.
Interpreting Your Results: A Checklist
1. Check the data first. Look for impossible values. If a file codes missing measurements as 0 where 0 cannot occur, such as glucose, treat those as missing before you fit anything.
2. Look at intervals, not just point estimates. An odds ratio of 1.8 with an interval of [0.9, 3.6] includes 1, so the data do not rule out no effect. A wide interval means weak evidence.
3. Separate statistical from practical size. A small odds ratio can have a tight interval in a large sample and still matter little in practice.
4. Read odds ratios as adjusted effects. Each one is the effect of that predictor with the others held constant. A predictor can look related to the outcome by itself and lose its effect after adjustment.
5. Check discrimination out of sample. Use held-out rows or cross-validation, and report an interval for AUC as the example does.
6. Check calibration separately if you will use the predicted probabilities.
Common Methodological Mistakes
Mistake 1: Treating missing-value codes as real measurements. A zero in a field where zero cannot occur distorts coefficients and predictions. Profile such values and handle them on purpose.
Mistake 2: Too many predictors for the events you have. Adding interaction terms multiplies the number of parameters. With few events, the model overfits and the intervals widen.
Mistake 3: Reading odds ratios as relative risks. See above: they overstate risk ratios when the outcome is common.
Mistake 4: Using AUC as the only metric. AUC measures ranking. It does not show calibration, sensitivity and specificity at your chosen cutoff, or clinical usefulness. Report those at a cutoff that fits your use.
Mistake 5: Claiming cause without an experimental design. Logistic regression on observed data finds associations. Even a strong glucose odds ratio does not show that lowering glucose prevents diabetes. Say "associated with" and not "causes."
Extending the Analysis
Interaction effects: Does the BMI effect differ by age group? Add interaction terms and test them. Interactions are harder to detect than main effects and need more events.
Risk scores: You can turn model coefficients into a points score for quick use. Test the score on held-out rows and check its calibration.
Time-to-event analysis: If you have diagnosis dates and follow-up, a Cox proportional hazards model estimates hazard ratios and handles patients lost to follow-up. It answers a different question: which predictors speed up onset?
Independent test: Fit on one population and score on another. A drop in AUC tells you the first fit was optimistic or the relationships do not carry over.
Machine learning comparison: Random forests and boosted trees can capture curves and interactions. Compare their AUC under cross-validation with the logistic model's, and weigh any gain against the loss of readable odds ratios.
Run This Analysis on Your Own Data
Upload a patient-level file with predictors and a yes-or-no outcome. MCP Analytics fits a logistic regression, reports odds ratios with 95% intervals, and draws an ROC curve with its AUC.
Your turn
Bring your own data and the question you actually need answered.
MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai
Frequently Asked Questions
What did the simulated example find for glucose, BMI and age?
The script built in a glucose odds ratio of about 2.0 per SD, a BMI odds ratio of about 1.4 and an age odds ratio of about 1.2. With 35 events in 600 rows, the fit found glucose (2.269, interval 1.427 to 3.609). The BMI estimate was 1.001 [0.662, 1.515] and the age estimate was 1.292 [0.836, 1.996], and both intervals include 1.
Why did the fit miss the BMI and age effects?
The effects were built in, but with only 35 events the run could not tell them from zero. That is low power, not no effect. The BMI estimate of 1.001 is also lower than the built-in 1.4, which is chance at this sample size.
What does a held-out AUC of 0.743 mean?
It means the model, scored on 180 rows it did not learn from, ranked a random flagged case above a random unflagged case about 74% of the time in this run. The bootstrap interval ran from 0.611 to 0.861, which is wide because the held-out set has few events.
How do you handle zero-coded missing data in logistic regression?
If a file uses 0 for a missing value in a measure where 0 cannot occur, such as glucose, treat those entries as missing. Impute without using the outcome, for example with the overall median of the recorded values, or use multiple imputation. Filling with medians computed within outcome groups leaks the outcome into the predictors.
How many events do I need to detect diabetes risk factors?
That depends on the effect size you want to detect and how common the outcome is, so use a power calculation for your case. The example shows what too few looks like: with 35 events in 600 rows, a built-in BMI effect and a built-in age effect could not be told apart from zero.