By MCP Analytics Team | Published , updated
Upload your data and get a cluster analysis report. Free.
The numbers below come from made-up data, made by the code on this page. They show the method. They are not output from this tool and not a finding about your data.
Question: If the data holds four built-in groups but two of them sit close together, how many groups does k-means with an average silhouette score choose, and which groups get merged?
Inputs: Simulated data. Seed 1 builds the main dataset: 400 rows, 3 numeric measures, 4 groups of 100 rows each (A, B, C, D). Every group is drawn from a normal (bell curve) distribution with standard deviation 1 on each measure. Built-in group centres: A (0,0,0), B (2.5,0,0), C (8,6,0), D (8,6,7). So A and B are only 2.5 apart (2.5 standard deviations), C and D are 7 apart, and the A/B pair is about 10 away from C. The four groups are real by construction, and so is the fact that A and B are close. Seeds 2 to 21 build 20 more datasets of the same design for the repeat check. The k-means start seed is fixed at 42. The tested range is k = 2 to 8, which is the range the page's sample report searches for this many rows (2 to min(8, rows/10)).
Method: K-means clustering with an average silhouette score to choose the number of groups, as the page's sample report describes. Steps: (1) Standardise each measure to mean 0 and standard deviation 1 so no measure dominates the distances. (2) For each k from 2 to 8, run k-means: pick k starting centres, assign every row to its nearest centre, move each centre to the mean of its rows, and repeat until nothing moves. Do this from 10 random starts and keep the run with the smallest total squared distance. (3) Score each k with the average silhouette width. For one row, a is its average distance to the rest of its own group and b is its average distance to the nearest other group. The silhouette is (b minus a) divided by the larger of the two, from -1 to 1. High means rows sit much closer to their own group than to any other. (4) Pick the k with the highest average. (5) Cross-count the four built-in groups against the groups k-means found, to see which built-in groups were merged. (6) Repeat steps 1 to 4 on 20 fresh datasets and count which k wins.
Result: For seed 1 the average silhouette is 0.628 at k=2, 0.687 at k=3, 0.505 at k=4, 0.389 at k=5, 0.289 at k=6, 0.250 at k=7 and 0.246 at k=8. The score picks k = 3. At k = 3 the cross-count shows built-in A and B (100 rows each) land together in one k-means group, C (99 of its 100 rows) in a second, and D (all 100) in a third. At k = 4, the built-in count, the score is 0.505, which is 0.182 below k = 3 and also below k = 2. Over seeds 2 to 21 the score picked k = 3 in all 20 datasets and never k = 4, so its detection rate for the four built-in groups was 0 of 20. Reading: the score chose fewer groups than were built in, and the merged pair is A and B, the two groups placed closest together. That is expected by construction: the A/B gap was set small on purpose, and the silhouette rewards a split only when rows sit clearly nearer their own group than a neighbour. The 0.182 gap and the 20 of 20 repeats are well beyond seed-to-seed wobble in this design, but this is one design. It does not mean the score is broken: with a gap of 2.5 on a spread of 1 the two groups overlap a lot. It does mean a top-scoring k can hide a split you know exists, so check which groups merge and compare k = 3 with k = 4 before accepting one. This shows the method on made-up data, not what the tool will find in yours.
Limit: This example uses one design (equal sizes, equal spread, one close pair, three measures) and one gap size, so it does not show how the pick changes as the A/B gap grows or shrinks. It also does not run the tool's own R code, only a numpy version of the steps the sample report describes.
Python script (numpy and scipy only):
import numpy as np
# Simulated data. Four groups are BUILT IN: A and B sit close together, C and D sit far away.
CENTRES = np.array([[0, 0, 0], [2.5, 0, 0], [8, 6, 0], [8, 6, 7]], float) # 3 measures
NAMES = "ABCD"; N_PER = 100; SD = 1.0
def make(seed):
rng = np.random.default_rng(seed)
X = np.vstack([rng.normal(c, SD, (N_PER, 3)) for c in CENTRES])
return X, np.repeat(np.arange(4), N_PER)
def kmeans(X, k, rng, starts=10, iters=50):
best = None
for _ in range(starts):
C = X[rng.choice(len(X), k, replace=False)]
for _ in range(iters):
lab = ((X[:, None] - C[None]) ** 2).sum(2).argmin(1)
C2 = np.array([X[lab == j].mean(0) if (lab == j).any() else C[j] for j in range(k)])
if np.allclose(C, C2): break
C = C2
sse = ((X - C[lab]) ** 2).sum()
if best is None or sse < best[0]: best = (sse, lab)
return best[1]
def avg_silhouette(X, lab, D):
s = []
for i in range(len(X)):
own = lab == lab[i]
if own.sum() < 2: s.append(0); continue
a = D[i, own].sum() / (own.sum() - 1)
b = min(D[i, lab == j].mean() for j in set(lab) if j != lab[i])
s.append((b - a) / max(a, b))
return float(np.mean(s))
def search(seed):
X, truth = make(seed)
Xs = (X - X.mean(0)) / X.std(0, ddof=1) # standardise: mean 0, sd 1
D = np.sqrt(((Xs[:, None] - Xs[None]) ** 2).sum(2))
rng = np.random.default_rng(42)
labs = {k: kmeans(Xs, k, rng) for k in range(2, 9)} # k = 2 to min(8, n/10)
sil = {k: avg_silhouette(Xs, labs[k], D) for k in labs}
return sil, labs, truth
sil, labs, truth = search(1)
print("Seed 1, 400 rows, 4 built-in groups of 100. Average silhouette by k:")
for k in sil: print(f" k={k}: {sil[k]:.3f}")
kbest = max(sil, key=sil.get)
print("Silhouette picks k =", kbest)
print("Rows of each built-in group that landed in each k-means group, at the picked k:")
for g in range(4):
print(f" built-in {NAMES[g]}:", [int(((truth == g) & (labs[kbest] == j)).sum()) for j in range(kbest)])
print("Silhouette at k=4 (the built-in count): %.3f" % sil[4])
picks = {}
for s in range(2, 22):
sl, _, _ = search(s); kb = max(sl, key=sl.get); picks[kb] = picks.get(kb, 0) + 1
print("Silhouette pick over seeds 2 to 21 (20 fresh datasets):", dict(sorted(picks.items())))
Printed output from running the code above:
Seed 1, 400 rows, 4 built-in groups of 100. Average silhouette by k:
k=2: 0.628
k=3: 0.687
k=4: 0.505
k=5: 0.389
k=6: 0.289
k=7: 0.250
k=8: 0.246
Silhouette picks k = 3
Rows of each built-in group that landed in each k-means group, at the picked k:
built-in A: [0, 0, 100]
built-in B: [0, 0, 100]
built-in C: [99, 1, 0]
built-in D: [0, 100, 0]
Silhouette at k=4 (the built-in count): 0.505
Silhouette pick over seeds 2 to 21 (20 fresh datasets): {3: 20}
Groups of similar rows in your data: how many the data supports, what sets each apart, how clearly they separate.
Charts, tables and the checks the results rely on.
Plain-English reading of the results.
See our FAQ for pricing, privacy and how the analysis works.
MCP Analytics. Cluster analysis. Free tool. https://mcpanalytics.ai/free/standard_clustering_v2. Add the date you accessed it.
Tell us what went wrong, in your own words. We capture the page you're on automatically, so no need to describe where you are.