Purpose

This section documents the data cleaning process that prepared raw transaction data for customer segmentation. Understanding data retention and removal reasons is critical because it directly affects the representativeness of the 162 customers ultimately segmented and the reliability of behavioral insights derived from RFM clustering.

Key Findings

Retention Rate: 78.2% (3,908 of 5,000 rows retained) - A substantial loss indicating significant data quality issues in the raw dataset
Rows Removed: 1,092 observations eliminated due to cancelled transactions, missing customer IDs, and invalid quantities/prices
Final Customer Base: 162 unique customers analyzed, derived from cleaned transaction data
No Train/Test Split: Unsupervised clustering applied to entire cleaned dataset without holdout validation

Interpretation

The 21.8% removal rate suggests the raw transaction data contained considerable noise—primarily cancelled orders and data entry errors. This aggressive filtering ensures the clustering model operates on valid, completed transactions only, which is appropriate for RFM analysis. However, the loss of over 1,000 records means the final 162-customer segment may not fully represent the original customer population, potentially biasing insights toward customers with cleaner transaction histories.

Context

Since K-Means is unsupervised, no train/test split was necessary. The high removal rate reflects typical e