Market Basket Analysis: Method Guide

1. Introduction and Scope

1.1 What Market Basket Analysis Does

Market basket analysis asks which items are purchased together. The canonical illustration, an association between beer and diapers, is retold so often in data science folklore that its origin is hard to trace; treat it as a story about the kind of question being asked, not as evidence of anything. The practical question is always the same: given a set of transactions, which combinations of items occur together often enough, and more often than chance, to be worth attention?

The technique applies wherever records group items into sets: retail baskets, online orders, restaurant tickets, or any other list of things that happened together. The output is a set of association rules of the form "if A, then B", each with metrics that describe how strong the pattern is.

1.2 Why It Is Easy to Misuse

The number of possible item combinations grows very quickly with the size of the catalog, so algorithms rely on thresholds to limit what they report. Poorly chosen thresholds either bury the reader in thousands of obvious rules or hide the few interesting ones. Separately, the metrics are easy to misread: a rule with high confidence may simply reflect a very popular item, and a rule with high lift may rest on only a handful of baskets.

1.3 Scope of This Paper

This paper covers the method only: the definitions, the algorithms, data preparation, threshold selection, interpretation and common mistakes. It does not report results from any dataset, and it makes no claim about how much revenue or conversion any approach produces. Where it gives numeric ranges, they are conventional starting points to adjust, not findings. The one numeric example, in Section 5, uses made-up numbers and is labelled that way.

2. Background

2.1 Association Rule Mining

Market basket analysis rests on association rule mining, formalised in the early 1990s. The Apriori algorithm, introduced by Agrawal and Srikant, established the standard approach for finding frequent itemsets in transaction databases. It uses the principle that if an itemset is frequent, all of its subsets must also be frequent, which lets it prune the search space through level-wise exploration.

Later algorithms addressed computational cost. FP-Growth avoids candidate generation by compressing the transactions into a compact FP-tree and extracting patterns through recursive traversal. ECLAT uses a depth-first search over sets of transaction identifiers and is often discussed for datasets with long transactions.

2.2 Where the Technique Is Used

The same idea is applied in several domains: "frequently bought together" features in online stores, store layout and promotion planning in retail, cross-sell analysis for financial products, and co-occurrence analysis of items in other record sets. The mathematics is identical in each case; what changes is what counts as a transaction and what counts as an item.

2.3 Known Limitations of the Method

Standard association metrics describe frequency, not value. They do not measure revenue, margin or customer lifetime value. High-support rules tend to involve popular items that already sell well. Low-support rules may point to niche pairings but fail minimum thresholds in aggregate data. Treating all transactions alike can also hide differences between customer groups, and a rule computed on a historical snapshot may not reflect current behaviour. These are properties of the method, and the guidance below is aimed at working around them.

3. Methodology: How the Method Works

3.1 The Input: Transactions and Items

The input is a list of transactions, each containing a set of items. Quantities and order are ignored by the basic method; only presence in the basket matters. Because most items appear in only a small share of transactions, the item-by-transaction matrix is sparse, and the algorithms are designed with that in mind.

3.2 Support, Confidence and Lift

For a rule A → B, where A and B are itemsets, the three standard metrics are:

• Support of an itemset is the fraction of transactions that contain it. Support(A ∪ B) = (transactions containing both A and B) / (all transactions).
• Confidence of A → B is the fraction of transactions containing A that also contain B. Confidence = Support(A ∪ B) / Support(A). It is the conditional probability of B given A.
• Lift of A → B is Confidence / Support(B), equivalently Support(A ∪ B) / (Support(A) × Support(B)). A lift of 1 means A and B appear together exactly as often as independence would predict. Above 1 indicates a positive association, and below 1 indicates the items appear together less often than chance.

Confidence alone is misleading when B is very common, because B will follow almost anything. Lift corrects for this by comparing against the base rate of B. Support matters because a rule built on very few baskets is unreliable whatever its lift.

3.3 The Apriori Algorithm

Apriori works level by level. It first counts single items and keeps those meeting the minimum support. It then builds candidate pairs only from the surviving items, counts them, and keeps the frequent pairs. It continues to triples and larger itemsets, each time generating candidates only from the frequent itemsets of the previous size. Because any subset of a frequent itemset must itself be frequent, large parts of the search space are never examined. Once frequent itemsets are found, rules are formed by splitting each itemset into an antecedent and a consequent and keeping those that meet the confidence (and optionally lift) threshold.

3.4 The FP-Growth Algorithm

FP-Growth works in two phases. First it scans the data to find frequent items and builds an FP-tree, a prefix tree in which transactions that share leading items share a path, so the data is stored compactly. Second, it mines the tree recursively, extracting frequent itemsets from conditional sub-trees without generating candidate lists. It typically needs fewer passes over the data than Apriori and is a common choice for large item catalogs. Both algorithms return the same frequent itemsets for the same thresholds; they differ in how they get there.

3.5 Data Preparation

Every metric is a ratio of counts, so anything that distorts the counts distorts the rules. Common preparation steps are:

• Remove records missing key fields such as transaction identifier or item code.
• Consolidate duplicates. The same transaction appearing twice inflates support for its items.
• Decide how to treat returns. Either remove them or analyse them as a separate stream; do not let them silently count as purchases.
• Exclude test and migration records, and review unusually large baskets such as bulk or wholesale orders that do not reflect ordinary shopping.
• Choose the item level. Merging variants (sizes, colours) or rolling items up to categories reduces sparsity and raises support, at the cost of specificity. A common approach is to explore at category level and then drill into item level for promising categories.

4. Key Points

5. Worked Example (Illustrative)

Illustrative example with made-up numbers. These figures are invented to show the arithmetic. They do not come from any real dataset.

Suppose a shop has 100 baskets. Suppose 40 contain bread, 25 contain butter, and 20 contain both. Suppose 60 contain milk, 30 contain eggs, and 18 contain both milk and eggs.

Reading the table in this made-up scenario: bread and butter appear together in a fifth of baskets, and a basket with bread contains butter half the time, which is twice as often as the 25 percent base rate for butter, so lift is 2.0. The reverse rule has the same support and the same lift but a higher confidence, because butter is the rarer item. Milk and eggs appear together in 18 baskets, but that is exactly what independence predicts (0.60 × 0.30 = 0.18), so lift is 1.0 and the rule carries no information beyond the popularity of the two items, even though its support is nearly as high as the bread and butter rule.

The example shows why lift is read alongside support and confidence, and why a rule's direction matters. It says nothing about what real shoppers do.

6. Practical Guidance

The steps below are a suggested order of work, not a measured or proven process. Adapt them to your data and to the decision the analysis is meant to inform.

Scale

Small catalogs and modest transaction counts are handled easily by standard libraries. Very large catalogs make low support thresholds expensive, since memory and time grow with the number of frequent itemsets. FP-Growth is generally preferred over Apriori at that scale, and raising the support threshold, rolling items up to categories, or sampling for initial exploration are the usual ways to keep a run tractable.

7. Conclusion

Market basket analysis is a well-established method for describing which items occur together. Its core is small: three metrics (support, confidence and lift) and two main algorithms (Apriori and FP-Growth) that find frequent itemsets under a support threshold. Most of the practical difficulty lies around the algorithm rather than in it: preparing the counts so they mean what they appear to mean, choosing thresholds that yield a reviewable list, reading lift and support alongside confidence, and remembering that a rule describes the past data and does not by itself predict the effect of acting on it.

This paper has explained those pieces and illustrated the arithmetic with an invented example. It makes no claim about results on real data. If you have transaction data of your own, the free tool linked at the top of this page runs the analysis on a CSV so you can see the rules your own baskets produce.