SVM on Imbalanced Data: A Guide to the Methods

1. Introduction and Scope

1.1 The Problem

Support Vector Machines find the hyperplane that maximizes the margin between two classes. When one class is far more frequent than the other, the same procedure can produce a classifier that scores well on accuracy by predicting the majority class for nearly every input, while identifying few or none of the rare cases that the model was built to find.

The dual formulation of the standard SVM contains the constraint that the coefficients, weighted by class label, sum to zero. Any non-trivial solution therefore needs non-zero coefficients on both classes. With a very rare class there are few minority points to supply them, and the fitted boundary tends to sit close to the minority class and favor the majority.

1.2 What This Paper Covers

This paper explains the following approaches and how they relate to the SVM objective:

1. Class weighting: adjusting the misclassification penalty per class
2. Resampling: random undersampling, random oversampling and SMOTE
3. Threshold moving: changing the cut-off applied to the decision score
4. Cost-sensitive learning: choosing models and thresholds by the cost of each error type
5. One-Class SVM: treating the rare class as anomalies against a model of the common class
6. Other algorithms: when a method without the margin formulation may be worth considering

1.3 What This Paper Does Not Do

It does not rank the methods by measured performance, give break-even imbalance ratios, or estimate costs. Those answers depend on the data, the kernel, the number of features, how well the classes separate and the cost of each error. They can only be found by testing on your own data, using the evaluation approach described in Section 4.

2. Background: How an SVM Handles Two Classes

2.1 How SVMs Construct Decision Boundaries

The SVM looks for a hyperplane that separates two classes while maximizing the margin, which is the distance between the hyperplane and the nearest points of either class. The points that lie on or inside the margin are the support vectors. The position and orientation of the hyperplane depend only on these points. Points that are not support vectors can be added, removed or moved, provided they stay clear of the margin, without changing the fitted boundary.

In the primal soft-margin formulation, the SVM solves:

minimize: (1/2)||w||² + C∑ξᵢ
subject to: yᵢ(w·xᵢ + b) ≥ 1 - ξᵢ, ξᵢ ≥ 0

Here w sets the hyperplane orientation, b is the bias term, ξᵢ are slack variables that allow some misclassification, and C controls the trade-off between a wide margin and a low training error. The dual formulation, which most implementations solve, is:

maximize: ∑αᵢ - (1/2)∑∑αᵢαⱼyᵢyⱼ(xᵢ·xⱼ)
subject to: ∑αᵢyᵢ = 0, 0 ≤ αᵢ ≤ C

The constraint ∑αᵢyᵢ = 0 ties the two classes together. Since yᵢ is either -1 or +1, any solution with non-zero α values must have non-zero α values on both classes. Only the support vectors have non-zero α.

2.2 Why Imbalanced Data Is Hard

With a soft margin, the objective adds up penalties over all training points. If 99% of the points belong to one class, a boundary that sacrifices a few minority points to gain a wider margin or fewer majority errors can have a lower total objective than one that fits the minority class well. The result is a boundary pushed toward the minority class region, a classifier that predicts the majority class almost everywhere, and an accuracy figure that looks high while minority recall is low.

Because only support vectors shape the boundary, the handful of minority points near the margin carry most of the influence of that class. If the minority class overlaps the majority class, many of its points end up at the upper bound α = C. If it is well separated and very small, it may supply only a few support vectors.

2.3 Common Approaches at a Glance

Random undersampling: discarding majority class points until the classes are closer to balance. It is cheap and shrinks training time, but it throws away information.

Random oversampling: duplicating minority class points. It keeps all the data, but repeated copies can encourage overfitting to those specific examples.

SMOTE: creating synthetic minority points by interpolating between a minority point and one of its nearest minority neighbors (Chawla et al., 2002). It avoids exact duplicates, but the synthetic points are only as realistic as the interpolation assumption.

Class weighting: multiplying C by a different weight for each class, which raises the cost of minority errors without changing the data.

Threshold moving: leaving the trained model alone and changing the cut-off applied to its decision score.

3. Methods in Detail

3.1 Class Weighting

Class weighting replaces the single penalty C with a per-class penalty. With weight w₁ on the minority class and w₂ on the majority class, the objective becomes:

minimize: (1/2)||w||² + C·w₁∑ξᵢ(minority) + C·w₂∑ξᵢ(majority)

The upper bound on the dual coefficients becomes C·w₁ for minority points and C·w₂ for majority points, so minority points are allowed to push the boundary harder. A common starting point is to set each weight inversely proportional to its class frequency, which is what the "balanced" option in scikit-learn does. Weights can then be tuned along with C and the kernel parameters by cross-validation. Very large weights can overcorrect, making the classifier flag many majority points as minority.

3.2 Resampling and SMOTE

Resampling changes the data the SVM sees. Undersampling shrinks the majority class, oversampling enlarges the minority class, and SMOTE oversamples with interpolated points. Several cautions are widely discussed in the literature:

• Resample only the training portion of each cross-validation fold. Resampling before the split leaks information into the evaluation data and inflates the results.
• SMOTE assumes that points on the line between two minority neighbors are plausible minority points. This can fail if the minority class occupies several separate regions, or if it overlaps the majority class.
• In spaces with many features, distances between neighbors carry less information, so the neighbors SMOTE chooses may be less meaningful. Feature selection or dimensionality reduction before oversampling is a common precaution.
• SMOTE variants for categorical features, for borderline points and for noisy data exist, and are worth reading about if the basic version looks unsuitable.

Oversampling also makes the training set larger, which increases SVM training time. Kernel SVM training time typically grows faster than linearly with the number of points.

3.3 Threshold Moving

An SVM produces a signed decision score, and the default rule predicts the positive class when the score is above zero. Moving the cut-off changes the balance between false positives and false negatives without retraining. The cut-off can be chosen on held-out data to maximize a metric such as F-beta, to reach a required recall, or to minimize an expected cost. Platt scaling or isotonic regression can convert decision scores into calibrated probabilities if probabilities are needed.

3.4 Cost-Sensitive Learning

Cost-sensitive learning starts from the observation that the two kinds of error rarely cost the same (Elkan, 2001). If a missed rare case costs more than a false alarm, the model and threshold should be chosen to minimize expected cost, not error count. Class weights and threshold moving are two ways to apply this. For a binary problem with cost c_fn for a false negative and c_fp for a false positive, the cost-minimizing probability threshold is c_fp / (c_fp + c_fn), provided the probabilities are well calibrated.

3.5 One-Class SVM

One-Class SVM (Schölkopf et al., 2001) learns a boundary around the training data of a single class and treats points outside it as anomalies. It can be used when the rare class is so scarce, or so varied, that it cannot be modelled directly. It needs no labelled rare examples to train, though a few labelled examples are still needed to evaluate it.

The key parameter is nu. It is an upper bound on the fraction of training points treated as outliers and a lower bound on the fraction of training points that are support vectors. It is not a direct anomaly threshold. The resulting boundary also depends on the kernel and its width parameter gamma. Because the right values depend on the data, nu and gamma are normally chosen by searching a grid and scoring each setting on held-out labelled data, using a metric matched to the cost of each error type.

3.6 Other Algorithms

The two-class support vector requirement belongs to the SVM formulation. Tree ensembles such as Random Forest, and gradient-boosted trees, support class weights and do not share this structure. Whether one of them is a better choice than an SVM depends on the data and the requirements. SVMs with a kernel are a reasonable option when the data has a smooth non-linear boundary and the dataset is of moderate size, and the comparison is best made by evaluating both on your own data.

4. Evaluating on Imbalanced Data

4.1 Metrics

• Accuracy is misleading. A model that always predicts the majority class gets an accuracy equal to the majority share of the data.
• Recall (sensitivity) is the share of true rare cases that were found. Precision is the share of flagged cases that were truly rare. Reporting both is more informative than either alone.
• F1 and F-beta combine precision and recall into one number. F-beta with beta above 1 weights recall more heavily.
• The precision-recall curve and its area are usually more informative than the ROC curve when the positive class is rare, because ROC-based measures can look good even when precision is poor.
• Cost-weighted measures multiply the counts of false negatives and false positives by their costs and compare the totals.

4.2 Evaluation Protocol

• Use stratified cross-validation so each fold has the same class proportions.
• Apply any resampling inside the training folds only.
• Choose thresholds and hyperparameters on data separate from the data used for the final performance estimate.
• Report the number of rare cases in the evaluation data. With few of them, any recall estimate is uncertain, so report an interval.

5. Worked Example (Illustrative)

Illustrative example with made-up numbers. These figures were invented to show the arithmetic. They are not results from any data or experiment.

Suppose a dataset has 10,000 transactions, of which 100 are fraudulent. Suppose a default SVM predicts "not fraud" for every transaction. Its accuracy is 9,900 / 10,000 = 99%, yet it finds none of the 100 fraudulent cases, so recall is 0.

Suppose class weighting is then added and the model flags 470 transactions, of which 70 are truly fraudulent. The counts would be:

Accuracy fell from 99.0% to 95.7% while recall rose from 0 to 0.70, which shows why accuracy is the wrong yardstick here. Now suppose a missed fraud costs 100 cost units and a false alarm costs 1 cost unit. The default model costs 100 x 100 = 10,000 units. The weighted model costs 30 x 100 + 400 x 1 = 3,400 units. Under these made-up costs the weighted model is cheaper, even though most of its alerts are false alarms. Different costs would give a different answer, which is why the costs should come from the real application.

6. Practical Guidance

The points below are general guidance drawn from the literature and common practice. They are starting points to test, not findings from our own research.

6.1 Start Simple

A common starting point is the plain SVM with class weights set inversely proportional to class frequency, evaluated with stratified cross-validation and the metrics in Section 4. It adds no data, no new parameters beyond the weights, and is a useful baseline against which any more elaborate method can be compared.

6.2 Add a Threshold Step

If the weighted model's precision or recall is not where the application needs it, moving the decision threshold on held-out data is a cheap adjustment that does not require retraining.

6.3 Use Resampling Carefully

If resampling is tried, apply it inside the training folds, compare against the class-weighted baseline, and inspect whether synthetic points look plausible. If the number of features is large, consider reducing it first.

6.4 Consider One-Class SVM When Labelled Rare Cases Are Very Few

When the rare class is too small or too varied to model directly, One-Class SVM or another anomaly detector is a reasonable option. Plan to search nu and gamma on held-out labelled data, and expect results to be sensitive to both.

6.5 Define the Costs

Write down the cost of a false negative and a false positive before comparing methods. Without them, "better" has no meaning. Compare methods on expected cost or on a metric chosen to reflect those costs.

6.6 Compare Against Other Algorithms

An SVM is one option among several. Evaluating a class-weighted tree ensemble alongside it, on the same folds and metrics, is a low-cost way to see whether the SVM is the right tool for the data.

7. Assumptions and Pitfalls

• Leakage through resampling: Oversampling before splitting puts copies or interpolations of the same point in both training and evaluation data.
• Unscaled features: SVMs with common kernels are sensitive to feature scale. Scale features, fitting the scaler on training data only.
• Accuracy as a target: Tuning hyperparameters to maximize accuracy on imbalanced data selects models that ignore the rare class.
• Prior shift: If the class proportions in production differ from those in training, precision in particular will change. Probability calibration and thresholds should be revisited.
• Few rare examples: With a small number of rare cases, recall estimates are noisy and a single fold may contain very few of them.
• Uncalibrated scores: SVM decision scores are not probabilities. Treating them as probabilities in a cost calculation can mislead.
• Label noise: Mislabelled minority points can have a large effect on a margin-based model, because they sit near the boundary.

8. Conclusion

This paper explained how a Support Vector Machine behaves on imbalanced data and the standard ways of handling it. The core point is that the soft-margin objective counts errors in total, so a rare class is easy to under-serve, and each method listed here changes that balance in a different way: class weights change the penalty, resampling changes the data, threshold moving changes the cut-off, and One-Class SVM changes the question being asked.

The paper makes no claim about which method works best, because that depends on the data and on the cost of each error type. What it offers is a set of methods, the metrics that suit them, a way of evaluating them without leaking information, and the pitfalls to watch for. To decide among them for a particular problem, define the costs, set up the evaluation in Section 4, and compare the options on your own data.