How Can One Treatment Win Every Group but Lose Overall?

A hospital compares two treatments for kidney stones. Treatment A succeeds more often for patients with smaller stones. Treatment A also succeeds more often for patients with larger stones.

Then the hospital combines every patient into one table. Treatment B has the higher success rate.

No patient changed outcome. No fraction was calculated incorrectly. The reversal happens because the two treatments were used on different mixtures of easier and harder cases. This pattern is called Simpson’s paradox.

The reversal in 700 records

A widely used example comes from a 1986 British Medical Journal study of kidney-stone treatments. To keep the arithmetic in view, call open surgery Treatment A and percutaneous nephrolithotomy Treatment B. A treatment counted as successful when it eliminated a stone or reduced the stone below the study’s specified size after three months.

The diagram below displays the two size groups first. Treatment A uses diagonal hatching; Treatment B uses dots. Switch to “All patients” to pool the same counts.

For smaller stones, Treatment A succeeded in $81$ of $87$ cases:

Treatment B succeeded in $234$ of $270$ smaller-stone cases:

Treatment A leads by about $6.4$ percentage points in the smaller-stone group.

For larger stones, the corresponding rates are

Treatment A leads again, this time by about $4.2$ percentage points.

Now add successes and patients across the two stone sizes. Treatment A has

Treatment B has

The pooled comparison favors Treatment B by about $4.6$ percentage points. The apparent winner has reversed.

The records in the 1986 study were not produced by a randomized head-to-head trial: the treatment groups came from different periods, and the researchers themselves discussed differences among the patients and procedures. The table demonstrates the arithmetic of reversal. It does not, by itself, establish which procedure caused better outcomes.

The hidden weights

The word “average” hides an important choice: what weights does each subgroup receive?

For Treatment A, $263$ of $350$ patients had larger stones. That proportion is

For Treatment B, only $80$ of $350$ patients had larger stones:

Treatment A therefore faced the harder category about three quarters of the time. Treatment B faced the easier category more than three quarters of the time. The overall rates are weighted averages, but they use different weights:

while

Here $p_A$ and $p_B$ denote the pooled success proportions for Treatments A and B. Each coefficient, such as $87/350$, is the fraction of that treatment’s patients who belong to one stone-size group.

Suppose instead that both treatments had exactly the same mixture: a proportion $w$ of smaller-stone cases and a proportion $1-w$ of larger-stone cases, where $0\leq w\leq1$. If Treatment A had the higher rate in both groups, then

Multiplying both inequalities by the same nonnegative weights and adding gives

With equal weights, the order cannot reverse. Simpson’s paradox needs unequal composition across the quantities being compared.

A definition, and why “paradox” is fair

Let $X$ identify which treatment or category is being compared, let $Y$ record an outcome such as success, and let $Z$ identify a subgroup such as stone size. A Simpson reversal occurs when the association between $X$ and $Y$ has one direction within every relevant value of $Z$, but the association has the opposite direction after the values of $Z$ are combined.

For two treatments and two subgroups, the pattern can be written as

for each subgroup $z$, while

in the pooled data. The notation $P(Y\mid X=A,Z=z)$ means the probability of outcome $Y$ among records with treatment $A$ and subgroup value $z$.

The equations are compatible because the pooled probabilities average over $Z$ with treatment-specific weights. What feels paradoxical is the verbal shortcut “A does better than B.” The sentence is incomplete until the comparison population is named.

Which table should you trust?

Neither “always split the data” nor “always use the total” is a sound rule. The question determines which comparison matters.

For the kidney-stone example, stone size existed before treatment and strongly affected the chance of success. A like-for-like treatment comparison should therefore account for stone size. If a hospital instead asks how many of its actual patients recovered under each historical program, the pooled totals describe those particular programs and their particular case mixes.

Causal questions require more than arithmetic. A third variable can be a confounder, meaning a prior cause connected to both the treatment and the outcome. Adjusting for a confounder can remove a misleading mixture. But conditioning on a variable caused by the treatment, or on a variable caused jointly by the treatment and the outcome, can create a misleading association. Judea Pearl’s discussion of Simpson’s paradox emphasizes that probability tables alone do not decide whether to pool or separate; the causal relationships among the variables do.

A practical audit therefore asks three questions before choosing a rate:

1. What population does the decision concern?
2. Did the grouping variable exist before the action being compared?
3. Does subject-matter knowledge support the proposed causal relationship?

The reversal is a warning to inspect those questions, not a machine that answers them.

How the idea acquired its name

Statisticians encountered mixture effects before the phrase “Simpson’s paradox” existed. In 1903, British statistician George Udny Yule devoted a section of “Notes on the Theory of Association of Attributes in Statistics” to fallacies caused by mixing distinct records. Karl Pearson and collaborators had discussed related aggregation effects in 1899.

Edward H. Simpson gave the phenomenon a compact mathematical treatment in his 1951 paper “The Interpretation of Interaction in Contingency Tables”. Simpson’s examples concerned whether associations in separate $2\times2$ tables survive when those tables are combined. Colin Blyth used the name Simpson’s paradox in a 1972 paper about reversals and decision-making. The longer history is why some authors use Yule–Simpson effect.

One famous application arrived in 1975. Peter Bickel, Eugene Hammel, and J. W. O’Connell studied 1973 graduate admissions at the University of California, Berkeley. The overall admission rate was lower for women, but applicants of different genders had applied in different proportions to departments with very different selectivity. Their department-level analysis in Science showed why an aggregate disparity cannot, on its own, identify where a selection process operates. The paper did not make fairness measurement easy; it showed that the unit of comparison matters.

Where reversals matter now

Clinical researchers routinely plan how to handle baseline variables that predict outcomes. The United States Food and Drug Administration’s 2023 guidance on covariate adjustment recommends deciding on adjustments before trial results are unblinded. Prespecification helps prevent analysts from searching many subgroup choices until one produces a preferred story.

The same danger appears in education, hiring, lending, online experiments, and recommendation systems. A new interface may improve conversion in every country but look worse overall if it receives unusually heavy traffic from countries with lower baseline conversion. An aggregate fairness measure can move because the composition of applicants changes, even when every within-group rate stays fixed. The calculation is the same weighted-average calculation; the causal and ethical questions differ by domain.

Simpson’s paradox does not imply that every aggregate is deceptive. It identifies one precise failure mode: comparisons can reverse when unlike mixtures are averaged as if the mixtures were the same.

Try the weights

Problem 1

Two online courses use different exercises. Among beginners, Course A has $18$ successes in $20$ attempts and Course B has $72$ in $90$. Among advanced learners, Course A has $12$ successes in $20$ attempts and Course B has $5$ in $10$.

Which course has the higher success rate within each experience group? Which has the higher pooled rate?

Problem 2

Treatment A succeeds $90\%$ of the time in Group 1 and $60\%$ in Group 2. Treatment B succeeds $80\%$ in Group 1 and $50\%$ in Group 2. Both treatments are used on $40\%$ Group 1 cases and $60\%$ Group 2 cases. Can the pooled rates reverse?

The idea to keep

A pooled rate is a weighted average. If two options are tested on different mixtures of easy and hard cases, the weights can reverse the comparison—even when one option has the higher rate in every like-for-like group.

Continue with Bayes’ theorem and base rates for another way that population composition changes a probability, or with the central limit theorem for the behavior of averages under repeated sampling.

Sources

• Edward H. Simpson, “The Interpretation of Interaction in Contingency Tables”, Journal of the Royal Statistical Society: Series B 13(2), 1951, 238–241.
• George Udny Yule, “Notes on the Theory of Association of Attributes in Statistics”, Biometrika 2(2), 1903, 121–134, especially the section on mixing distinct records.
• Colin R. Blyth, “On Simpson’s Paradox and the Sure-Thing Principle”, Journal of the American Statistical Association 67(338), 1972, 364–366.
• C. R. Charig, D. R. Webb, S. R. Payne, and J. E. Wickham, “Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy”, British Medical Journal 292, 1986, 879–882.
• Peter J. Bickel, Eugene A. Hammel, and J. W. O’Connell, “Sex Bias in Graduate Admissions: Data from Berkeley”, Science 187(4175), 1975, 398–404.
• Judea Pearl, “Understanding Simpson’s Paradox”, The American Statistician 68(1), 2014, 8–13.
• U.S. Food and Drug Administration, Adjusting for Covariates in Randomized Clinical Trials for Drugs and Biological Products, guidance for industry, May 2023.