Simpson paradox

Simpson's paradox is the emergence of a trend in several groups of data which disappears or even reverses when the groups are combined. This happens because of a confounding variable—one that is correlated with both the grouping variable and the outcome. It's a mathematical inevitability of weighted averages, not a flaw in data collection. It is also a particular case of the well-known "correlation is not causation".

Originally, the trend disappears from aggregating the sub-groups. Blyth observed the reversal of correlations instead, which is more surprising.

With Daniel Salazar, I refer to this paradox to explain boson correlations for classical states in the version of the paradox where each group of data shows "no" correlations, but their combination gives rise to apparent bunching patterns, that can be described with a many-body wavefunction.

This case of appearance of correlations in aggregated groups is discussed by J. Aldrich in his review of genuine versus spurious correlations, covering in particular the mechanism by which mixing heterogeneous groups creates apparent aggregate correlations where none exist within subgroups.

The first mention, historically, is by K. Pearson, who showed that skull length and breadth were nearly uncorrelated within males (r=0.087) and within females (r=-0.042), but when the two groups were pooled, a substantial spurious correlation of r=0.197 emerged. The subgroup correlations are near zero; the aggregate one is not. This was introduced earlier theoretically.

The full spectrum of cases, including the one where association appears only in the aggregate, is covered by I. J. Good and Y. Mittal who introduce the term amalgamation paradox.

Y. Mittal coins the term "Yule's Association Paradox" specifically for the case where subgroups are independent but the aggregate is not.

It was also discussed how correlation appearing only in aggregates is structurally distinct from confounding, but arises specifically when the grouping variable is a common cause of both X and Y, rather than a mediator or confounder in the usual sense.

Important texts:

• The Interpretation of Interaction in Contingency Tables. E. Simpson in J. R. Stat. Soc. B: Stat. Methodol. 13:238 (1951). This is the paper the paradox is named after, though E. H. Simpson wasn't the first to notice it.
• Notes on the theory of association of attributes in statistics. G. Yule in Biometrika 2:121 (1903). The first mention of the effect (decades earlier than Simpson).
• On Simpson's Paradox and the Sure-Thing Principle. C. Blyth in J. Am. Stat. Assoc. 67:364 (1972). Formalizes the paradox and connects it to decision theory.
• 🕮Causality. J. Pearl. Cambridge University Press, 2000. [ISBN: 0521773628] Gives the definitive modern treatment, explaining the paradox through the lens of causal graphs (DAGs) and showing that resolving it requires causal—not merely statistical—reasoning.
• Comment: Understanding Simpson's Paradox. J. Pearl in Am. Stat. 68:8 (2014). Particularly readable account of the above.
• Sex Bias in Graduate Admissions: Data from Berkeley. P. Bickel, E. Hammel and J. O'Connell in Science 187:398 (1975). UC Berkeley admissions study, famous real-world illustration of the paradox.

In a quantum context:

• Two quantum Simpson's paradoxes. M. Paris in J. Phys. A.: Math. Gen. 45:132001 (2012).
• Experimental investigation of quantum Simpson's paradox. Y.-L. Li, J.-S. Tang, Y.-T. Wang, Y.-C. Wu, Y.-J. Han, C.-F. Li, G.-C. Guo, Y. Yu, M.-F. Li, G.-W. Zha, H.-Q. Ni and Z.-C. Niu in Phys. Rev. A 88:015804 (2013).
• The Simpson's paradox in quantum mechanics. A. Selvitella in J. Math. Phys. 58:032101 (2017).

Links

• Stanford Encyclopedia of Philosophy.