Overlapping confidence intervals

tae kwon do bikarmot 050 by Bara via Flickr

From AP's report on patch contraceptives and the risk of venous thromboembolism:

However, because the confidence intervals of the results for the two forms of contraceptive overlap, there actually may be no increased risk from the patch or it may be more than double.

It's a common mistake to use confidence intervals of 2 measures to assess whether they are significantly different. Two measures can have overlapping CIs yet remain statistically significantly different. As an counterexample, let's take this extreme situation:

* Experiment 1 arrives at the conclusion that if the experiment were repeated thousands of times, the outcome measure X will turn out to be 0 with probability 2.5%, 1 with probability 94.5%, 2 with probability 0.5%, and 3 with probability 2.5% (see top graph). Obviously from the graph, the 95% CI for outcome X is from 1 to 2.

* Another (indepedent) experiment 2 arrives at the conclusion that if the experiment were repeated thousands of times, the outcome measure Y will turn out to be 0 with probability 2.5%, 1 with probability 0.5%, 2 with probability 94.5%, and 3 with probability 2.5% (see bottom graph). Again the CI for outcome Y is simple to see and spans 1 and 2.

Now we have 2 CIs that not only overlap but are in fact identical!

Given this fact that X and Y have identifcal CIs (ignoring the tail probabilities for now), can we conclude that they are not statistically significantly different? Since outcome X is so strongly concentrated at value 1 vs the strong weight of outcome Y at value 2, inspecting the graph and relying on our intuition, we are forced to conclude that despite their identical CIs, outcomes X and Y are statistically significantly different.

How does our intuition compare with statistical reality? A simple simulation test found X and Y to be significantly different with p basically equal to 0 (by Mann-Whitney test).

That overlapping CIs do not imply lack of significant difference is true in real-world situations too. More on this later.


Technorati Tags:
Medicine / Statistics / Contraceptives