Subgroup analysis and post hoc analysis

In reference to the last post, Dr. James Gaulte said...

Could you explain how risk stratified analysis avoid the problem(s) of post hoc sub group analysis namely the multi-comparison problem of false positives and reduced power of the subgroup(s) to show a real difference.

I believe a distinction should be made between subgroup analysis and post hoc analysis. (What follows is all IMHO. Comments welcome)

Post hoc analysis--the more infamous of the 2--refers an exploration of the data that is both unplanned and potentially unlimited, yielding misleading results that are merely superficially significant. It is misleading because the conventional p-value cutoff of 0.05 is no longer valid in the face of repeated testing. It is as if we flipped a coin until we observed 5 heads in a roll, and reported this finding by itself.

Dr. Pocock has proposed the use of interaction test, a rather conservative test, for post hoc analysis.

I think of post hoc analysis as an optimization of the Results section with respect to the Methods section of a paper. This is in itself not problematic as long as we recognize its exploratory nature.

Subgroup analysis refers to an investigation into whether the treatment effect is different in a subset of the patients defined by some baseline characteristic. Subgroup analysis may be exploratory or confirmatory in nature. It may be a priori or a posteriori. For example, investigators may plan, ahead of a trial on heart failure, to analyze the subgroup of patients in NYHA Class IV at baseline.

Nevertheless, subgroup analysis is problematic in that by dividing the patient population into smaller sets, it loses statistical power.

What Hayward et al is proposing is in my opinion a priori subgroup analysis of RCTs. By stipulating that RCTs incorporate stratified analysis in terms of commonly used risk categories, they in effect are requiring investigators to plan to study their treatment with respect to pre-defined subgroups. In fact they make the explicit distinction from �one-variable-at-a-time� post hoc subgroup analysis.

To answer Dr. Gaulte�s concerns, then, I believe Hayward et al�s stipulation of subgroup analysis is a priori and not exploratory. Hence it is not prone to problems associated with post hoc analysis.

A priori or a posteriori, subgroup analysis suffers from reduced statistical power. Hayward et al suggests that

Even for small studies that have marginal statistical power, risk-stratified analysis will still be valuable for comparing results between different studies or conducting meta-analyses.

In addition, as an a priori analysis, investigators are in good position to design their RCTs to endow it with sufficient power to detect treatment differences with respect of risk-stratified subgroups.

References: Pocock 2002.

Comments welcome.


Technorati Tags:
medicine statistics clinical trials subgroup analysis