Objective: The extent of disagreement between large and smaller randomized, controlled trials on mental health issues is unknown. The authors aimed to compare the results of large versus smaller trials on mental health-related interventions.
Method: The authors screened 161 Cochrane and 254 Database of Abstracts of Reviews of Effectiveness systematic reviews on mental health-related interventions. They identified 16 meta-analyses with at least one "large" randomized trial with sample size >800 and at least one "smaller" trial. Effect sizes were calculated separately for large and smaller trials. Heterogeneity was assessed between all studies, within each group (large and smaller studies), and between large and smaller studies.
Results: Significant between-study heterogeneity was seen in five meta-analyses. By random-effects calculations, the results of large and smaller trials differed beyond chance in four meta-analyses (25%). In three of these disagreements (effect of day care on IQ, discontinuation of antidepressants, risperidone versus typical antipsychotics for schizophrenia), the smaller trials showed greater effect sizes than the large trials. The inverse was seen in one case (olanzapine versus typical antipsychotics for schizophrenia). With fixed-effects models, disagreements beyond chance occurred in five cases (31%). In four meta-analyses, the effect size differed over twofold between large and smaller trials. Various quality and design parameters were identified as potential explanations for some disagreements.
Conclusions: Large trials are uncommon in mental health. Their results are usually comparable with the results of smaller studies, but major disagreements do occur. Both large and smaller trials should be scrutinized as they offer a continuum of randomized evidence.