You’ll see me refer to meta-analysis in this blog with some regularity, and as it is increasingly being referenced in the popular press, I thought I’d provide a brief explanation for those of you who may not be familiar with it.
Science is conducted one study at a time. A researcher develops a hypothesis, designs a study and gathers data to test the hypothesis, analyzes the data, and then writes up and publishes the findings. After enough researchers have studied a similar topic, a “qualitative” review of the research is often conducted where someone reviews the relevant studies and offers their conclusions about the topic.
Meta-analysis is a statistical technique for quantitatively combining the results from prior studies. It attempts to make integrated reviews less subjective. Meta-analyses have been conducted in many fields including medicine and psychology; several have examined aspects of team effectiveness. Here’s what you should know about meta-analysis:
- A meta-analysis is typically based on a systematic search to find all relevant studies, even unpublished ones if possible. The meta-analyst is expected to explain how they searched for studies, what they found, and why they included or excluded studies.
- They extract one or more “effect sizes” from each study. An effect size is simply a number that indicates the direction and strength of the finding. A correlation coefficient is one example of an effect size, but there are other types as well.
- Characteristics of interest are also “coded.” For example, if we were conducting a meta-analysis of team training we might record the size of the teams in each study, whether the participants were students or employees, and even the quality of the research design employed (e.g., with or without a control group).
- The results from across all the studies are then mathematically combined, for example by calculating a weighted average of all the effect sizes. If there are any unexplained inconsistencies across studies, the characteristics that were coded are then examined (e.g., does team training work better with smaller teams?).
- There are a variety of statistical meta-analytic techniques and programs that can used, but essentially they all attempt to yield numerical indicators that summarize the findings from prior studies.
A meta-analysis is based on multiple studies, so it doesn’t rely too heavily on any single study. Moreover, while meta-analysis doesn’t eliminate subjectivity, it can reduce it and at least make any judgments, such as why a study was excluded, a bit clearer. While not perfect – for example, a meta-analysis is only as good as the body of research it summarizes – I am usually more confident when I can make recommendations that are based on a solid meta-analysis.
I’d like to offer a big thank you to Terry Dickinson, who long ago taught me about meta-analysis, research rigor, and many other things!
In case you are thinking, “I’d like to learn more about meta-analysis” or if you are having difficulty falling asleep tonight, you can learn more here.