Summary of Green et al. (2010): Enough Already about “Black Box” Experiments: Studying Mediation is More Difficult than Most Scholars Suppose.

Summary of Green et al. (2010): Enough Already about “Black Box” Experiments: Studying Mediation is More Difficult than Most Scholars Suppose.

            Understanding that vitamin C mediates the outcome between the consumption of lime and the lowered incidence of scurvy, like numerous other scientific discoveries, highlights the importance of understanding causal effects.  But understanding causation is not only relegated to scientific experimentation.  We, social workers (and other social scientists), are also a curious people, not content with mere associations and outputs but continually wanting to understand more: when an intervention or program has worked (or failed to work) we want to know why.  I may (in the future) conduct research that indicates that parents of children with neurodevelopmental disorders (NDD) have an improved quality of life (QOL) once they receive support services, or that the children themselves report higher levels of QOL following their parents’ receipt of such supports, but I want to understand what other factors may (or may not) have also impacted that outcome.  In short, I want to open the “black box” however, is it possible to do so effectively?

Green et al. state that social science journals “abound” with articles purporting “… well established…” mediating results with a “… growing enthusiasm for regression models…”.  However, their view is that these models “…rest on naïve assumptions… that it is a relatively simple matter to establish the mechanism by which causality is transmitted”. In Enough Already about “Black Box” Experiments: Studying Mediation is More Difficult than Most Scholars Suppose, Green et al. highlight complex issues to consider about mediation and causation:

1) Mediation (which grew in popularity following the publication of Baron and Kenny (1986), is conventionally tested using regression approaches that are flawed and that assume many factors, i.e. that “… M be independent of unmeasured factors that affect Y; researchers often examine several mediators (“one at a time or in different combinations), as establishing “causal pathways” and/or their causation direction is difficult which leads to two critiques: concerns for omitted variables and poor measurement that leads to the underestimation of M’s effect. According to Green et al. even the use of structural equation modeling, albeit a “… step in the right direction insofar as it addresses the problem of measurement error… does nothing to address the problem of omitted variables”.

2) Although “… experiments are the gold standard for estimating causal parameters…”, designing an experiment that will only manipulate the M we are interested in rather than other Ms that may also mediate the effects is very challenging.  In addition, conclusions of mediation effects that are based on “single interventions” cannot truly be generalized to a larger population unless “enough experimental interventions” are conducted, which is “a formidable undertaking”.

3) “Unobserved sources of variation in effect size can throw off any attempt to draw inferences about mediation”.  Green et al. state that it is possible for subjects within the same study to be ruled by different “causal laws” and this counters the usual assumption that all observations within a study will be structured by the same parameters. Green et al. demonstrate this using 4 different models (that I understand with great difficulty).  What I easily understand is that as a result, it is possible to have very different outcomes.  Again, Green et al. suggest that “… multiple experiments- maybe decades worth- will be necessary”  to deal with “heterogeneous treatment effects”.

Contrasting with Guo (2014), who strongly advocates for the use of quantitative statistical methods in social work research, Green et al. seem to question whether it is necessary to open the black box at all.  They state that many theoretical and practical contributions are derived by establishing significance in the effects of variables on each other without having to understand their causal pathways. In fact, they suggest that researchers “… measure as many outcomes as possible when conducting experiments” rather than invest time and resources into trying to manipulate mediators “… as there is no guarantee that the experimental intervention will produce a substantially interesting average effect on the outcome”.

To return to my opening examples, in order for me to promote the importance of support services effectively, is it enough to demonstrate that parents and children have improved QOL following receipt of support or would understanding the mediator effects provide a sounder argument? Should we simply be content that lime consumption reduced the incidence of scurvy or as the saying goes, “When life gives you limes, rearrange the letters so it says smile” (Unknown). Perhaps the answer lies with whom our audience is and what we are trying to accomplish in our studies.

Reflection: Thinking about your future study: a) When and for what reasons do you think it might be wise to open the black box? b) When and for what reasons might it be wise to keep the box shut?

References:

Baron, R.M., & David A. K. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51:1173-82.

Guo, S. (2014). Shaping Social Work Science: What Should Quantitative Researchers Do? Research on Social Work Practice, 1-12. DOI: 10.1177/1049731514527517

2 responses to “Summary of Green et al. (2010): Enough Already about “Black Box” Experiments: Studying Mediation is More Difficult than Most Scholars Suppose.”

  1. Jaime Lenet says:

    Never heard the limes-smile saying before. Your summary raises some interesting questions. I have also wondered about the supreme value of conducting causality-seeking research, especially when we know that we can only ever infer about causality.

  2. Lukas says:

    In response to “rearranging “Limes” into “Smile”, here is a video that advocates just this: The Single Most Important Thing You Can Do For Your Stress
    https://www.youtube.com/watch?v=I6402QJp52M

Leave a Reply

Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.