Summary of Rose & Stone (2011): Instrumental Variable Estimation (IVE) in Social Work Research: A technique for estimating causal effects in nonrandomized settings

Although there already exist sophisticated statistical techniques for estimating causal effects in nonrandomized (e.g., observational) studies such as fixed effect models and propensity score matching, Rose and Stone attempt to make a case for the use of the Instrumental Variable Estimation (IVE) technique in social work research. Traditionally used by econometricians, the authors claim that the use of IVE in the field of social work can lead to less biased causal estimations by properly disentangling causal pathways involved in the study of effects of nonrandomly assigned treatments or interventions. The authors discuss the issue of endogeneity (also see Stone & Rose, 2011) and nonignorability in nonrandomized research, due to the fact that the effect (X) on the outcome variable (Y) cannot be completely manipulated or controlled by the researcher, which introduces the possibility that other unobserved/missing/unmeasured factors play a role in the treatment assignment and thus are also related to the outcome. These confounding variables (e.g.., history, maturation) dilute the true effect of X on Y as well as threaten internal validity and if left unmeasured or unaccounted for will cause biased causal estimates. Thus the goal is to determine which part of X is truly (and directly) casually related to Y, despite the presence of possible confounding variables.

Since observational studies are not viewed as the golden standard as per Rubin’s causal model (as read in Shadish, 2010) due to their nonrandomized (or non-experimental) nature, Rose  and Stone claim that IVE can artificially create the same conditions as random assignment, isolate the effect of X on Y and therefore produce results that can be held to the same golden standard as Rubin’s causal model. Another (and more commonly) used method to inform causality, correlation analysis, is discouraged as it is time consuming and requires a well established theory for the domain being researched (which can often be lacking in the social work discipline). In addition, correlation analysis requires the consideration of ALL possible confounding relationships in order to include them in the statistical model, which can be a daunting exercise. According the authors, IVE allows the researcher to bypass these demanding requirements through the use of a statistical control technique.

However, there are still some pre-requisites in order to properly use IVE – the researcher’s task is to identify the instrumental variable (IV) that will act as a control variable.  This variable is usually unrelated to the model under study, is non-endogenous and is a causal antecedent to X (the endogenous variable). Rose and Stone outline two rules for the selection of an appropriate instrument variable, also referred to as the exclusion requirement:

  1. The instrumental variable (Z) must be highly correlated to the endogenous variable (X) (aka Z must be the causal antecedent to X – pretty straight forward)
  2. Z must be uncorrelated with the error (aka Z cannot be associated with the outcome variable Y through error – not as straight forward).

These two rules must be met in order to be able to make definitive conclusions from the analysis. However, the authors point out that rule #2 is much more difficult to determine, as it cannot be demonstrated through statistical tests or the data itself. Thus, justification of rule #2 has to be derived from information that is external to the sample under study, which requires careful thought/logic exercises. The goal is to determine whether the instrument variable effectively randomizes persons to the condition of X (endogenous variable – cause). Since it is highly unlikely that a perfect instrumental variable will be found (variables have at least one possible causal antecedent in most cases), then most instruments will tend to be imperfect in nature.

Rose and Stone then present the statistical model for IVE in two ways: (1) IVE is contrasted with the Ordinary Least Squares (ORL) causal estimator approach and (2) a multistage estimation of the instrumental variable via a two-stage least squares approach. This part of the article was a bit difficult to follow and grasp, since my level of statistical formula familiarity is hazy at best.

I did find the examples useful in illustrating the ‘practical’ use of IVE in social work research and it allowed me to make possible linkages to my own research interests. For instance, if I want to examine the best age for youth to transition from foster care (and other types of residential care) to independent living (let’s say age 21 compared to 18) in order to achieve the best outcomes (let’s say, well-being) as young adults, I could use date of birth as an instrumental variable in order to determine the true effect of X (transitioning out of foster care) on Y (well-being of former youth in care). However, there are some pitfalls in using the IVE approach, even in my example, since even with a supposedly exogenous variable such as date of birth can become endogenous if (as also discussed in the article) for instance some of the parents had a role in determining exact date of birth of their child and thus making it a non-random occurrence. In addition, the role of human choice or behaviour (or as Rose and Stone call unobserved human agency) could also influence the outcome in my study, since some youth might chose on their own accord to transition to independent living at an earlier or later age, for various personal reasons.

Although I do agree that having knowledge of the IVE technique is definitely useful to the social work discipline in terms of being able to properly read and assess studies that employ the technique, it is important to weigh the pros and cons associated with its use within the particular research context it is being considered for. The authors caution that in order to properly use IVE, the sample size should be sufficiently large; however they do not explain how one can determine the appropriate sample size for their particular study (perhaps Maxwell, Kelly and Rausch’s (2008) sample size planning techniques could be of use). I am not convinced that it should be held at a golden standard similar to Rubin’s causal model, as justification of rule #2 seems to be a lot more challenging (and perhaps at times impossible) than Rose and Stone would like to admit since the burden of proof requires an extensive inductive analysis to rule out any possible unwanted associations. Even when rule #2 seems to be satisfied from the researcher’s perspective, explaining the rationale behind it to others might not be as obvious and can thus propel the “black box” phenomenon (as discussed in Green et al., 2010).

References:

Green, D. P., Ha, S. E., & Bullock J. G. (2010). Enough Already about “Black Box” Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose. The Annals of the American Academy of Political and Social Science, 628, 200-208.

Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563.

Rose, R. A., & Stone, S. I. (2011). Instrumental Variable Estimation in Social Work Research: A technique for estimating causal effects in nonrandomized settings. Journal of the Society for Social Work and Research, 2(2), 76-88.

Shadish, W. R. (2010). Campbell and Rubin: A Primer and Comparison of Their Approaches to Causal Inference in Field Settings. Psychological Methods, 15(1), 3–17

Stone, S. I., & Rose, R. A. (2011). Social Work Research and Endogeneity Bias. Journal of the Society for Social Work and Research, 2(2), 54-75.

Leave a Reply

Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.