summary of Cook, Scriven et.al.,(2010): Thinking about causation in Evaluation: A dialogue with Tom Cook and Michael Scriven

As I mentioned in the last class, I’m a type of critical researcher in the RCT (Randomization Clinical Trial) rather than the enthusiasts for it. My concerns with the RCT are mostly focused on its feasibility and ethical issues in the social work field. However, as one of the advocates of the evidence-based studies, and I always feel so confused with many issues related to RCT or its alternatives.

RCT is certainly the most idealistic way to conduct a research on the causal relationship, but from my perspective, it seems impossible to implement in a real research world. So what choice left for us? Or, is it possible to make it real?

This question seems to be difficult to answer for most researchers, even for the distinguished one like Tom Cook and Michael Scriven (phew).

While I was reading the debate between Tom Cook and Michael Scriven, I was usually on the side of Scriven, but Tom Cook’s idea was quite persuasive, especially the case example of trial to control random assignment problem was quite impressive.

In this debate, Michael Scriven criticized for tough requirements required by the current RCT designs. That is, he argued that not to lose the valid RCT design, researchers are required to keep watching two endemic problems with RTCs, attrition and cross-contamination, but the real problem is that there are not many researchers who can hold up those conditions imposed by RCTs.

According to Scriven, the hard-line RCT position is supported by three main pillars: (a) only the RCT design excludes all alternative causes, (b) it’s the only design that supports the true meaning of causation, (c) no other design (quasi-experimental) does it as well. Scriven criticized these three pillars with raising the problem of RCT design such as contamination or attrition problem, counterfactual mistake and not admitting the quasi-experimental designs. Especially, he emphasized the importance of quasi-experimental designs, not as just the alternatives for RCTs, but better ways in a certain situation. In addition, he raised the unethical issues related random assignment.

Tom Cook is one of the leader early leaders of the RCT gangs, but also critical user of it with open-mind. He was not obsessed by RCT method, and accepted the alternatives which have the ability to recreate experimental results such as regression discontinuity, a geographically local intact comparison group that is matched on pretest scores, and the research conducted when the process of assignment to treatment is perfectly known. However, he contradicted Scriven’s criticism for the RCTs problem with reasons as below.

First, Tom Cook argued that random assignment cannot by itself guarantee a secure causal inference, but with accumulated studies, they can find the way how to minimize the problem such as attrition or contamination. Second, he said that the counterfactual problem of overdetermination is solved by the use of statistical matching of groups rather than individual matching. About the third pillar mentioned by Scriven, Tom Cook argued that even the most fervent advocates of random assignment would not deny that you can make causal inferences without random assignment, and admitted that random assignment is limited to the situation where, given a specific cause, we want to know its effect. But, he said that in social ameliorative field like education, the interest is more often in going from causes to effects rather than from observing an effect to identifying its cause. Regarding to the ethical issues, Tom Cook put in this way. “If we concern about the ethical issues derived from random assignment, how about carrying on for years with practice based on poor evidence. Is it ethical?” He argued that the malpractice could be more harmful compared to the randomized assignment (What a pragmatist approach!!).

Lastly, he said that the RTC “crusade” in education referred by Scriven does not make sense. He argued that unlike the “crusade” in Middle-Ages, the advocates of randomized assignment have attempted for a long time to institutionalize randomized experiments as the method of choice in education research, but they did not have any intentions to dominate a source of all education research funds. According to his opinion, since over the last 30 years, quantitative studies like randomized experiments were downgraded systematically, they just hoped to set the historical records correctly by overemphasizing it.

Scriven challenged against this idea “little dictatorship is needed to fix up the bad times in the past”, because RCT has almost won the research fight. He kept saying that certainly there are somewhere left not conquered by RCT, but they should stop now before the situation worse. He underlined that because RCT is not the only way and there exist many different ways to achieve their research goals, researchers should be eligible for getting funded regardless of whatever methods they chose.

In his final remark, Tom Cook expressed similar opinion with Scriven by referring this praise “questions come first, and method choice second.” He pointed out that they both agree with that “randomized assignment is not the only way, but the best single method for causal inference in a certain condition.” He said that probably their dispute would be about how much better RCTs are when compared to other alternatives for causal inference.

In his final mark, Michael Scriven also approved that his main target is not Tom Cook, a critical user of RCT, but the RCT super-enthusiast. And he finally mentioned that even though he still cannot agree with the idea about controlled and controllable random assignment problems, it is true that their final goal is to pursue a middle ground: don’t fund bad studies, of whatever kind, and do fund good and efficient use of research resources, whatever the design they use.

After reading their debate, I still cannot find the answer to my question, but at least, I can feel it is not a big deal. “Question comes first, method choice second, there are so many ways to get there including experiment and quasi-experiment. And we should scrutinize all the ways with critical perspectives before we choose one.” That’s the main lesson that I got from this debate.

Leave a Reply

Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.