Article about publishing replication studies

Came across this call for replication studies in the Public Finance Review (link below). It discusses the importance and challenges to publishing replication studies and offers guidelines/standards. Granted, the authors are referring specifically to publishing in the Public Finance Review, a journal that most of us are not likely to target, but it does contain valuable information and guidance that certainly has broader application.


Secondary data sources on the web

In class today someone asked about some of the best resources on the web for secondary data. Here are a few that come to mind.

Teaching Good Research Practice

I attended a webinar on how to teach students to document empirical research by Richard Ball and Norm Medeiros from Havorford College and hosted by the Interuniversity Consortium for Political and Social Research (ICPSR). This idea aims to counter current norms, policies and practices in teaching empirical research by having students submit all their statistical analyses with their final project. This should include all the necessary documentation to allow a third-party to replicate all statistical results, what Ball and Medeiros call “a soup-to-nuts approach”. This approach in turn enhances professional norms and practices through a trickle-up effect, students actually understand what they are doing, and students know they are being held accountable. The webinar used an example from an economics course, but it is easy to imagine the potential for social work education and research.

The slides are available on their YouTube channel. It’s worth checking out and rethinking how we can use this in our classrooms and research.

Amsterdam Manifesto on data citation and sharing

Below you can find the Amsterdam Manifesto on data citation and sharing. These principles allign well with the philsophy of replication and reproducible research that we’ve discussed so much at the brownbag.

For those who expressed concern about data sharing in social work HERE is a post by @carlystrasser who tackles many of the arguments against data sharing, including “my data is embarrasingly bad”.
Unfortunately, she does not discuss sensitive data topics with vulnerable populations.

The Amsterdam Manifesto on Data Citation Principles
We wish to promote best practices in data citation to facilitate access to data sets and to enable attribution and reward for those who publish data. Through formal data citation, the contributions to science by those that share their data will be recognized and potentially rewarded. To that end, we propose that:
1. Data should be considered citable products of research.
2. Such data should be held in persistent public repositories.
3. If a publication is based on data not included with the article, those data should be cited in the publication.
4. A data citation in a publication should resemble a bibliographic citation and be located in the publication’s reference list.
5. Such a data citation should include a unique persistent identifier (a DataCite DOI recommended, or other persistent identifiers already in use within the community).
6. The identifier should resolve to a page that either provides direct access to the data or information concerning its accessibility. Ideally, that landing page should be machine-actionable to promote interoperability of the data.
7. If the data are available in different versions, the identifier should provide a method to access the previous or related versions.
8. Data citation should facilitate attribution of credit to all contributors

This Manifesto was created during the Beyond the PDF 2 Conference in Amsterdam, 20 March 2013.
Original authors are Mercè Crosas, Todd Carpenter, David Shotton and Christine Borgman.

See more on the Amsterdam Manifesto HERE

Replication called for in psychology ‘priming’ studies

Daniel Kanemann calls for ‘priming’ studies in psychology to be replicated in NATURE.

I find it interesting that one of Kanemann’s motivations is to protect graduate students on the job market.

h/t @deankarlan

06 July update:

I see that the funder 3ie now has a replication program to fund replications of past programs/studies. HERE.

They have an informative bibliography and FAQ there.

And here is Stevenson and Wolfers review of replication in economics literature

For a scholar, replication offers an unappealing bet. Heads, you discover that the findings of an original study are largely correct, and no journal will publish your paper because there’s no interest in learning that something is still true. Tails, you find a serious flaw, but your results still probably won’t be published and you’ve earned enemies who may try to land some reputational punches against you.


Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.