Updated reference list on external validity

My review of the external validity literature is slowly working its way through the peer review process. Parts were presented from 2011 onwards, but it was first published as a full working paper here and then updated for the Annual Bank Conference in Development Economics in 2014. A short version of one key contribution from that work has been published here.

Since these pieces, however, the reference list has been expanded in two important ways.

First, I became aware of a number of references and parallel literatures outside of economics that had either been missed in the original review, or published subsequent to the first version. Notably: in biostatistics (Elizabeth Stuart and co-authors), educational statistics (Elizabeth Tipton) and causal graphs (Elias Bareinboim and Judea Pearl).

Second, feedback through peer review questioned the omission of structural contributions to the topic – suggesting that this favoured the ‘design-based’ literature most closely associated with randomised control trials (RCTs). That was certainly not the intention. The rationale of the original review was to focus on the problem of external validity within the theoretical framework used by most RCT studies, in order to clearly delineate structuralist critiques from more fundamental external validity challenges.

I still think that it is absolutely critical to emphasise this distinction. However, there are contributions from the structural literature that propose something of a middle ground. Notably, work by Heckman, Vytlacil and co-authors argues for the merits of using the theoretical framework of Marginal Treatment Effects (MTEs). And one interesting recent, empirical contribution (by Amanda Kowalski) which has done so is forthcoming in the Journal of Economic Perspectives. Given this, I have added a number of references from that literature and expanded the review to cover this middle-ground between structural and design-based contributions.

While the paper proceeds through the publication process, I thought it would be useful to post the most recently submitted (May 2016) version of the reference list for those who may be interested. It can be found here.