CAUSAL EFFECTS IN NON-EXPERIMENTAL STUDIES: Sensitivity to Selection on Observables 2

Posted by Kathryn Schwartz on May 12, 2014

This paper demonstrates how to estimate the treatment impact in an observational study using propensity score methods.
These methods are assessed using Lalonde’s influential re-creation of a non-experimental setting. Our results show that the estimates of the training effect are close to the benchmark experimental estimate, and are robust to the specification of the comparison group and the functional form used to estimate the propensity score. A researcher using our method would arrive at estimates of the treatment impact ranging from $1,473 to $1,774, very close to the benchmark unbiased estimate from the experiment of $1,794. Click Here
Furthermore, our methods succeed for a transparent reason: they use only the subset of the comparison group that is comparable to the treatment group, and discard the complement. Although Lalonde attempts to follow this strategy in his construction of other comparison groups, his method relies on an informal selection among the pre-intervention variables. Our application illustrates that even among a large set of potential comparison units, very few may be relevant. But it also illustrates that even a few comparison units can be enough to estimate the treatment impact.

The methods we suggest are not relevant in all situations: there may be important unobservable covariates, for which the propensity score method cannot account. But rather than giving up, or relying on assumptions about the unobserved variables, propensity score methods may offer both a diagnostic on the quality of the comparison group and a means to estimate the treatment impact.

Tags: , ,