CAUSAL EFFECTS IN NON-EXPERIMENTAL STUDIES: Sensitivity to Selection on Observables

Posted by Kathryn Schwartz on May 10, 2014
CAUSAL EFFECTS IN NON-EXPERIMENTAL STUDIES:

One important assumption underlying propensity score methods is that all of the variables that affect assignment to treatment and are correlated with the potential outcomes, Yi1 and Y0, are observed. This assumption led us to restrict Lalonde’s data to the subset for which two (rather than one) years of pre-intervention earnings data is available. In Table 5 (Panel B), we consider how our estimators would fare in the absence of two years of pre-intervention earnings data by reestimating the treatment impact without making use of earnings in 1974. For PSID-1, the stratification estimators yield less reliable estimates than in Table 3, ranging from -$1,023 to $1,727 as compared with $1,473 to $1,691, although the matching estimator is more robust. In contrast, even though the estimates from the CPS are farther from the experimental benchmark than those in Table 3 ($861 to $1941 compared with $1,582 to $1,774), they are still more concentrated around the experimental estimates than the regression estimates in Panel B of Table 2.

This illustrates that the results are sensitive to the set of pre-intervention variables used. For training programs, a sufficiently lengthy pre-intervention earnings history clearly is important. Table 5 also demonstrates the value of using multiple comparison groups. Even if we did not know the experimental estimate, in looking at Table 5 we would be concerned that the variables that we observe (assuming that earnings in 1974 are not observed) do not control fully for the differences between the treatment and comparison groups, because of variation in the estimates between the CPS and PSID. If all relevant variables are observed, then the estimates from both groups should be similar (as they are in Table 3). When an experimental benchmark is not available, multiple comparison groups are valuable because they can suggest the existence of important unobservables (see Rosenbaum 1987, which develops this idea in more detail). Click Here

NSW (1)Unadjusted

1,794

(633)

(2)Adjusteda

1,672

(638)

(3) (4) (5)Unadjusted Adjusteda

A. Dropping higher-order terms

(6)Obs.d (7)Unadjusted (8)Adjustedb
PSID-1: -15,205 218 294 1,608 1,254 1,255 1,691 1,054
Spec. 1 (1154) (866) (1389) (1571) (1616) (2209) (831)
PSID-1: -15,205 105 539 1,524 1,775 1,533 2,281 2,291
Spec. 2 (1154) (863) (1344) (1527) (1538) (1732) (796)
PSID-1: -15,205 105 1,185 1,237 1,155 1,373 1140 855
Spec. 3 (1154) (863) (1233) (1144) (1280) (1720) (906)
CPS-1: -8,498 738 1,117 1,713 1,774 4,117 1,582 1,616
Spec. 4 (712) (547) (747) (1115) (1152) (1069) (751)
CPS-1: -8,498 684 1,248 1,452 1,454 6,365 835 904
Spec. 5 (712) (546) (731) (632) (2713) (1007) (769)
CPS-1: -8,498 684 1,241 1,299 1,095 6,017 1,103 1,471
Spec. 6 (712) (546) (671) (547)B. Dropping RE74 (925) (877) (787)
PSID-1: -15,205 -265 -697 -869 -1,023 1,284 1,727 1,340
Spec. 7 (1154) (880) (1279) (1410) (1493) (1447) (845)
PSID-2: -3,647 297 521 405 304 356 530 276
Spec. 8 (959) (1004) (1154) (1472) (1495) (1848) (902)
PSID-3: 1,069 243 1,195 482 -53 248 87 11
Spec. 8 (899) (1100) (1261) (1449) (1493) (1508) (938)
CPS-1: -8,498 525 1,181 1,234 1,347 4,558 1,402 861
Spec. 9 (712) (557) (698) (695) (683) (1067) (786)
CPS-2: -3,822 371 482 1,473 1,588 1,222 1,941 1,668
Spec. 9 (670) (662) (731) (1313) (1309) (1500) (755)
CPS-3: -635 844 722 1,348 1,262 504 1,097 1,120
Spec. 9 (657) (807) (942) (1601) (1600) (1366) (783)

 

Tags: , ,