CAUSAL EFFECTS IN NON-EXPERIMENTAL STUDIES: LALONDE’S RESULTS 3

Posted by Kathryn Schwartz on April 22, 2014
CAUSAL EFFECTS IN NON-EXPERIMENTAL STUDIES:

Lalonde estimates linear regression, fixed-effect, and latent variable selection models of the treatment impact. Since our analysis focuses on the importance of pre-intervention variables, we consider primarily the first of these. Non-experimental estimates of the treatment effect are based on the two distinct comparison groups used by Lalonde (1986), the Panel Study of Income Dynamics (PSID-1) and Westat’s Matched Current Population Survey-Social Security Administration File (CPS-1). Lalonde also considers subsets of these two comparison groups, PSID2-3 and CPS2-3.

Table 1 presents the pre-intervention characteristics of the comparison groups. It is evident that both PSID-1 and CPS-1 differ dramatically from the treatment group, especially in terms of age, marital status, ethnicity, and pre-intervention earnings (all the mean differences are statistically significant). In order to bridge the gap between the treatment and the comparison groups in terms of pre-intervention characteristics, Lalonde extracts subsets from PSID-1 and CPS-1 (PSID-2 and -3, and CPS-2 and -3) which resemble the treatment group in terms of single pre-intervention characteristics (such as age or employment status; see Table 1, notes). But as the table indicates, the subsets still remain substantially different from the treatment group (the mean differences in age, ethnicity, marital status, and earnings are smaller, but remain statistically significant). Link

ControlSample No of Obs. Age Educ Black Hisp Nodegree Married RE74US$ RE75US$
NSW/Lalonde:aTreated 297 24.63 10.38 0.80 0.09 0.73 0.17 3,571
Control 425 24.45 10.19 0.80 0.11 0.81 0.16 3,672
RE74 subset:bTreated 185 25.81 10.35 0.84 0.059 0.71 0.19 2,096 1,532
Control 260 25.05 10.09 0.83 0.1 0.83 0.15 2,107 1,267
Comparisongroups:0PSID-1 2,490 34.85 12.11 0.25 0.032 0.31 0.87 19,429 19,063
PSID-2 253 36.10 10.77 0.39 0.067 0.49 0.74 11,027 7,569
PSID-3 128 38.25 10.30 0.45 0.18 0.51 0.70 5,566 2,611
CPS-1 15,992 33.22 12.02 0.07 0.07 0.29 0.71 14,016 13,650
CPS-2 2,369 28.25 11.24 0.11 0.08 0.45 0.46 8,728 7,397
CPS-3 429 28.03 10.23 0.21 0.14 0.60 0.51 5,619 2,467

Table 2 (Panel A) replicates Lalonde’s results using his original data and non-experimental comparison groups (the results are identical to those presented in his paper, with the exceptions noted in the footnote of Table 2). Table 2 (Panel B) applies Lalonde’s estimators to our reduced experimental sample and the same comparison units. Comparing the two panels, we note that the treatment effect, as estimated from the randomized experiment, is higher in Panel B ($1,794 compared with $886). This is due to the cohort phenomenon –individuals with a later month of assignment seem to have benefitted more from the program. Otherwise, the results are qualitatively similar. The simple difference in means, reported in column (1), yields negative treatment effects for the CPS and PSID comparison groups in both panels (except PSID-3).

Tags: , ,