Workforce Development in the United States:

Changing Public and Private Roles and Program Effectiveness

Carolyn J. Heinrich
Vanderbilt University

June, 2016

Prepared for the book:

Labor Activation in a Time of High Unemployment:

The U.S. Department of Labor (USDOL) was established in 1913, although an active role

for the department in labor activation and training did not begin until the 1930s, when President

Roosevelt appointed Frances Perkins to his Cabinet to develop plans to alleviate unemployment

and spur recovery from the Great Depression. Prior to the Manpower Development and Training

Act (MDTA) of 1962 that officially established the federal public employment and training

system in the U.S., programs including the Civilian Conservation Corps, the Works Progress

Administration (WPA), the Public Works Administration and the National Youth Administration

were viewed as temporary solutions to workforce challenges, with unemployment the primary

concern. The Comprehensive Employment and Training Act (CETA) of 1973, which succeeded

MDTA, extended the WPA approach in that it sought to provide work for the long-term

unemployed and those with low incomes, as well as summer jobs for low-income youth. CETA

also aimed to cede more control to state governments in administering employment and training

programs, a trend that was advanced under the Job Training Partnership Act (JTPA) of 1982,

reflecting the Reagan era of “New Federalism.”

Compared to its predecessors, and consistent with the Reagan administration agenda to

reduce the role of government, JTPA was distinguished by a more decentralized administrative

structure that enlarged the role of the private sector in arranging for and delivering publicly-

funded employment and training services. JTPA also substantially diminished the public sector’s

part in directly creating employment opportunities by eliminating the public service employment

and participant stipend components of CETA. In addition, JTPA introduced a performance

standards system to measure program outcomes across states and local service delivery areas,

with the objectives of increasing local-level accountability and encouraging more efficient

1
program management. JTPA was also the first federal program in the U.S. to adopt an

outcomes-based performance management system that set national standards for program

performance and attached incentives and consequences to the results reported by states. These

reforms to U.S. labor administration reflected the overarching goal of the Reagan Administration

to lessen the federal government’s fiscal responsibility and managerial role in addressing social

problems.

The Workforce Investment Act of 1998 (WIA) replaced JTPA beginning in July 2000,

but continued the governing philosophy that centralized authority should be limited so that state

and local agencies can adapt employment and training programs to their own political and

economic contexts. This has also contributed to considerable variation across states and local

areas in how workforce development programs are organized and how and what services are

delivered. Under JTPA, the non-overlapping program jurisdictions were known as Service

Delivery Areas, but some job-training agencies were organized as public entities at the state,

county or municipal government level, while others were formed as private, not-for-profit or for-

profit organizations. In the WIA program, states were required to establish a State Workforce

Investment Board, including the Governor, members of the state legislature, and representatives

of business, labor, educational entities, economic development agencies and community-based

organizations, and the local jurisdictions (Workforce Investment Areas) were directed and

supervised by a board of representatives from business, labor, the community and local elected

officials. The boards play a central role in determining who is served, the types of services made

available, and who should provide the services (within the limitations of the statute).

Currently, one of the key mechanisms for local level planning and coordination is the

one-stop career center that every local workforce investment area is required to operate. One-

2
stop career centers (also known as American Job Centers or AJCs) are intended to coordinate

and co-locate more than a dozen federally funded education, workforce and worker support

programs to offer a basic menu of services that can help to meet the needs of a diverse set of

individuals seeking assistance with training and/or employment. The primary services provided

include vocational training, on-the-job-training (OJT), basic or remedial education, job search

assistance, work experience, and other services such as counseling and assessment, job-readiness

activities and case management. The WIA (Gold Standard) experimental evaluation, for which

data collection is ongoing, found considerable variation across the AJCs in how job seekers

access these services, but also increasing collaboration among the co-located partners to support

client participation (Social Policy Research Associates, 2016). Under the new Workforce

Innovation and Opportunity Act (WIOA), which became effective July, 2015, expectations for

coordination have been further elevated, as partnerships and co-location with other programs

such as the Wagner-Peyser Employment Services, Temporary Assistance for Needy Families

(TANF) and U.S. Department of Education programs are now mandatory rather than encouraged

(Civic Impulse., 2016).

With new legislation, the first in 15 years, and new legislative and program priorities

under WIOA, now is an appropriate time to reflect more broadly on the public sector’s role in

workforce development and labor activation, and to also consider what the research base to date

suggests about how effective our programs have been in supporting our overarching workforce

development goals of helping job seekers access the employment, education, training, and

support services they need to succeed in the labor market and for employers to compete

effectively in the global economy.

3
 Role of the Public Sector in Supporting Human Capital Development

Since their origins (as described above), employment and training efforts in the U.S. have

relied on some combination of public and private sector resources, although private sector

employers account for the lion’s share of workforce development activity and continue to dwarf

public sector investments. U.S. government spending on workforce development has averaged

less than 0.5 percent of Gross Domestic Product (GDP) in recent decades (and closer to 0.2

percent in recent years), shares that are well below most western European countries, such as

Denmark, Belgium, the Netherlands and Finland that have devoted up to 10 times greater shares

of GDP to labor market policy expenditures (e.g., Auer et al. 2008; Martin, 2014). In terms of the

incidence of employer-sponsored training, the U.S. is in about the middle of the distribution

(relative to other countries), although U.S. employers do less well in particular categories, such

as occupational training for younger workers (Lerman et al. 2016). These patterns raise questions

about whether current levels of U.S. workforce investment are adequate, as well as the extent to

which public workforce investments should complement or undergird employer-led training, or

whether they should be targeted toward individuals or the types of workforce investments where

private sector efforts are lacking.

Economic theory on returns to training suggests that workers who acquire more training,

if it in turn increases their individual productivity, should realize returns in the form of higher

wages (Mincer 1974). Employers that provide training specific to their firm’s needs are likely to

increase a worker's wage to reduce turnover and are less likely to provide training when

competition for employees is higher among firms (Rzepka and Tamm, 2016), with the

implication that returns to (firm-specific) employer-provided training are more likely to be

privately realized. This suggests an unpersuasive case for public subsidization of this type of

4
training. However, by the same line of reasoning, employers may underinvest in more general or

portable types of training that would be more likely to generate external benefits, not only for

other employers, but also for economic growth and efficiency (that improve societal well-being),

if training increases worker productivity. Using firm-level data from Ireland that distinguished

between general and specific training, Barrett and O’Connell (2001) found that general training

has a statistically significant, positive effect on productivity growth that persists when

controlling for a range of factors (e.g., firm size, initial level of human capital, corporate re-

structuring, etc.), but they did not find comparable effects for specific training. The American

Society for Training and Development estimated that about three-quarters of employer spending

on training is for formal internal workplace learning (Rivera and Paradise 2006), and Lerman et

al.’s (2004) analysis likewise found that employer training efforts disproportionately favor

better-educated and skilled workers. In addition, Bassanini et al. (2007) similarly found that in

Europe (as in the U.S.), the provision of training by private firms increases with educational

attainment and the skill-intensity of occupations. In sum, privately funded training is more often

likely to be narrowly targeted both in terms of who gets training (the higher skilled in more

competitive markets) and in the type of training offered (i.e., firm-specific, internally oriented).

Both theoretical and empirical analyses (Gersbach and Schmutzler 2006; Holzer et al.

2011; Holzer 2013) suggest that as labor markets become more globally competitive and

integrated, an even smaller segment of the workforce will have sufficiently high skills and

productivity levels to induce additional investments by their employers. This, in turn, suggests a

worsening inequality between higher- and lesser-skilled workers in access to private sector

training opportunities and wage increases. Gersbach and Schmutzler attribute at least part of the

decline in apprenticeships in Germany and some of the widespread decline in the provision of

5
general training to product market integration (associated with globalization) that reduces

training investments made by firms. Citing his own work with colleagues (2011) and that of

Acemoglu and Autor (2012), which points to “a growing complementarity over time between

personal skills and firm wage premia, and strong labor market demand relative to supply for

workers with these skills,” Holzer (2013, 6) questions whether the U.S. would be competing

more effectively in the global labor market for “good jobs” if its public policies were more

effectual in increasing human capital. There appears to be a growing consensus in labor market

analyses that we are under-supplying workers with the required skills and credentials to satisfy

labor demand for well-paying middle- and high-skill jobs, despite the apparently attractive labor

market incentives for young and working-age individuals to make these investments (Fouarge et

al., 2013; Goldin and Katz 2008; Autor and Handel 2009).

This raises another question about the role of the public sector in workforce development:

if young, working-age people are not responding to labor market incentives to pursue

postsecondary education and training opportunities that would prepare them for well-paying jobs

that are in high demand, is there a role for the public sector to address this disconnect or the

market failings that contribute to it (e.g., imperfect or asymmetric information, labor markets that

are not perfectly competitive, externalities, etc.)? In the U.S. and in Europe, some suggest that

we need to increase and improve opportunities for career and technical education before young

people leave high school (Biavaschi 2013; Rumberger 2011), and debate in the U.S. is ongoing

about whether an over-emphasis on college preparation in high schools has steered students

away from technical course-taking (or squeezed them out of high school course offerings),

resulting in an inadequate pipeline of students trained for or on a trajectory to work in well-

paying, middle-skill jobs. A growing body of research points to the importance of offering young

6
people education and training opportunities that they see as relevant to their future job prospects

and that provide this career context for learning, particularly for low-income or disadvantaged

youth who might otherwise drop out of high school (Center for Education Policy 2012; Holzer

2013; Lerman 2007). New program models are also being tested for disconnected youth (i.e.,

those not working or in education or training), such as Project Rise (in New York, New Jersey

and Kansas City) that are aimed at increasing their educational attainment and employment

opportunities through services that combine classroom education, internships, case management

and community projects (Manno et al., 2015). Although the evidence base of proven youth

programs is still relatively thin, we are not lacking for promising interventions (based on initial

outcomes or impacts) that engage youth in career and vocational education that is targeted

toward economically growing sectors (Heinrich and Holzer, 2011); however, funding for these

programs is not keeping pace with the level of program need among youth (Field 2011).

Both public and private sector investments in training will likely be constrained by tight

budgets for some time to come, making it increasingly important that spending is well-targeted

in terms of how and for whom it can be most effective, as well as in consideration of where skills

shortages lie. The existing evidence base on the effectiveness of workforce development

programs, however, is limited in many ways. With the possible exception of the WIA Gold

Standard impact evaluation (in progress), even the most comprehensive evaluations have been

restricted in terms of the coverage and representativeness of the programs they have evaluated

and the outcomes they have examined. Still, there are some consistent findings across rigorous

research efforts that offer some basic guidance for workforce development policy, as well as

research that illuminates where findings are mixed or suggest promising interventions that would

benefit from further study (and/or where better data are needed for evaluation).

7
 The Evidence Base on Training Program Effectiveness and its Limitations

The literature on employment and training program impacts is vast and spans

approximately four decades of research and evaluations. Fortunately, in recent years, scholars

have undertaken efforts to synthesize this literature, including meta-analyses of active labor

market policy evaluations (Card et al. 2010; Haelermans and Borghans, 2012), training programs

worldwide (Fares and Puerto 2009) and U.S. government-sponsored training programs and

welfare-to-work programs (Greenberg et al. 2003, 2005), as well as other summaries of the

empirical evidence (Decker 2011; Fares and Puerto 2009; Brunello et al. 2007; Greenberg et al.

2006; Heckman, LaLonde, and Smith 1999). The meta-analysis by David Card, Jochen Kluve

and Andrea Weber includes 97 studies of active labor market policies from 26 countries between

1995 and 2007 and considers short-term, medium-term and long-term impact estimates, as well

as the effectiveness of different program types. Most of the studies that they analyze are

nonexperimental in design, although they find, along with Greenberg, Michalopoulos and Robins

(2006), that experimental and nonexperimental evaluations of government-funded training

programs (or active labor market policies) yield similar results and conclusions about their

effectiveness. Of course, that does not imply that these studies are without limitations regarding

what conclusions we might draw or what generalizations we might make from them.

Table 1 presents a summary of the evidence base that focuses on more recent and/or

comprehensive studies and existing reviews (e.g., syntheses and meta-analyses) of the workforce

development/active labor market policies and programs. This summary is not intended to be all-

inclusive of the large and continually expanding body of research and individual studies on these

programs, but rather to focus on some of the latest evidence and on sources of cumulative

knowledge and findings to date. The WIA Gold Standard impact evaluation is not included in

8
this table because as of 2016, the study findings on program impacts have not yet been formally

released. The table provides basic information on the studies included, the types of programs

and policies they examined, and findings on program and policy outcomes. Other findings and

limitations of the studies are also indicated in the summary table.

Perhaps what stands out most in the summary table is how limited the evidence base for

workforce development/active labor market programs and policies is in terms of the

measurement of outcomes, program costs and coverage, and longer-term impacts. If numeric

estimates of program impacts are reported, they are almost exclusively focused on average

employment and/or earnings or wages. Only a few studies monetize other impacts, such as

government savings or reductions in welfare and crime, and there is little discussion or

measurement of skills, credentials or qualifications gained through training. Of 345 studies of

training programs in 90 countries reviewed in Fares and Puerto’s (2009) meta-analysis, only 16

attempted some accounting of costs and benefits, and obtaining accurate data on even direct

program costs is a frequently acknowledged limitation in this body of research. The studies also

vary in the length of time that they are able to follow program participants after receipt of

services, and those studies that have followed outcomes over a longer period provide ample

evidence that program impacts may change (grow or decay) over time. At the same time, one can

make some broad generalizations across the study findings that hold in a wide range of study

samples and even different country contexts.

Evidence on impacts of different types of training

One of the most commonly provided types of training across countries is vocational

training, which the majority of studies find to be effective in increasing adult earnings.

However, the research base consistently reports that there are initial “lock-in” effects of

9
classroom or vocational training, with early negative impacts that turn positive and increase over

time (Andersson et al. 2012; Caliendo et al. 2011; Card et al. 2010; Decker 2011; Heinrich et al.

2008; Schochet et al. 2006). These studies suggest that vocational training program impacts

typically turn positive by about 18–24 months after program entry and then grow for at least

several years. Comparing vocational training effect sizes across studies is somewhat more

challenging, because of the variation in how impacts are reported. In fact, the meta-analysis by

Card et al. (2010) was only able to quantitatively compare training effect sizes for a single

outcome (employment) and a subset of studies reviewed, so the authors opted instead to

summarize the research findings according to whether program impact estimates were

significantly positive, significantly negative, or null or inconclusive.

Looking at the studies with results for adults, the bulk of average impact estimates come

from U.S. program evaluations, which typically estimate training impacts on earnings per

quarter. Across these studies, the estimates for JTPA and WIA training programs are within a

fairly narrow range of $320–$887 per quarter for participants, particularly given the varying

study samples and methodologies (Andersson et al. 2012; Bloom et al. 2003; Decker 2011;

Heinrich et al. 2008; Hollenbeck et al. 2005). Some of these studies, along with others, translate

earning effects into percentage terms, with estimated effects (earnings increases) of training

programs in the U.S. and abroad ranging from 5 to 26% of average earnings (Bloom et al. 2003,

1997; Caliendo et al. 2011; Decker 2011; Fares and Puerto 2009; Greenberg et al. 2005;

Haelermans and Borghans 2012; Heinrich et al. 2008; Hollenbeck et al. 2005). Estimated effects

of training on the probability employment are also positive and statistically significant across a

majority of studies (and in different countries). These estimates of employment increases range

from about 5 to 29 percentage points (measured monthly or quarterly), with some differences

10
observed between women and men, and by specific training type and time following program

entry (Caliendo et al. 2011; Card et al. 2010; Decker 2011; Fares and Puerto 2009; Heinrich et al.

2008; Hollenbeck et al. 2005).

Studies that examine program effects by training type also consistently find that job

search assistance is more likely to generate positive impacts in the short run that then fade in

magnitude with time, in contrast to the impacts of vocational training that take a longer time to

mature but then turn positive and grow larger (as noted above). Unfortunately, a number of

studies group together job search assistance and on-the-job training or wage subsidies in

analyzing their effectiveness, which makes it challenging to identify their differential impacts or

effect sizes, to the extent that they vary. Caliendo et al. (2011) find wages subsidies to regular

employment to be the most effective component of active labor market policies, with 20

percentage point impacts on monthly employment (vs. 10 percentage points for vocational

training). Similarly, Haelermans and Borghans (2012) compare the average number of hours in

on-the-job training with the average number of hours spent on schooling and conclude that the

returns to on-the-job training are substantially higher (yielding a wage increase of 30 percent,

compared to an 8 percent average return to education). Haelermans and Borghans also report that

there is substantial heterogeneity in the wage effects of different training courses (identified via

the Q-statistic in their fixed effects model), but their study does not shed any light on what types

of courses are more effective. In their meta-analysis, Fares and Puerto (2009) distinguish

between programs that combine classroom and workplace training and those that offer only one

type of training or the other, and they conclude that impacts are larger and positive for those

programs that offer these training services together. However, their study appears to be

exceptional in its attempt to consider the combined effects of participating in multiple types of

11
training; it is unclear if existing data are not sufficiently fine-grained to make these distinctions

at the micro or participant level, or if the research approaches to estimating program impacts

have been too coarse.

In the U.S., there are several new federally-funded subsidized employment demonstration

programs that are being experimentally evaluated and offer an opportunity to identify the

impacts of on-the-job training (as a training tool) or subsidized employment that is intended to

provide work-based income support. These programs include the Subsidized and Transitional

Employment Demonstration (STED) and the Enhanced Transitional Jobs Demonstration (ETJD),

which are designed to provide evidence-based “enhancements” or supports (e.g., case

management and supportive services such as transportation) to increase their effectiveness

compared to earlier OJT or subsidized employment program models. These programs are

currently focusing on two target populations among job seekers: low-income non-custodial

parents and ex-offenders. Previous studies of subsidized, public sector employment programs;

from the early JTPA study results to more recent summaries of evaluation evidence, find that

programs offering subsidized public jobs are the least likely to yield positive impacts on

employment and earnings (Bloom et al. 2003; Caliendo et al. 2011; Card et al. 2010). This may

explain in part why even with extraordinarily high unemployment rates for working-age adults

that occurred in the recent Great Recession, there was little discussion or public calls for bringing

back programs such as those under the CETA program that offered “make-work” public jobs.

This new “generation” of subsidized employment initiatives works closely with area workforce

boards and aims to place participants in private sector (“competitive”) employment.

12
Variation in program effectiveness

The evidence on the extent to which training impacts vary by subgroups is largely mixed.

For example, some studies find differences in training impacts for men and women, with women

generally realizing larger gains from vocational training (Bloom et al. 2003; Decker et al. 2011;

Heinrich et al. 2008), while other studies find no gender differences in impacts (Andersson et al.

2012; Card et al. 2010). Alternatively, the evidence base is fairly consistent in finding

considerably smaller impacts on employment and little or no impacts on earnings of training

programs targeted toward dislocated workers in the U.S. (Andersson et al. 2012; Decker et al.

2011; Heinrich et al. 2008; Hollenbeck and Huang 2006; Social Policy Research Associates

2013). In general, it appears that the “lock-in effects” (or foregone earnings associated with

training) are more costly for dislocated workers, who tend to have stronger (higher) earnings

histories than the average training program recipient. The most recent study of U.S. trade

adjustment assistance programs suggests that dislocated worker trainees fare better after training

when they find employment in their training field and when they receive a degree or certificate

through training, particularly women who receive training in health care professional fields

(Social Policy Research Associates 2013).

For youth, the evidence base on training impacts is probably more mixed than the

conventional wisdom might suggest. On average, most studies find that the impacts of youth

training programs are smaller than those for adults. However, possibly even more so than adult

programs, they are diverse in design and service mix, which contributes to considerable variation

in their effectiveness, and new models for serving youth are emerging that have built on the

existing evidence base to improve program design and youth engagement. Caliendo et al. (2011)

report positive impacts of German active labor market policies for youth, both shorter-term (for

13
wage subsidies) and longer-term (for vocational training), with the exception of job creation

programs and preparatory training programs (that youth enter before taking apprenticeships).

Using World Bank data to look across “country clusters,” Biavaschi et al.’s (2013) research

examines the various forms of youth vocational education and training (both at school and on the

job) and argues for the importance of combining both elements (in what they describe as a “dual

apprenticeship”) to better link youth competencies with employers’ needs. Although they

emphasize that their analysis is not causal, they generally find that countries with substantial dual

apprenticeship systems (e.g., Austria, Denmark, Germany and Switzerland, which also reach

larger fractions of their young people) have more successful youth transitions from school to

work, lower youth unemployment rates and fewer disconnections or repeated unemployment

spells among their youth. Their findings are echoed by those Eichhorst et al. (2015), who in a

similar cross-country analysis find that a dual system which combines school-based education

with firm-based training is the most effective. And in Fares and Puerto’s (2009) meta-analysis,

they likewise showed that combining vocational education and on-the-job training yields larger

impacts, although they reported that youth training program impacts were largest in the Latin

American countries, where they observed increases in employment of 5–21 percent and increases

in earnings of 10–35 percent.

Indeed, there has been considerable innovation over time in youth training efforts, and

the knowledge base on what “works” for youth has likewise been steadily growing, with a wave

of new experimental study results expected to be released in the U.S. in coming years (Bloom

2009; Bowles and Brand 2009; Heinrich and Holzer 2011; Research and Evaluation Conference

on Self-Sufficiency, 2016). There appears to be a clear trend toward combining

classroom/vocational training with career or on-the-job training for youth, with some promising

14
new approaches to implementing these youth interventions. Some of the innovative program

features include: creating smaller “learning communities” to foster a more personalized learning

environment and provide more customized instructional support and academic advising; work-

based learning components, such as curriculums tightly linked with work/skills training and

partnerships with employers to facilitate job-shadowing, on-the-job training, and internships;

career fairs, guest speakers and career guidance; college-readiness counseling and pre-college

course-taking, along with financial incentives for youth to reach educational or career

milestones, and strong peer supports (Heinrich and Holzer 2011; Research and Evaluation

Conference on Self-Sufficiency, 2016). Career Academies and Year Up are two such programs

that incorporate a number of these features, and for which there is now experimental evidence of

their positive impacts on youth and young adults. One year after participation in Year Up, the

annual earnings for those who participated were on average 30 percent higher than earnings for

control group members. And participants in Career Academies realized an 11 percent increase in

average annual earnings ($2,203 per year) that was sustained over an eight-year follow-up period

(Kemple and Willner 2008). Career Academies participants were also 23 percent more likely to

be living independently with a child and partner, although the experimental evaluation did not

find effects on attainment of postsecondary credentials, standardized test scores, receipt of public

assistance, drug use, criminal activity, or health insurance coverage.

With funding from the Department of Health and Human Services, the potential for

scaling up the Year Up program is currently being experimentally evaluated in eight program

sites that are serving 18-24 year olds with skills training, work experience (through corporate

internships) and extensive supportive services. Other experimental evaluations of youth

employment and training programs that are underway include Project Rise, which serves 18- to

15
24-year-olds who lack a high school diploma and are out of school and work. Project Rise

participants engage in a year-long sequence of activities (case management, classroom

education, paid internships) that are intended to prepare them for unsubsidized employment,

postsecondary education, or both. Another example is the Promotor Pathway program, which

implements a long-term approach to intensive case management for disconnected and

disengaged youth to help them access resources and achieve educational, employment, and

healthy living goals. The 18-month experimental evaluation results suggest that participants in

Promotor Pathways are experiencing improvements on a range of outcomes (compared to the

control group), including educational attainment, reduced births, residential stability and reduced

risk-taking behaviors (Theodos et al., 2016).

Like the Career Academies evaluation, the experimental study of the U.S National Job

Corps program (shown in table 1) also stands out from other youth and adult program

evaluations in terms of its scope (the broad range of program impacts examined) and its longer-

term follow-up (Schochet et al. 2006). Academic and vocational instruction and job training are

the core components of the Job Corps program, which aims to help youth attain certificates or

credentials and to then place them in jobs that match well with the skills they have acquired. Job

Corps is also distinctive, however, in its residential component that is intended in part to remove

disadvantaged youth from risky contexts that might otherwise interfere with their progression

through the program. Schochet et al. find a number of number of positive impacts of the Job

Corps program, including an increase in the receipt of GED and vocational certificates by more

than 20 percentage points each; positive earnings impacts beginning in the third year after

random assignment that yielded an average earnings gain of about $1,150 or 12 percent by the

fourth year; an increased likelihood of having a job with fringe benefits; significantly reduced

16
welfare receipt (by $640 on average) and lower arrest, conviction and incarceration rates and

reduced criminal activity for all youth subgroups. Still, the estimated impacts on earnings

endured through the fifth to tenth years only for 20- to 24-year-olds (who tended to participate in

Job Corps longer), and because of the Job Corps program’s substantially higher cost per

participant, the study authors ultimately concluded that despite the multiple dimensions of

positive program impacts, the program did not pass a cost-benefit test when the longer-term

effects were taken into account.

The results of the longer-term National Job Corps program evaluation probably served to

reinforce a generally negative view of youth training program impacts. However, so few studies

undertake a longer-term impact and cost-benefit analysis as did Schochet et al., whether for adult

or youth programs, that it is difficult to examine the Job Corps program on equal footing with

other programs. For example, some limited information suggests that the per-student cost of

Career Academies is probably considerably lower than Job Corps, but Career Academies did not

generate the broader impacts of Job Corps (e.g., reducing crime and reliance on public welfare),

and no formal analysis of its net benefits to participants and society has yet been performed. In

addition, in estimating the impacts of training interventions for dislocated workers, no

consideration to has been given to potential benefits (or reduced negative impacts) on other

family members (e.g., particularly children), despite fairly robust evidence (discussed earlier in

this paper) that parental job loss has significant negative impacts on children’s educational

outcomes and even their later life earnings.

More generally, as noted above, we have done a poor job of measuring both the costs and

benefits of our active labor market policies and workforce development programs and in

attempting to assess rates of return. Researchers contributing to this body of evidence lament the

17
idiosyncratic definitions of training that they encounter across surveys and country data; the lack

of data on the duration of training, skills acquired and completion of qualifications or credentials,

and productivity gains; and the even scarcer data on costs (Bassanini et al. 2007; Card et al.

2010; Fares and Puerto 2009; Haelermans and Borghans 2012; Hendra et al. 2012). Card et al.

concluded that a cost-benefit analysis or calculation of social returns to training was not feasible

from the 97 studies in their meta-analysis, and Fares and Puerto found only 16 of 345 studies in

their research base made an attempt to conduct a cost-benefit analysis. And even in the

evaluation of a single national program (WIA), costs incurred per WIA participant were not

available across the 12 state programs assessed, and Heinrich et al. (2008) relied instead on

available data from published sources to estimate average per capita direct expenditures.

Although the limited availability and quality of data will continue to challenge our efforts

to comprehensively measure the costs and benefits of workforce development programs/active

labor market policies everywhere, the recent efforts to conduct more experimental evaluations

that also include assessments of program implementation and costs is encouraging. And while

resource constraints and policymakers’ demands for timely information will inevitably limit the

timeframes over which we measure program impacts, both researchers and government officials

should continue to push for and rustle up resources for longer-term evaluations that are essential

for better targeting workforce development resources toward the right interventions and at the

best time in the trajectory of an individual’s development. Overall, the most recent evidence on

training program effectiveness is generally positive, showing impacts on employment

probabilities and earnings capacity that are realized by most sub-groups (see again table 1 and

also Lechner and Mellya 2007), and this is based on a fairly narrow approach to the

measurement of program benefits. In addition, there is still considerable debate in the literature

18
as to how much heterogeneity in effects exists across different subgroups of participants that

could be exploited to improve overall program effectiveness (Huber et al. 2011; Rinne et al.

2011). The fact that these findings suggest that there are most likely positive returns to the

government’s role in workforce development brings into question again our meager spending in

this area relative to our developed country peers.

Targeting Employment and Training Services

One of the major changes in the U.S. WIA program from its predecessor, the JTPA

program, was in its targeting of services. The JTPA legislation specifically required 90 percent of

all program enrollees to be disadvantaged, as well as minimum levels of service to particular

hard-to-serve subgroups, including youth, high school dropouts, and welfare recipients. In the

WIA (and now WIOA) programs, however, the core services—intake and assessment, job search

assistance and labor market information—are made available to the general public, with no

eligibility requirements. Those who are unemployed and unable to obtain employment through

core services can access intensive or training services, which include comprehensive assessment

and case management and vocational and on-the-job training. As a result of these program

changes, the share of low-income individuals receiving workforce development services was

reduced by one-third under WIA, and the length of time they spend in training (as well as

expenditures per trainee) has also declined significantly (Osterman 2007). And outside of the

Jobs Corps program, federally funded efforts to train youth primarily focus on summer

employment. Are current U.S. workforce development programs structured and operated to

adequately reach and engage those who are least likely to get access to training without public

support?

19
 Research has fairly clearly shown that the lower-skilled and less advantaged are least

likely to be offered training by their employers, while employers acknowledge that an important

reason they have been slow to increase hiring is due to their inability to find workers with the

requisite skills (Besharov and Call 2013). Besharov and Call suggest that employers

increasingly see it as the responsibility of the worker (or prospective employees) to seek ways to

build skills on their own. If the evidence base on training effectiveness suggested that

disadvantaged workers were less likely to gain from receipt of training, then one might make the

case that there may be no under-provision of training, and the market or employers (along with

the individual workers themselves) have sorted out where the investments in human capital are

most likely to be productive. Albeit mixed overall, there is rather considerable support in the

evidence base (discussed above) that shows that vocational and on-the-job training can generate

significant impacts on individual earnings and employment among the disadvantaged, which

presumably reflect gains in productivity to the employers of these workers as well.

These findings suggest a potential policy response in the form of a reallocation of federal

training resources. Under WIA, we spent more on the comparatively poor-performing WIA and

trade adjustment assistance programs for dislocated workers than we did on training for

disadvantaged adults. In fact, WIOA now grants states more flexibility to transfer funding

between Adult and Dislocated Worker programs, which could enable states to spend more on the

Adult programs that produce higher returns on average and to support a more equitable

distribution of training opportunities in the economy (as disadvantaged adults are less likely to be

offered training on the job from employers). The plight of dislocated workers gets more media

and political attention, in part because plant closings and downsizings are more visible

manifestations of employment loss (than those of discouraged workers or the long-term

20
unemployed), and also because these workers’ earnings losses tend to be large. An analysis by

LaLonde and Sullivan (2010) suggests that some of the same vocational and technical training

strategies that work well for unemployed adults could be more effective for dislocated workers,

but for both of these groups, we have not targeted these resources well within the programs. One

possible policy response would be for the USDOL to consider folding dislocated workers and

funding for this program into an adult training program that more explicitly targets

disadvantaged workers, with dislocated workers being one subgroup of disadvantaged workers.

An encouraging finding of the WIA Gold Standard impact evaluation (implementation study) is

that at the AJCs, local level program staff had already informally started merging resources from

the Adult and Dislocated Worker programs by serving adult and dislocated worker clients

according to the needs they presented rather than the funding source (Social Policy Research

Associates, 2016). LaLonde and Sullivan also offer a number of strategies for improving

program effectiveness, such as tying aid for community college course-taking to past

performance (e.g., completion rates) for both the individual and the educational institution, as

well as more active use of data by workforce development agencies to identify higher-value

training programs.

Expanding public and private support and program reach for youth and young adults

The cross-country comparisons referenced in this paper and made by many others

contributing to this discussion clearly show that the U.S. lags behind a number of its developed

country peers in what it spends both publicly and privately on training relative to GDP.

Bassanani et al. (2007) identified the Scandinavian countries, France and New Zealand as the

most training intensive countries, and noted that 80 percent of vocational training courses are

21
paid for or provided by employers in Europe. Is there something that we can learn from other

countries about how their public/private partnerships work to sustain higher levels of

expenditures on training, as well as to support broader program coverage, particularly for young

people and those who are least likely to access training privately?

Robert Lerman (2013; 2016) points out that the U.S. spends more heavily on education

but does far less than its OECD peers in the provision of high-quality occupational training for

young people. Indeed, the most recent European literature on training effectiveness is focused

on discussions about how to blend vocational and on-the-job training and expand partnerships

with employers in the provision of education and training, beginning at much earlier ages than

we do in the U.S. Lerman reports that apprenticeship programs in Germany, Switzerland,

Austria, Australia and even in the United Kingdom are now reaching over 50 percent of young

people, while Caliendo et al. (2011) add that dual apprenticeship programs (combining

vocational training with on-the-job training) currently account for half of all of German student

entries into vocational training each year in secondary schooling. In other words, about a quarter

of German youth are engaging in on-the-job training alongside of vocational training while

completing their secondary education. The training offered is not perceived of as lower-grade or

an inferior track, but rather is high-quality and career-focused, leading to a certification that

youth can take directly to the labor market or on to additional university-level education.

These systems of education and career preparation for youth stand in sharp contrast to

what has been described as a typical U.S. “college for all” approach to secondary schooling.

There is considerable debate currently taking place in the U.S. about whether we have moved too

far away from career and technical education, compounding the skills and labor market

disadvantage for youth who are ultimately not college bound (i.e., only about 25 percent of high-

22
school graduates attend a four-year university upon graduation). For example, the Texas

Workforce Commissioner stepped forth with employers and other community members to decry

the shortage of young people entering skilled trades due to neglect of vocational education at the

high school level. Texas subsequently passed legislation in spring 2013 to temper its restrictive,

college-preparatory curriculum and make it easier for students interested in career and technical

education to take courses that are necessary to get an industry-certified credential by the time of

graduation from high school. Although U.S. education policymaking is largely in the purview of

state and local educational agencies, Holzer and Edelman (2013) argue that it is important to

develop more systemic and comprehensive approaches for youth, so that fewer of them fall off

track. WIOA is now placing greater emphasis on work-based learning by requiring the Title I

youth program to spend at least 20 percent of the funding on work experience, and CTE

stakeholders can participate in the development of state plans to ensure that CTE is incorporated

into a state’s vision and goals for increasing workforce skills.1

At the same time, as effective as these European approaches to labor market preparation

for young people appear to be, these systems also still struggle with the least advantaged. As

Caliendo et al. (2011) point out, there is a separate preparatory system for German youth with the

lowest educational attainment before they have the opportunity to enter an apprenticeship, and it

also takes these youth more time to move from subsidized work experience into employment. In

the U.S., any discussion of separate “tracks” for K–12 students raises angst about early

“segregation” of students that might further limit their opportunities for higher education and

skills development. Instead, we have experimented with alternative program approaches to

serving our disadvantaged and vulnerable youth, both in school and out of school, many of

1
See http://www.nationalskillscoalition.org/resources/publications/file/2015-06-Aligned-by-Design-WIOA-and-
CTE.pdf.

23
which aim for early targeting to help youth stay engaged and prevent them from dropping out.

These programs are very diverse, from the comprehensive Career Academies program described

above that includes career and technical education as a core feature, to other programs that

emphasize mentoring and individualized attention, afterschool and summer school programming,

career guidance and postsecondary education, and more (Heinrich and Holzer 2011; Theodos et

al., 2016). Can the U.S. find a balance that shifts our approach closer to being more systemic and

formalized, as in the German and other European systems, while preserving flexibility for locally

innovative and adaptive strategies for youth while they are still in school?

Targeting youth program resources and keeping youth engaged is undoubtedly easier

when these efforts begin while the youth are still in school. As Holzer and Edelman (2013) point

out, once youth have “disconnected” both from school and the labor market, they are more likely

to give up on “mainstream” institutions and opportunities, and their prospects for entering the

labor market will become increasingly poor. We are also gradually coming to terms with the fact

that once they are disconnected, there is probably no way that is both cheap and effective to re-

engage these young people in education, training and the workforce. The Year Up program, for

example, asks its corporate sponsors to contribute over $23,000 to a single student’s program

costs, which include an education stipend, tuition for college credits, transportation and other

direct and indirect costs of training, job placement and support services.2 This amount is

comparable to the Job Corps program costs. While Heckman (2008) argues that the most cost-

effective way to address the challenges of these youth is to do so before they reach school age (a

now widely accepted claim), we are still a long way from having the programs and resources in

place to do that for all disadvantaged children (before they enter the school system), and we will

2
http://www.yearup.org/partners/main.php?page=federal_partners.

24
continue to need targeted interventions that address these youth needs at this sensitive period

during which they transition to adulthood and their future careers.

Public Sector Role in Seeding and Supporting Innovative Training Strategies

One of the more promising strategies for workforce development identified in recent U.S.

program evaluations are those that emerge locally and are targeted to one or more specific

sectors of the labor market in arranging education and training opportunities. These sectoral

training programs—which aim to advance basic and occupational skills of participants in sectors

with expanding labor market opportunities—are intended to respond to the needs of both

jobseekers and employers simultaneously (and thereby also reduce labor skills shortages). An

experimental evaluation of three sectoral training programs found that participants earned, on

average, 18 percent more than controls over a 24-month study period, and 29 percent more

during the second half of the period, suggesting that these impacts have the potential to be both

substantial and enduring (Maguire et al. 2010).3 The two-year impacts of the more recently

implemented WorkAdvance program, which combines a sectoral approach with career

advancement strategies, are highly encouraging, likewise showing an 18% increase in earnings

over the experimental control group, with significant increases in earnings for the long-term

employed as well (Hendra, 2016). Other experimental evaluations of sectoral training programs,

such as the Accelerating Connections to Employment (ACE) program, are currently underway

(with impact results expected in 2017-2019). Should the U.S. be doing more at state or federal

levels to support the expansion or scaling up of successful models of sectoral training?

3
The study, funded by the Charles and Stewart Mott Foundation, focused on three well established sectoral training
programs: Jewish Vocational Services (Boston), Per Scholas (the Bronx, New York City), and the Wisconsin
Regional Training Partnership (Milwaukee).

25
 A report from the National Network of Sector Partners (Mangatt 2010) estimated that

they were approximately 1,000 sectoral training partnerships operating in the U.S. several years

ago, and about half the states have explored the potential for implementing these strategies. As

these collaborations depend on relationships with employers who are willing to combine firm-

specific skills training with more general skills training in the context of a public-private

partnership, identifying and incentivizing these partnerships is likely to take time and resources.

In a number of European countries, sectoral training funds are being used to mobilize resources

and encourage public-private sharing of both the costs and responsibilities of providing

vocational education and training in promising labor market sectors. These national funds are

typically financed by a tax on wages and are explicitly intended to create a “more equal

redistribution of training opportunities among underrepresented groups” (European Centre for

the Development of Vocational Training 2008, 4). The funds are used to strengthen cooperation

between the public and private partners in a number of areas, including the identification of

employer skills needs, frameworks for specifying training qualifications and mechanisms for

skills recognition and certification, and resource mobilization at the national, local/sectoral and

firm levels.

In the U.S., sectoral training strategies first began emerging in the 1980s and 1990s, in a

kind of grass-roots response to the needs of key industry groups in varying labor market sectors

and the low-skilled individuals looking for work nearby them (King 2013). A number of sectoral

training initiatives have been funded by the $146.9 million Workforce Innovation Fund (WIF),

which was announced in June 2012 by the U.S. Secretary of Labor to support innovative service

delivery in the public workforce system. The U.S. Department of Health and Human Services

(DHHS) is also funding sectoral training program efforts, including through the Health

26
Profession Opportunity Grants (HPOG) program, which is designed to help low-income

individuals acquire the education and training needed to enter occupations in the health care field

that are in high demand. Currently, studies of the types of systems change required to implement

these programs and interim outcomes of the HPOG program have been released, suggesting

promise for sectoral program success (Loprest et al., 2015; Bernstein et al., 2016); results from

an impact evaluation of the HPOG program are expected to become available in 2017. Although

the U.S. investment in these strategies to date still pales compared to that of its European

counterparts—looking at Spain alone, it has spent over $1 billion in national and regional funds

on sectoral training initiatives (with the large majority of these funds coming from the national

level) since 1993 (European Centre for the Development of Vocational Training, 2008)—both

DHHS and USDOL have now made sectoral training strategies core program priorities and are

embarking on “second-generation” efforts to improve sectoral training program models and their

implementation..

The U.S. Department of Labor and its federal agency partners such as DHHS are also

beginning to expand the type of coordinating and directing efforts that their European

counterparts have long undertaken in sectoral training program efforts. The Europeans point to a

number of private market failures to account for their greater involvement in these initiatives

(European Centre for the Development of Vocational Training, 2008). First, they explain that

employers frequently lack adequate information on training returns, and they are prone to

worrying about the possibility of poaching or free-riding by other employers of their newly

trained workers. This contributes to employers’ inclination to invest only in firm-specific

training for their workers, and it is also most likely to be in an area of a high return for the

business and/or for employees who are already highly qualified or in leadership roles. In turn,

27
low-educated or low-skilled adults most in need of education and training may be the least aware

of its potential benefits, or may be less able to take advantage of the opportunities in the absence

of support services or flexible training arrangements. The policy levers that the European public

authorities draw on in their intermediary role to support sectoral training range from legislation

and regulation to both financial (e.g., direct subsidies, tax credits and deductions) and non-

financial measures to stimulate firm investment in workplace training (e.g., information,

advisory and referral services, qualification and certification systems, dissemination of best

practices, etc.). In fact, in some countries, public and private entities work together as “social

partners” to operate sectoral training funds, into which firms pay a certain percentage of their

payroll and from which they can have their own training efforts partially reimbursed. In the

United Kingdom (UK), a Sector Skills Development Agency, created in 2002, provides funding,

support and monitoring of a network of sector skills councils that covers approximately 85

percent of the workforce.

King (2013) suggests, however, that in the absence of a sizeable increase in funding for

sectoral training programs, it is unlikely that the U.S. will be able to take these initiatives to a

sufficiently large scale to realize their potential benefits in the coming years. At the same time,

WIOA presents new opportunities for the U.S. to substantially strengthen efforts to actively

promote sectoral training initiatives and incentivize and support (financially and non-financially)

partnerships at both federal and state levels. Section 101 of WIOA describes a number of

existing best practices that the ETA would like to see adopted or expanded under the new law,

including: career pathways (integrated with adult basic education, English as a second language,

and occupational training); industry or sector partnerships convened or implemented by local

WIBs, and an increased focus on the attainment of industry-recognized certificates and

28
credentials linked to in-demand occupations. In light of tight federal resources, one possibility

would be to leverage a federal commitment through the AJCs by encouraging (or incentivizing)

them to play an elevated coordinating and information dissemination role in support of sectoral

partnership-building (akin to the sector skills councils in the UK). While the current political

environment is not amenable to the imposition of new payroll taxes, states and localities might

explore other ways to redirect existing sources of state and local tax revenues toward support of

sectoral training efforts, much in the same way that they convince the public to offer tax breaks

to employers who are considering the creation or relocation of firms and jobs.

Public Sector Role in Improving the Evidence Base

In both the U.S. and elsewhere, our knowledge base on what works in workforce

development (or active labor market policies) is still limited in terms of its usefulness for

informing both public and private decision making about investments in training. For example,

considering what type of evidence that employers might look for to convince them to engage in

sectoral training programs, King (2013) points out that we still have no evidence (or measures)

showing whether these strategies increase worker and firm productivity (either immediately or

over time), increase efficiency or lower firm costs over time, or ultimately affect firms’ bottom

line (i.e., profits). Our evaluations rarely go beyond the worker as the unit of analysis and/or the

returns to individuals in relatively narrow terms of their employment and earnings. We have

struggled to get even basic data on costs of services for programs that we are currently operating,

which makes any type of “bottom-line” calculation difficult, whether for public entities or

private investors in training.

In some countries, such as Germany, the Netherlands and other Nordic countries,

29
comprehensive and generous access to large and informative administrative databases on active

labor market policy implementation has allowed researchers in these countries to undertake

considerably richer analyses than are typically possible with administrative data from U.S. states

or the Department of Labor (Lechner and Wunsch 2009; Smith 2011). This probably goes a long

way toward explaining the dominance of these countries in the databases of recent meta-analyses

on training effectiveness. Evaluations in the U.S. are more likely to be experimental than in

Europe (because of strong political resistance elsewhere to random assignment), yet Smith

(2012) argues that we may have become too focused on methods, to the neglect of data quality.

Whereas administrative data in Germany and some other countries are made available to

researchers cleaned and linked, if we can get data from our public training programs in the U.S.,

it is left to the researchers to identify and clean up errors and other problems. In the

nonexperimental WIA evaluation (Heinrich et al. 2008), we were only able to secure cooperation

from 12 states to obtain their administrative data, and some could only provide those data for

part of the period for which they were requested. In addition, there were numerous

inconsistencies from one state to another in how those data were recorded and managed, which

ultimately placed the burden on researchers to make assumptions about how they should be used.

Smith (2012) adds that it very inefficient for different groups of researchers to be cleaning the

same data over and over again, and because states receive federal funds, they should be obliged

to provide program data and to also support the linking of those data over time to facilitate

longer-term follow up of program outcomes.

And even though random assignment experiments are more likely to be launched in the

U.S., we take too long to get them underway. The current WIA experimental evaluation did not

get off the ground until approximately a decade after WIA first became operational, and the first

30
results are only now emerging (after we have transitioned to WIOA). If the U.S. Department of

Labor, the states and other public and private partners can work together to coordinate and

support more effective evaluation within the country, we might increasingly look in the future to

partner or cooperate with others in cross-country, comparative evaluations, which would give us

a new window into how alternative organizational, economic and political structures and

contexts mediate program effectiveness. Currently, we rely on organizations such as the privately

funded, nonprofit Institute for the Study of Labor in Germany (IZA), which provides a valuable

service in helping to support exchanges across a network of approximately 1,200 researchers in

more than 45 countries and to disseminate study findings that inform a richer, cross-national

understanding of active labor market policy implementation and program impacts. There are

numerous, currently pressing issues that would benefit from more cross-national collaboration—

such as the need to address declining rates of labor force participation among working-age

adults, the limited success of training efforts with dislocated workers, and the relationship

between training and job quality, among others—in addition to more concerted efforts to build

the evidence base around them.

In Europe, the Public Employment Service (PES) is ramping up efforts to promote cross-

national learning and collaboration through a process described as “benchlearning.”

Benchlearning aims to facilitate indicator-based performance comparisons that support mutual

learning and the identification of best practices for improving program efficiency and

effectiveness. A PES-network taskforce has been designated to further develop the concept of

benchlearning and the methods and processes of its implementation. In March 2015, a shared

definition of benchlearning was established, and work is ongoing on the details of its

implementation, with ultimate goal to support evidence-based, mutual learning across the PES in

31
Europe (European Network of Public Employment Services, 2015). Given that the USDOL and

DHHS are similarly pursuing strategies to promote evidence-based policymaking and could

potentially benefit from mutual learning opportunities with developed country peers, they might

consider working with the European PES Network to expand benchlearning to other countries.

The earlier the U.S. engages, the more likely it could have some influence in setting the

performance benchmarks by which these international comparisons of program performance will

be made.

It is also worth noting that the new WIOA program mandates implementation of new

“Pay for Success” efforts, with the objective to incentivize and develop new public-private

partnerships to address some of the most difficult programmatic challenges through the use of

payment structures that provide funds only to programs that achieve pre-determined outcomes.

The intent is to only expend tax payer dollars if the programs are getting measurable results, as

specified in the contracts between government and a service provider (or provider coalition). To

facilitate implementation of this new approach to performance-based contracting, WIOA creates

a permanent authority within the three formula funding streams (Adult, Youth, and Dislocated

Workers) to establish Pay for Performance as an eligible use of WIOA funds. Local workforce

boards can devote up to 10 percent of funding across all three formula funding streams for Pay

for Performance programming. States can also use their WIOA funds to provide technical

assistance to help local workforce areas implement Pay for Performance strategies. This will be

an important new provision of WIOA to watch, as to date, there is limited evidence of the

success of “Pay for Success” strategies themselves. In fact, this could also be an important area

for benchlearning, given that the United Kingdom is a few steps ahead in implementing Pay for

Success initiatives and has already established a “Pay for Success Learning Hub”

32
(http://www.payforsuccess.org/networks).

Finally, the National Association of State Workforce Agencies and other stakeholders of

the U.S. workforce system—who have been studying the WIOA legislation, reviewing the

details and new provisions section by section, and anticipating challenges in its

implementation—have urged the Secretary of Labor and the USDOL’s federal program partners

to ensure that workforce system partners (and the research community) have flexibility and

opportunities through state options, waivers, pilot demonstrations, and other means to learn from

state and local innovations and improve on the system design as WIOA is rolled out. They have

also asked the USDOL to continue to facilitate the development of evidence and the translation

of evidence into policy by writing policy into guidance documents rather than relying on more

formal regulations, allowing for modifications to the guidance as new knowledge is gained in the

work of state and local agencies with their system partners. In effect, flexibility is needed and

will be highly valued as federal, state and local agencies and system partners work to identify the

best approaches to and measure(s) of effectiveness in serving workers and employers and

ensuring continuous performance improvement and increased program and labor market success

in the new WIOA program.

33
 References

Acemoglu, Daron and David Autor. 2012. What Does Human Capital Do? A Review of Goldin
and Katz’ Book The Race Between Education and Technology.” Journal of Economic
Literature, 50(2), 426–63.
Andersson, Fredrik, Harry J. Holzer, Julia I. Lane, David B. Rosenblum, and Jeffrey Smith.
2012. Does Federally-Funded Job Training Work? Nonexperimental Estimates of WIA
Training Impacts Using Longitudinal Data on Workers and Firms. Working paper.
Arum, Richard and Josipa Roska. 2011. Academically Adrift: Limited Learning on College
Campuses. Chicago: University Of Chicago Press.
Auer, Peter, Umit Efendioglu, and Janine Leschke (2008). Active labour market policies around
the world: Coping with the consequences of globalization, Geneva: International Labour
Office, Second Edition.
Autor, David H., and Michael J. Handel. 2009. “Putting Tasks to the Test: Human Capital, Job
Tasks and Wages.” NBER Working Paper No. 15116. Cambridge, MA: National Bureau
of Economic Research.
Barnett, W.S. (1996). Lives in the Balance: Age 27 Benefit–Cost Analysis of the High Scope
Perry Preschool Program. High/Scope Press, Ypsilanti, MI.
Barrett, Alan and Philip J. O'Connell. 2001. Does Training Generally Work? The Returns to In-
Company Training. Industrial and Labor Relations Review 54 (3): 647–662.
Bassanini, Andrea, Alison Booth, Giorgio Brunello, Maria De Paola and Edwin Leuven. 2007.
Workplace Training in Europe. Chapter 8-13 in: Brunello, Garibaldi and Wasmer (eds.),
Education and Training in Europe, Oxford University Press.
Becker, Gary S. (1964, 1993, 3rd ed.). Human Capital: A Theoretical and Empirical
Analysis with Special Reference to Education, Chicago: The University of Chicago
Press.
Belfield, C.R., Nores, M., Barnett, W.S., Schweinhart, L. (2006). The High/Scope Perry
Preschool Program: cost–benefit analysis using data from the age-40 follow-up.
Journal of Human Resources 41 (1), 162–190.
Bernstein, Hamutal, Lauren Eyster, Jennifer Yahner, Stephanie Owen, and Pamela Loprest.
2016. Systems Change under the Health Profession Opportunity Grants (HPOG) Program
(OPRE Report # 2016-50). Washington, DC: Office of Planning, Research and
Evaluation, Administration for Children and Families, U.S. Department of Health and
Human Services.
Besharov, Douglas J. and Douglas M. Call. 2011. Labour Activation in a Time of High
Unemployment Key Developments in the OECD. Working paper.
Biavaschi, Costanza, Werner Eichhorst, Corrado Giulietti, Michael J. Kendzia, Alexander
Muravyev, Janneke Pieters, Nuría Rodríguez-Planas, Ricarda Schmidl and Klaus F.
Zimmermann. 2013. Foundations and Trends in Microeconomics, 9 (1-2), 1-157.
Blakemore, S. J., 2010. The developing social brain: implications for education. Neuron 65 (6),
744–747.
Bloom, Dan. 2009. Out-of-school youth and the transition to adulthood. Working paper, MDRC,
New York.
Bloom, Howard, Larry Orr, George Cave, Steve Bell and Fred Doolittle. (2003). The National
JTPA Study: Title IIA Impacts on Earnings and Employment at 18 Months. Bethesda,
MD: Abt Associates.

34
Bowles, Anne, and Betsy Brand. 2009. Learning around the clock: Benefits of expanded
learning opportunities for older youth. Washington, DC: American Youth Policy Forum.
Brunello, Giorgio, Pietro Garibaldi and Etienne Wasmer (eds.) 2007. Education and Training in
Europe.
Caliendo, Marco, Steffen Künn, and Ricarda Schmidl. 2011. Fighting Youth Unemployment:
The Effects of Active Labor Market Policies, IZA Discussion Paper 6222.
Card, David, Jochen Kluve, and Andrea Weber. 2010. “Active Labor Market Policy
Evaluations: A Meta-Analysis.” The Economic Journal, 120, F452-F477.
Carneiro, Pedro and James J. Heckman (2005). “Human Capital Policy,” In Benjamin M.
Friedman, Ed., Inequality in America: What Role for Human Capital Policies?
Cambridge, Massachusetts: The MIT Press, pp. 77–239.
Center for Education Policy. 2012. Student Motivation: An Overlooked Piece of School Reform.
Washington, DC: Center for Education Policy.
Chase-Lansdale, P. Lindsay and Brooks-Gunn, Jeanne. 2013. Two-Generation Programs in the
21st Century. Working paper, prepared for The Future of Children.
Chertavian, Gerald. 2012. A Year Up: How a Pioneering Program Teaches Young Adults Real
Skills for Real Jobs-With Real Success. London: Viking Penguin.
Civic Impulse. 2016. H.R. 803 — 113th Congress: Workforce Innovation and Opportunity Act.
Retrieved from https://www.govtrack.us/congress/bills/113/hr803.
Cunha, Flavio and James J. Heckman. 2006. Investing in our Young People. Working paper,
University of Chicago.
Decker, Paul. 2011. Ten Years of WIA Research. In The Workforce Investment Act:
Implementation Experiences and Evaluation Findings, edited by D. Besharov and P.
Cottingham. Kalamazoo, Michigan: W.E. Upjohn Institute, 2011.
Dougherty, Conor. 2011. Young Men Suffer Worst As Economy Staggers. The Wall Street
Journal, November 7.
Dyke, Andrew, Carolyn Heinrich, Peter R. Mueser, Kenneth R. Troske, and Kyung-Seong Jeon.
2006. “The Effects of Welfare-to-Work Program Activities on Labor Market Outcomes.”
Journal of Labor Economics 24:3 (July), 567–608.
Edelman, Peter B. and Harry J. Holzer. 2013. Connecting the Disconnected: Improving
Education and Employment Outcomes among Disadvantaged Youth. Institute for
Research on Poverty Discussion Paper No. 1412–13.
Eichhorst, Werner, Núria Rodríguez-Planas, Ricarda Schmidl, and Klaus F. Zimmermann. 2015.
A Roadmap to Vocational Education and Training Systems Around the World. Industrial
and Labor Relations Review, 68 (2), 314-337.
Enayati, Amanda. 2012. The aging brain: Why getting older just might be awesome. Special to
CNN, June 19, 2012. http://www.cnn.com/2012/06/19/health/enayati-aging-brain-
innovation.
European Centre for the Development of Vocational Training. 2008. Sectoral Training Funds in
Europe. Cedefop Panorama series; 156. Luxembourg: Office for Official Publications of
the European Communities.
European Network of Public Employment Services. 2015. Benchlearning.
http://www.pesboard.eu/EN/pesboard/Benchlearning/benchlearning_node.html.
Fares, Jean and Olga Susana Puerto. 2009. Towards Comprehensive Training. The World Bank
Social Protection and Labor Discussion Paper No. 0924.

35
Field, Kelly. 2011. President's Budget Protects Pell Grants, but Makes Deep Cuts to Career and
Technical Education. The Chronicle of Higher Education, February 14.
Fouarge, Didier, Trudie Schils and Andries de Grip. 2013. Why Do Low-Educated Workers
Invest Less in Further Training? Applied Economics, 45 (18), 2587-2601.
Garces, E., Thomas, D., Currie, J. (2002). Longer-term effects of Head Start. American
Economic Review 92 (4), 999–1012 September.
Gersbach, and Armin Schmutzler. 2006. The Effects of Globalization on Worker Training. IZA
Discussion Paper No. 2403.
Goldin, Claudia, and Lawrence F. Katz. 2008. The Race Between Education and Technology.
Cambridge, MA: Harvard University Press.
Greenberg, David H., Charles Michalopoulos, and Philip K. Robins. 2006. Do Experimental and
Nonexperimental Evaluations Give Different Answers about the Effectiveness of
Government-funded Training Programs? Journal of Policy Analysis and Management,
Vol. 25, No. 3 (Summer), pp. 523–552.
Greenberg, David H., Andreas Cebulla and Stacey Bouchet. 2005. Report on a Meta-Analysis of
Welfare-to-Work Programs. Institute for Research on Poverty Discussion Paper no.1312-
05
Greenberg, David H., Charles Michalopoulos, and Philip K. Robins. 2003. A meta-analysis of
government-sponsored training. Industrial and Labor Relations Review 57(1): 31–53.
Grogger, Jeffrey and Lynn A. Karoly. 2005. Welfare Reform: Effects of a Decade of Change.
Cambridge: Harvard University Press, 2005.
Haelermans, Carla and Lex Borghans. 2012. Wage Effects of On-the-Job Training: A Meta-
Analysis. British Journal of Industrial Relations, 50 (3), 502–528).
Haskins, Ron, Harry Holzer, and Robert Lerman. 2009. “Promoting Economic Mobility by
Increasing Postsecondary Education.” Washington, DC: Economic Mobility Project Pew
Charitable Trusts.
Heckman, James J. 2000. Policies to foster human capital. Research in Economics 54(1), 3–56.
Heckman, James J. 2005. Inequality in America: What Role for Human Capital Policies?
Focus 23(3): pp. 1–10.
Heckman, James J. 2008. The Case for Investing in Disadvantaged Young Children. In Big
Ideas for Children: Investing in our Nation’s Future. First Focus, pp. 49–58.
Heckman, James J., Moon, Seong Hyeok, Pinto, Rodrigo, Savelyev, Peter A., and Adam Yavitz.
2010. “Analyzing social experiments as implemented: A reexamination of the evidence
from the High Scope Perry Preschool Program.” Quantitative Economics, 1(1):1–46.
Heckman, James J. , Robert J. Lalonde and Jeffrey A. Smith. “The Economics and
Econometrics of Active Labor Market Programs.” In Orley Ashenfelter and David Card,
editors, Handbook of Labor Economics, Volume 3A. Amsterdam and New York:
Elsevier, 1999: 1865–2095.
Heinrich, Carolyn, J. and Harry J. Holzer. 2011. “Improving Education and Employment
for Disadvantaged Young Men: Proven and Promising Strategies.” The Annals of The
American Academy of Political and Social Science, Volume 635 (May): 163–191.
Heinrich, Carolyn .J., Mueser, Peter and Kenneth R. Troske. 2008. Workforce Investment Act
Non-Experimental Net Impact Evaluation. Final Report to U.S. Department of Labor,
December.

36
Hendra, Richard. 2016. Impacts of WorkAdvance, an Evidence-based Sectoral Training and
Career Pathways Program. Presented at the Research and Evaluation Conference on Self-
Sufficiency, June 3, Washington, DC.
Hendra, Richard, Kathryn Ray, Sandra Vegeris, Debra Hevenstone and Maria Hudson. 2011.
Employment Retention and Advancement (ERA) demonstration: Delivery, take-up, and
outcomes of in-work training support for lone parents. Department for Work and
Pensions Research Report No. 727, United Kingdom
Hollenbeck, Kevin and Wei-Jang Huang. 2006. Net Impact and Benefit-Cost Estimates of the
Workforce Development System in Washington State, Kalamazoo, Michigan: W. E.
Upjohn Institute for Employment Research, Technical Report No. TR06-020.
Hollenbeck, Kevin, Daniel G. Schroeder, Christopher T. King, and Wei-Jang Huang. 2005. Net
Impact Estimates for Services Provided through the Workforce Investment Act,
Baltimore: The Jacob France Center, ADARE Project, University of Baltimore, October.
Unpublished report prepared for USDOL/ETA.
Holzer, Harry J., Julia I. Lane, David B. Rosenblum, and Fredrik Andersson. 2011. Where Are
All the Good Jobs Going? What National and Local Job Quality and Dynamics Mean for
U.S. Workers. New York: Russell Sage Foundation.
Holzer, Harry J. 2013. Good Workers for Good Jobs: Improving Education and Workforce
Systems in the US. Institute for Research on Poverty Discussion Paper No. 1404-13.
(Published online in the IZA Journal of Labor Policy, 1:5,
http://link.springer.com/article/10.1186/2193-9004-1-5).
Hotz, V. Joseph, Guido W. Imbens and Jacob A. Klerman. 2006. Evaluating the Differential
Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the
California GAIN Program. Journal of Labor Economics 24 (July), 521–566.
Howard-Jones, P.A., E.V. Washbrook, and S. Meadows. 2012. The Timing of Educational
Investment: A Neuroscientific Perspective. Developmental Cognitive Neuroscience 2S:
S18– S29.2012
Huber, Martin, Michael Lechner, Conny Wunsch and Thomas Walter. 2011. Do German
Welfare-to-Work Programmes Reduce Welfare Dependency and Increase Employment?
German Economic Review, 12 (2), 182-204.
Huston AC, Duncan GJ, Granger R, Bos J, McLoyd V, Mistry R, Crosby D, Gibson C,
Magnuson K, Romich J, Ventura A. Child Dev. 2001. Work-based antipoverty programs
for parents can enhance the school performance and social behavior of children. Jan-
Feb;72(1):318–36.
Johnson, Mark H. 2005. Developmental Cognitive Neuroscience: An Introduction. 2nd edition,
Oxford: Blackwell.
Johnson, Sandra and Kathleen Taylor (editors). 2006. The Neuroscience of Adult Learning: New
Directions for Adult and Continuing Education. Jossey-Bass.
Kelly, Patrick J. 2010. Closing the College Attainment Gap between the U.S. and Most Educated
Countries, and the Contributions to be made by the States. National Center for Higher
Education Management Systems (NCHEMS).
Kemple, James, with Cynthia Willner. 2008. Career Academies: Long-Term Impacts on Labor
Market Outcomes, Educational Attainment, and Transitions to Adulthood. New York:
MDRC.
Kerwin and Stephens, Charles and Melvin Stephens, Jr. 2004. Job Displacement, Disability,
and Divorce. Journal of Labor Economics 22, no. 2: 489–522.

37
King, Christopher T. 2013. Sectoral Workforce and Related Strategies: What We Know…
and What We Need to Know. Working paper, Ray Marshall Center, LBJ School of
Public Affairs, University of Texas at Austin.
King, Christopher T., Tara C. Smith and Robert W. Glover. 2011. Investing in Children and
Parents: Fostering Dual-Generation Strategies in the United States.Working paper, Ray
Marshall Center, LBJ School of Public Affairs, The University of Texas at Austin.
King, Christopher T. and Carolyn, J. Heinrich. 2011. “How Effective are Workforce
Development Programs?” Working Paper, University of Texas at Austin.
Knudsen, Eric I., James J. Heckman, Judy L. Cameron, and Jack P. Shonkoff. 2006. Economic,
neurobiological, and behavioral perspectives on building America’s future workforce.
Proceedings of the National Academy of Sciences 103(27): 10155–10162.
LaLonde, Robert and Daniel Sullivan. 2010. Retraining Displaced Workers. The Hamilton
Project Policy Brief, The Brookings Institution, October.
Lechner, Michael and Conny Wunsch. 2009. Active Labor Market Policy in East Germany:
Waiting for the Economy to Take Off. Economics of Transition, 17 (4), 661-702.
Lechner, Michael and Blaise Melly. 2007. Earnings Effects of Training Programs. IZA
Discussion Paper No. 2926
Leonard, J.S. 2000. Comments on James Heckman’s Policies to Foster Human Capital. Research
in Economics 54 (1), 61–64.
Lerman, Robert I. 2016. Restoring Opportunity by Expanding Apprenticeship. Ch. 10 in The
Dynamics of Opportunity in America, Irwin Kirsch and Henry Braun (eds), Springer
International Publishing, pp 359-385.
Lerman, Robert I. 2013. Employer-Led Training in the United States. Working paper.
Lerman, Robert I., Signe-Mary McKernan, and Stephanie Riegg. 2004. “The Scope of
Employer-Provided Training in the United States, “ In Christopher J. O’Leary, Robert A.
Straits, and Stephen A. Wandner, Eds., Job Training Policy in the United States,
Kalamazoo, Michigan: W.E. Upjohn Institute for Employment Research, pp. 211–243.
Livingston, Gretchen and D’Vera Cohn. 2012. U.S. Birth Rate Falls to a Record Low; Decline
Is Greatest Among Immigrants. November 29.
Loprest, Pamela, Julie Strawn, and Elizabeth Giardino. 2015. Health Profession Opportunity
Grants: Summary of Interim Outcomes, OPRE Report # 2015-68. Washington, DC:
Office of Planning, Research and Evaluation, Administration for Children and Families,
U.S. Department of Health and Human Services.
Lynch, Lisa M. 1992. Private-Sector Training and the Earnings of Young Workers. The
American Economic Review, Vol. 82, No. 1: 299–312.
Maestas, Nicole and Julie Zissimopoulos. 2010. How Longer Work Lives Ease the Crunch of
Population Aging. Journal of Economic Perspectives 24(1): 139–160.
Magnuson, Katherine. 2007. Investing in the Adult Workforce: An Opportunity to Improve
Children’s Life Chances, prepared for the Annie E. Casey Foundation Initiative on
Investing in Workforce Development, March
http://www.aecf.org/news/fes/dec2008/pdf/Magnuson.pdf.
Maguire, Sheila, Joshua Freely, Carol Clymer, Maureen Conway and Deena Schwartz. 2010.
Tuning In to Local Labor Markets: Findings from the Sectoral Employment Impact
Study, Philadelphia: Public/Private Ventures.
Mangatt, Ravinder. 2010. Sector Snapshot: A Profile of Sector Initiatives. Oakland, CA:

38
 National Network of Sector Partnerships, Insight Center for Community Economic
Development. http://www.insightcced.org/uploads/publications/wd/Sector--‐
Snapshots.pdf.
Manno, Michelle S., Edith Yang and Michael Bangser. 2015. Engaging Disconnected Young
People in Education and Work: Findings from the Project Rise Implementation
Evaluation. MDRC, October.
Martin, John P. 2014. Activation and Active Labour Market Policies in OECD Countries:
Stylized Facts and Evidence on their Effectiveness. IZA Discussion Paper No. 84, June.
Martin, John P. 2000. “What Works Among Active Labour Market Policies: Evidence from
OECD Countries’ Experiences.” OECD Economic Studies No. 30.
Mincer, Jacob.1974. Schooling, Experience, and Earnings. New York: Columbia University
Press.
Oreopolous, Phillip, Marianne Page, and Ann Huff Stevens. 2008. The Intergenerational Effects
of Worker Displacement. Journal of Labor Economics 26, no. 3 (2008): 455–483.
Osterman, Paul (2007). “Employment and Training Policies: New Directions for Less-Skilled
Adults,” In Harry J. Holzer and Demetra Smith Nightingale, Eds., Reshaping the
American Workforce in a Changing Economy, Washington, D.C.: The Urban Institute
Press, pp. 119–154.
Rege, Mari, Kjetil Telle and Mark Votruba. 2011. Parental Job Loss and Children’s School
Performance,” Review of Economic Studies 78: 1462–1489.
Research and Evaluation Conference on Self-Sufficiency. 2016. Connecting Young People to
Employment: Promising New Evidence, June 2, Washington, DC.
Rzepka, Sylvi and Marcus Tamm. 2016. Local Employer Competition and Training of Workers.
Applied Economics, 48 (35), 3307-3321.
Rinne, Ulf, Marc Schneider and Arne Uhlendorff. 2011. Do the skilled and prime-aged
unemployed benefit more from training? Effect heterogeneity of public training
programmes in Germany. Applied Economics, 2011, 43 (25), 3465–3494.
Rivera, Ray J. and Andrew Paradise (2006). 2006 State of the Industry in Leading Enterprises:
ASTD’s Annual Review of Trends in Workplace Learning and Performance, Alexandria,
VA: American Society for Training and Development.
Rolnick, A., Grunewald, R., 2003. Early Childhood Development: Economic Development
with a High Public Return. Tech. Rep. Federal Reserve Bank of Minneapolis,
Minneapolis, MN.
Rosenbaum, James E., Regina Deil-Amen and Ann E. Person. 2006. After Admission: From
College Access to College Success. New York: Russell Sage Foundation.
Ruhm, Christopher J. 1991. Are Workers Permanently Scarred by Job Displacements? American
Economic Review 81, no. 1: 319–324.
Rumberger, Russell W. 2011. Dropping Out: Why Students Drop Out of High School and What
Can be Done About It. Cambridge: Harvard University Press.
Sala, Hector and José I. Silva. 2013. Labor Productivity and Vocational Training: Evidence from
Europe. Journal of Productivity Analysis, 40 (1), 31-41.
Schneider, Mark and Lu Michelle Yin. 2012. Completion matters: the high cost of low
community college graduation rates. American Enterprise Institute, April 03, 2012.
Downloaded at: http://www.aei.org/outlook/education/higher-education/community-
colleges/completion-matters-the-high-cost-of-community-college-graduation-rates/.

39
Schochet, Peter, John Burghardt, and Sheena McConnell. 2006. National Job Corps Study and
Longer-Term Follow-Up Study: Impact and Benefit-Cost Findings Using Survey and
Summary Earnings Records Data, Final Report. Washington, DC: Mathematica Policy
Research.
Shonkoff, Jack P., Andrew S. Garner, Committee on Psychosocial Aspects of Child and Family
Health; Committee on Early Childhood, Adoption, and Dependent Care; Section on
Developmental and Behavioral Pediatrics. 2012. Technical Report: The Lifelong Effects
of Early Childhood Adversity and Toxic Stress, Pediatrics 129, no. 1: e232–46.
Smith, Jeffrey. 2012. World Class Active Labor Market Programs: Necessary Conditions.
Presentation at “Getting America Back to Work: Can Training Programs Do the Job?”
American Enterprise Institute, September 7.
Smith, Jeffrey. 2011. What Can the ESF Learn from US Evaluations of Active Labor Market
Programs? Working paper, University of Michigan.
Social Policy Research Associates. 2016. Lessons Learned from Examining the Services
Provided by the WIA Adult and Dislocated Workers Programs. Presented at the Research
and Evaluation Conference on Self-Sufficiency, June 3, Washington, DC.
Social Policy Research Associates. 2013. The Evaluation of the Trade Adjustment Assistance
Program: A Synthesis of Major Findings. Report downloaded from:
http://www.spra.com/admin/PDFFile/taa%20synthesis%20report%20-
%20revised%202013-1-9.pdf.
Stevens, Ann Huff and Schaller, Jessamyn. 2011. Short-run effects of parental job loss on
children's academic achievement. Economics of Education Review 30, no. 2: 289–299.
Taylor, Kathleen, Catherine Marienau and Morris Fiddler. 2000. Developing Adult Learners:
Strategies for Teachers and Trainers. Jossey-Bass.
Theodos, Brett, Michael R. Pergamit, Alexandra Derian, Sara Edelstein and Allison Stolte. 2016.
Solutions for Youth: An Evaluation of the Latin American Youth Center’s Promotor
Pathway Program. Final Report, The Urban Institute, April.
Toossi, Mitra, "Projections of the labor force to 2050: a visual essay," Monthly Labor Review,
October 2012.
U.S. Government Accountability Office. 1993. Dislocated Workers: Trade Adjustment
Assistance Flawed. GAO/T-HRD-94-4.

40
Table 1: Summary of findings from evidence base on workforce development program/active labor market policy effectiveness

Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Andersson, Data on Workforce Investment Act Earnings differentials tend to be negative during first Authors suggest that
Holzer, Lane, (WIA) participants (WIASRD data) several quarters after WIA registration for training their findings imply that
Rosenblum and are linked to data on workers, recipients; earnings impacts become positive around job training efforts
Smith, 2012, employers and employment the 6th quarter and grow larger over the next several should consider the jobs
Does Federally outcomes from the Longitudinal quarters, peaking at approximately $400-500 per and firms for which
Funded Job Employer Household Dynamics quarter; estimated annual impacts for adults are $1250-
workers are being
Training Work? (LEHD) program for two states; 1700; results are less favorable for dislocated workers
Nonexperimental workers who received training are (peak lower in one state and do not turn positive over trained (e.g., akin to
Estimates of matched to workers who only 12 quarters in the other state) sectoral approaches) if
WIA Training received other (core or intensive) Training appears to increase the probability of we are to increase the
Impacts Using services at One-Stop Centers and switching industries over time and is associated with effectiveness of training
Longitudinal inverse propensity score weighting some measures of firm quality (i.e., may help workers
Data on Workers is used to estimate impacts; gain employment in higher-paying firms and
and Firms Objective to measure a wider range industries)
of impacts on worker outcomes with Estimated impacts do not differ by gender
richer controls
Bassanini et al., Use large cross-country datasets Scandinavian countries, France and New Zealand Documenting cross-
2007, Education available for OECD countries to identified as the most training intensive countries country variation in
and Training in examine education and training in (participation rates above 45%, more than 30 hours per training is difficult due
Europe, Europe, theoretically and employee); US participation rates estimated at 41.4% to idiosyncratic
empirically: i) OECD aggregate and 17.9 hours per employee; 80% of vocational definitions of training in
training data; ii) Continuing training courses paid for or provided by employers, yet
different surveys and
Vocational Training Survey there are few studies on the impact of training on
(CVTS); iii) International productivity (due to lack of data on productivity); rates country data
Adult Literacy Survey (IALS); and of return estimates are even scarcer because data on
iv) European Community cost are even more difficult to find than data on output
Household Panel (ECHP) It is difficult to make a strong case for under-provision
of workplace training; more research and information
needed on externalities and costs, and more
methodological checks on existing estimates

41
Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Caliendo, Kunn German active labor market policies Main outcome: probability of being in regular Dual apprenticeship
& Schmidl, for youth; administrative data for employment; also look at participation in higher program accounts for
2011, Fighting youth (age 25 or younger) entering education half of all vocational
Youth unemployment in 2002 (n=51,019) Except for job creation and preparatory training, training entries each
Unemployment: and followed until 2008; quasi- programs improve probability of regular year (in secondary
The Effects of experimental methods applying employment—initial lock-in phase, with impacts
schooling); preparatory
Active Labor inverse probability weighting to 7 stabilizing at around 2 years after entry; 5 to 20
Market Policies programs: job search & assessment, percentage point increase in monthly employment from system for low
short-term training (max=8 weeks), third year on (varying by program & region); wage education attainment
wage subsidies for regular subsidies to regular employment most effective (20 youths; low-education
employment, job creation, long-term percentage point impact); long-term training impacts youth most vulnerable—
training (max=approx 1 year), around 10 percentage points (severe lock-in effects); need more time to turn
preparatory training (max=1 year) job creation consistently negative effects subsidized work
Probability to participate in unsubsidized education: experience into
positive increase in education probabilities of about 10 employment; by sample
percentage points through longer-term training, and design, majority in job
professional qualifications increase by 20%; search or short-term
preparatory programs do prepare youth for entering
training
apprenticeships; no effects for employment programs
Card, Kluve & Meta-analysis of 97 studies (199 Subsidized public sector programs have least favorable 70% of impact estimates
Weber, 2010, estimates) from 26 countries, 1995 outcomes; job search assistance has positive shorter- from programs targeting
Active Labor and 2007; classified impact term impacts; classroom training more positive over the registered
Market Policy estimates as significantly positive, medium-term (short-term impact estimates– measuring unemployed; in Anglo
Evaluations: A significantly negative, or effects approximately one year after program countries, 15% are from
Meta-analysis” insignificantly different from zero; completion – and medium-term for approximately 2
unemployment
ordered probit regression with years after completion available for about ½ the
controls for program type and sample; longer-term 3-year impacts for ¼ of sample); insurance recipients;
sample and study characteristics to more favorable distribution of outcomes (% cost-benefit analysis or
estimate effects on employment, significantly positive) over the longer-term; country calculation of social
wages, unemployment duration, differences are small after controlling for program type returns not feasible
future unemployment No differential effects for men vs. women
Median short-term effect size for probability of
employment (when available)=.21; median medium-
term effect size on probability of employment=.29

42
Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Decker, 2011, Review of studies on the JTPA: 15% earnings increase for women, 8% increase JTPA evaluation was
“Ten Years of implementation and impacts of for men, and net benefits per enrollee of $763/quarter experimental but WIA
WIA Research” Workforce Investment Act (WIA) for women & $781/quarter for men; OJT/JSA impacts evaluations were
programs, as well as pre- and post- higher for women and larger long-run earnings effects nonexperimental;
1995 evidence (MDTA, CETA & (over $5,000 on average for women & men) potential for selection
JTPA) WIA: larger estimated effects than JTPA on earnings;
bias remains a concern
Heinrich et al. (2008) estimates of $320-692 per
quarter for 4 years after program entry and higher with program impact
employment (5-13% per quarter); Hollenbeck et al. estimates; study samples
(2005) earnings impacts higher starting at program exit are not nationally
($773-887 per quarter over 8 quarters) and employment representative
effects of 10.6% for women & 6.2% for men); impacts
of training increase over time
JSA effects more immediate but short-lived
Trade adjustment assistance and dislocated worker
programs: a number of studies find small and/or
statistically insignificant effects; differing estimation
approaches suggest forgone earnings costs are high
during program participation
Fares & Puerto, Meta-analysis framework to review 41% of 345 interventions found to have positive Report increasing
2009 “Towards findings from 345 studies of training effects; 18% negative or no effects; 34% insufficient convergence toward
Comprehensive programs in 90 countries (controls evidence; only 16 studies include cost-benefit analyses comprehensive active
Training” for country characteristics), Interaction of in-classroom + workplace training labor market programs;
distinguishing in-classroom training increases positive impacts better evidence was not
(37% of studies), workplace training Youth programs in LAC effective in increasing
generated until early
(15%), classroom+workplace (19%), employment (by 5-21%) and earnings (by 10-35%),
classroom+workplace+supplemental although overall, impacts of programs targeting youth 1990s (63% of studies in
services (29%); 61% were publicly have significantly lower impacts (30% lower) than sample 1990 or later);
financed training programs those for adults little discussion of
Training programs more effective in low- and low- outcomes
middle income countries

43
Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Greenberg, Data from 31 random assignment Mandatory programs: job search more effective; Sample is from welfare-
Cebulla & evaluations of welfare-to-work impacts positive for 5-7 years but declining in to-work programs and
Bouchet, 2005 programs (27 mandatory, 4 magnitude after 2-3 years; more effective for less includes over 90%
“Report on a voluntary); measures of impacts on advantaged (without recent employment and longer- single parent families;
Meta-Analysis earnings, % in employment, welfare term welfare receipt); net benefits are small (societal study also examined
of Welfare-to- received & % receiving welfare (up net benefits of about $500 and $400 in govt savings per
child outcomes
Work Programs” to 20 quarters after random treatment member)
assignment) For voluntary programs, more expensive programs (emotional &
produce larger impacts behavioral)—small,
Program participants earn about 10% more than the mixed effects found
control group, but the effect fades (as does the
employment effect); welfare receipt is reduced
Labor market controls suggests programs are more
effective when demand for labor is greater
Haelermans & Meta-analysis based on 71 Main finding: average wage effect of on-the-job Too few studies
Borghans, 2012, estimates of returns to on-the-job training is 2.6%, which is larger than the average return measure the duration of
Wage Effects of training from 38 studies published to education (reported by Ashenfelter et al., 1999); training, so the authors
On-the-Job between 1981 and 2010; only using estimation techniques that correct for selectivity measured training as a
Training: studies that computed the effect of bias, the age until which an average training course is dummy variable;
A Meta-Analysis on-the-job training on wages were profitable is 55 years;
methodology and data
included Substantial heterogeneity in wage effects of training
courses is also found quality play a
Comparing the average number of hours spent on on- major role in
the-job training with the average number of hours spent determining the return
on schooling gives a wage increase of 30% for on-the- to on-the-job training
job training, compared with 8% for the return to
schooling

44
Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Heinrich, Administrative data from 12 states In almost all states, Adult program impacts are Costs incurred in the
Mueser and used with propensity score matching positive—earnings benefits are smaller in the first 4-6 WIA program were not
Troske, 2008, methods to evaluate program effects quarters than after 2-3 years; average increment in available; using
Workforce on average earnings and earnings for women is nearly $2400 per year, about available data from
Investment Act employment for ~160,000 WIA 26% of average earnings, and for men it is nearly published sources,
Non- participants up to four years $1700, about 15% of average earnings; program
average per capita direct
experimental Net following program entry in the participation increases employment in a given quarter
Impact period July 2003-June 2005 (Adult for women by about 7 percentage points, and for men expenditures were
Evaluation (with and Dislocated Worker programs); by about 6 percentage points estimated to be in the
IMPAQ comparison group members drawn Increments in annual earnings for dislocated workers range of $2400-$2700,
International) from those who filed are much smaller than for the Adult program, just over with higher costs for
Unemployment Insurance benefit $500 for women and less than $150 for men (less than Dislocated Workers
claims or who participated in U.S. 3 percent of average earnings); employment increases ($2800-$3200)
Employment Service program are greater at 4-5 percentage point increments (a 7-8%
increase in employment proportions)
Adult program benefits estimated to exceed costs for
men and women if earnings impacts continue for 2-3
years
Hendra, Ray, Employment Retention and Examined course-taking (types) and the completion of Data suggest that not all
Vegeris, Advancement program designed to qualifications or credentials; ERA increased the of the training was
Hevenstone & encourage human capital likelihood of course-taking and the probability of motivated by the ERA
Hudson, 2011, development; personal adviser and combining work and training, but there is no evidence financial incentives
Employment financial support for training among yet of an effect of this increased training on
Retention and low-wage workers and financial qualifications; it also did not affect total time spent in
Advancement incentives (bonuses) for completing training, but it did increase enrollment in courses
(ERA) training and working full time; relevant to specific occupations
demonstration: targeted lone parents and long-term Outcomes from training were only analyzed
Delivery, take- unemployed in UK qualitatively in this report; 5-year impact evaluation
up, and Randomized controlled trial with findings were expected in 2011, but no publication is
outcomes of in- outcomes measured 12 months and evident yet
work training 24 months after random assignment;
support for lone sample sizes of approx. 2,293 and
parents 1,248

45
Study Sample/methods Outcomes by different types of programs and/or Other findings and
country limitations
Schochet, Random assignment experimental Job Corps increased education and job training Average program
Burghardt and study of eligible applicants from received both inside and outside the program by ~1,000 length=8 months, ~ ¼
McConnell, 1994-96, using four years of follow- hours; 89% received vocational training (ave. of 1,140 participated for over a
2006, National up survey data and 10 years of hours of academic and vocational instruction= about year, and 28 percent for
Job Corps Study administrative data one year of high school classroom instruction); Job less than 3 months
and Longer-term Key questions: Does Job Corps Corps substantially increased the receipt of GED and
49% completed a
Follow-up Study increase educational attainment and vocational certificates by more than 20 percentage
literacy, reduce criminal behavior points each; no effects on college attendance or vocational trade or GED
and the receipt of welfare benefits, completion; participants’ functional literacy improved (were enrolled for about
and improve postprogram Job Corps generated positive earnings impacts 11 months on average)
employment and earnings? Do beginning in 3rd year after random assignment;
impacts differ by subgroups and impacts persisted through end of 4-year follow-up
center characteristics? Do program period; in year 4,
benefits exceed costs? earnings gain was about $1,150, or 12% (gains were
Research sample includes 11,313 smaller in administrative records data); decomposition
youths (6,828 program group and analysis suggested 2/3 of earnings impact was due to
4,485 control group members) who the impact on hours worked and 1/3 due to impact on
completed a 48-month interview earnings per hour; employed program participants
(response rate =81.5% for the slightly more likely to hold jobs that offered fringe
program group and 77.4% for the benefits
control group) Estimated impacts in years 5-10 for full sample all near
zero; 20- to 24-year-olds had earnings gains in years 5
to 10 (remained in Job Corps longer)
Job Corps significantly reduced welfare receipt (by
$640) and the arrest rate (by 16% or about 5 percentage
points); similar reductions found for conviction and
incarceration rates; reductions in criminal activity were
found across all youth subgroups
Job Corps costs exceed benefits to society by about
$10,300 per participant (benefits from increased
lifetime earnings=$1,119, reduced use of other
programs and services=$2,186 and reduced
crime=$1,240)

46