The Impacts of Regular Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High School Graduation
Final Report
Prepared by:
Neil S. Seftor
Arif Mamun
Allen Schirm
Mathematica Policy Research, Inc.
Princeton, N.J.
Prepared for:
U.S. Department of Education
Office of Planning, Evaluation and Policy Development
Policy and Program Studies Service
2009
This report was prepared for the U.S. Department of Education under Contract No. EA97030001. The project monitor was Margaret Cahalan in the Policy and Program Studies Service. The views expressed herein are those of the contractor. The content of this report does not necessarily reflect the views or policies of the U.S. Department of Education, nor does the mention of commercial products or organizations imply endorsement by the U.S. government. The inclusion of such information is for the reader's convenience and is not intended to endorse any views expressed, or products, programs, models or services offered.
U.S. Department of Education
Margaret Spellings
Secretary
Office of Planning, Evaluation and Policy Development
Bill Evers
Assistant Secretary
Policy and Program Studies Service
Alan L. Ginsburg
Director
Program and Analytic Studies Division
David Goodwin
Director
January 2009
______
This report is in the public domain. Authorization to produce the report in whole or in part is granted. Although permission to reprint this publication is not necessary, the citation should be: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, The Impacts of Regular Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High School Graduation, Washington, D.C., 2009.
ACKNOWLEDGMENTS
This report on the effects of Upward Bound on students’ postsecondary outcomes reflects the contributions of many individuals. The authors would like to thank David Goodwin and Margaret Cahalan of the Policy and Program Studies Service at the U.S. Department of Education. We are grateful to David for his continued support, substantive guidance, and encouragement throughout the study and to Maggie for her many very valuable comments and other guidance. Marisol Cunnington and Jay Noell of the U.S. Department of Education also provided helpful suggestions. In addition, the report has been improved by comments from three anonymous external reviewers selected by the Institute of Education Sciences.
David Myers, who directed the national evaluation of Upward Bound for many years, played a key role in shaping the study design and led previous impact analyses. Mary Moore, a principal investigator for the national evaluation, was also instrumental in shaping the study design. Mark Dynarski and Peter Schochet provided insightful suggestions throughout the analysis and drafts of the report.
The authors would also like to thank the individuals who contributed to the data collection and analysis and the production of the report. David DesRoches oversaw the data collection, and Zhanyun Zhao created the survey weights. Karin Zeller constructed the analysis variables and computed the impact estimates, and Mary Grider provided technical assistance with data and programming issues. Jennifer Baskwell produced the document.
iv
CONTENTS
Chapter Page
EXECUTIVE SUMMARY xiii
I INTRODUCTION 1
A. Context and Purpose of Upward Bound 1
B. Context for Interpreting Program Effects 7
1. Value-added of Upward Bound 8
2. Upward Bound Applicants 11
3. Effects for Students Who Participated During the Mid-1990s 12
4. Variation in Exposure to Upward Bound 14
C. Previous Findings 16
II RESEARCH DESIGN AND ANALYTIC ISSUES 19
A. Research Design 19
1. Selection of Upward Bound Projects and Random Assignment 19
2. Outcome Measures 22
3. Data Sources 24
4. Construction of the Outcome Measures 26
B. analytic issues 28
1. Estimation of Program Impacts 28
2. Subgroup Analysis 31
3. Use of Weights to Account for Sample Design and Survey Nonresponse 35
4. Potential Threats to the Study Design 37
III THE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES 39
A. The Effect of the Opportunity to Participate in Upward Bound (ITT) 40
1. The Effect of Upward Bound on Postsecondary Enrollment and College Selectivity 43
2. The Effect of Upward Bound on Financial Aid Application and Pell Grant Receipt 43
3. The Effect of Upward Bound on Postsecondary Completion 44
Chapter Page
b. The Effect of Participation in Upward Bound (CACE) 44
C. Summary of Sensitivity Analyses 44
1. Sensitivity Analyses Pertaining to the Measurement of Outcomes 47
2. Sensitivity Analyses Pertaining to Sample Weighting 48
IV THE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES
FOR SELECTED SUBGROUPS 55
A. The Effect of the Opportunity to Participate in Upward
Bound for Selected Subgroups (ITT) 56
1. Grade at Application to Upward Bound 56
2. Applicants’ Educational Expectations at the Time of Application
to Upward Bound 57
3. Level of Ninth-Grade Mathematics Class 60
4. Grade Point Average in Ninth Grade 63
B. The Effect of Participation in Upward Bound on Selected Subgroups (CACE) 65
V THE EFFECT OF ADDITIONAL UPWARD BOUND PARTICIPATION ON
POSTSECONDARY OUTCOMES 67
A. Research Questions 67
B. Research Methods 68
C. The Effect of Additional Upward Bound Participation 71
1. Postsecondary Enrollment 71
2. Financial Aid 73
3. Postsecondary Completion 74
D. Interpretation of the Findings 74
References 77
APPENDIX A: Sample Design, Unit Nonresponse, and Weights A.1
APPENDIX B: DATA COLLECTION and outcome measures B.1
Chapter Page
APPENDIX C: sensitivity analyses pertaining to the
measurement of outcomes C.1
APPENDIX D: SAMPLE SIZES AND WEIGHTED STANDARD
DEVIATIONS FOR ALL OUTCOME VARIABLES D.1
APPENDIX E: ESTIMATION OF IMPACTS AND STANDARD ERRORS E.1
APPENDIX F: METHODS USED TO ESTIMATE THE EFFECTS
OF ADDITIONAL UPWARD BOUND PARTICIPATION F.1
APPENDIX G: SENSITIVITY ANALYSES pertaining to sample
weighting G.1
APPENDIX H: OTHER SUPPLEMENTAL SERVICE PROGRAMS H.1
APPENDIX I: ADDITIONAL SUBGROUP TABLES I.1
viii
TABLES
Table Page
I.1 Receipt of Supplemental Services 9
I.2 Receipt of Upward Bound and Other Supplemental Services 10
I.3 Unweighted Rates of Participation in Upward Bound Among Treatment
Group Members 15
II.1 Response Rates 25
II.2 Control Variables Included in the Regression Models 31
II.3 Sample Size in Applicant Characteristic Subgroups by Treatment Status 32
II.4 Sample Size in Project Characteristic Subgroups by Treatment Status 34
III.1 Impact of Upward Bound on Postsecondary Outcomes (ITT) 41
III.2 Impact of Upward Bound on Postsecondary Outcomes (CACE) 45
III.3 Illustrative Impact Estimates from Sensitivity Analyses 52
IV.1 Impact of Upward Bound on Postsecondary Outcomes by Students’ Grade
at Application (ITT) 58
IV.2 Impact of Upward Bound on Postsecondary Outcomes by Students’
Educational Expectations (ITT) 59
IV.3 Impact of Upward Bound on Postsecondary Outcomes by Students’
Ninth-Grade Math Class (ITT) 62
IV.4 Impact of Upward Bound on Postsecondary Outcomes by Students’
Ninth-Grade GPA (ITT) 64
V.1 Duration of Upward Bound Participation and Completion Rates,
Excluding No-Shows 69
V.2 Quasi-Experimental Impact of an Additional Year of Upward Bound
on Postsecondary Outcomes 72
V.3 Quasi-Experimental Impact of Upward Bound Completion on
Postsecondary Outcomes 73
x
FIGURES
Figure Page
I.1 Conceptual Framework for the Evaluation 7
xii
Executive Summary
Policymakers have long been concerned about the disparities in college attendance between more and less advantaged groups of high school students. Data from the 1990s indicate that students from low-income families were less than half as likely to attend a four-year college or university as students from high-income families. This difference is not surprising given disparities in financial resources and college preparation between high- and low-income high school students. While the vast majority of high-income high school graduates are qualified to attend a four-year college—based on grades and test scores—only half of low-income students have adequate qualifications (U.S. Department of Education 1997), and low-income students face greater financial barriers to college attendance (Kane 1999).
Upward Bound is one of the largest and longest-running federal programs designed to help disadvantaged students prepare for, enter, and succeed in college.[1] Upward Bound is “designed to generate skills and motivation necessary for success in education beyond high school among young people from low-income backgrounds and inadequate secondary school preparation” (Public Law 90-222, Dec 23, 1967). Including the grants funded under the College Cost Reduction and Access Act of 2007, 971 grants were awarded for fiscal year 2007 to serve over 65,000 students in the regular Upward Bound program. The majority of Upward Bound projects are hosted by colleges and universities. According to the program’s regulations, at least two-thirds of each project’s participants must be both low-income and potential first-generation college students. Students typically enter Upward Bound while in ninth or tenth grade or the summer prior to those grades. Although students may participate in Upward Bound through the summer following twelfth grade (for three to four years total), participants typically remain in Upward Bound for about 20 months (Myers et al. 2004). Projects provide students with a variety of services, including instruction, tutoring, and counseling. In addition to regularly scheduled meetings throughout the school year, projects offer an intensive instructional program that meets daily for about six weeks during the summer.
In 1991, the Department of Education launched the National Evaluation of Upward Bound. Conducted by Mathematica Policy Research, Inc. (MPR), the evaluation has included an implementation study—to assess how the program is implemented—and a longitudinal impact study. The impact study was based on a random assignment design implemented in a nationally representative sample of 67 Upward Bound projects hosted by two- and four-year colleges and universities. From 1992 to 1994, eligible applicants to these projects were enrolled in the study. About 1,500 students were randomly assigned to the evaluation’s treatment group and allowed to participate in Upward Bound, and about 1,300 students were randomly assigned to the control group. Comparing the experiences of treatment group members with the experiences of control group members, the evaluation has assessed the effects of the opportunity to participate in regular Upward Bound on high school and postsecondary outcomes.
From 1992 to 1994, a baseline survey collected information on students who applied to Upward Bound projects in the study. Follow-up surveys of all treatment and control group members were conducted in 1994–95, 1996– 97, 199–99, 2001–02, and 2003–04, and high school and postsecondary transcripts were collected after each survey. Upward Bound project staff reported on the participation of students in the program. In addition to the survey, transcript, and participation data that were collected specifically for the evaluation, data from two administrative sources—the National Student Clearinghouse (NSC) and the federal Student Aid (FSA) records—were used in the evaluation.
This report is the last in a series of study reports from the Upward Bound evaluation. It analyzes data from the final round of data collection as well as administrative records, and provides the national evaluation’s first estimates of the effects of Upward Bound on postsecondary completion. It also updates previous estimates of the program’s effects on other postsecondary outcomes. The survey data were collected between 2003 and 2004, approximately seven to nine years after sample members were scheduled to graduate from high school. Other sources of data from the evaluation—previous surveys, high school and postsecondary transcripts, and data on Upward Bound participation provided by program staff—have also informed the findings.
The research questions addressed in this report are:
· What effect does Upward Bound have on the likelihood of attending a postsecondary institution and on the highest level of postsecondary attendance?
· What is the effect of Upward Bound on the likelihood of attending a relatively selective four-year college or university?
· What is the effect of Upward Bound on the likelihood of receiving financial aid in college?
· What is the effect of Upward Bound on the likelihood of earning a postsecondary degree, certificate, or license?
· For which groups of eligible applicants are the effects of Upward Bound greatest?
· What is the effect of Upward Bound participation length and completion on postsecondary outcomes?
Study results
By comparing the study’s treatment group to its control group, this evaluation estimates the value-added effect of the opportunity to participate in Upward Bound—an unusually intensive precollege program—for the students who seek that opportunity and are eligible to participate in the program. The main findings are:
· Upward Bound had no detectable effect on the rate of overall postsecondary enrollment or the type or selectivity of postsecondary institution attended for the average eligible applicant. About four-fifths of both treatment group members and control group members attended some type of postsecondary institution, including four-year institutions, two-year colleges, and vocational schools, and the estimated impact is an increase of less than 2 percentage points in the rate of enrollment (effect size = 4 percent). For enrollment at four-year colleges and universities, the estimated impact is 1 percentage point (effect size = 3 percent). These effects are not statistically significant.
· Upward Bound had no detectable effect on the likelihood of applying for financial aid, or, the likelihood of receiving a Pell Grant. The 1 and 2 percentage point increases in the rates of financial aid application and Pell Grant receipt (effect sizes = 3 and 5 percent) are not statistically significant.
· Upward Bound increased the likelihood of earning a postsecondary certificate or license from a vocational school. It had no detectable effect on the likelihood of earning a bachelor’s degree or the likelihood of earning an associate degree. While about 4 percent of control group members received a vocational certificate or license, nearly 9 percent of treatment group members did, implying an impact of 5 percentage points (effect size = 23 percent). The impacts on receiving any postsecondary credential and receiving a bachelor’s degree are 2 and 0 percentage points (effect size = 5 and 0 percent), respectively, and are not statistically significant.
· Upward Bound increased postsecondary enrollment or completion rates for some subgroups of students. For the subgroup of students with lower educational expectations at baseline—that is, the students who did not expect to complete a bachelor’s degree—Upward Bound increased the rate of postsecondary enrollment and the likelihood of receiving a degree, license, or certificate by 6 and 12 percentage points, respectively, raising the overall postsecondary completion rate to about the level observed for students with higher expectations. Because targeting on the basis of lower educational expectations might be challenging if it creates an incentive for applicants to understate their expectations, further analyses were conducted to examine the effects of Upward Bound on subgroups that might be more readily targeted. According to these exploratory analyses, Upward Bound increased postsecondary enrollment rates for students who were in tenth grade or above at the time of application, students who took a mathematics course below algebra in ninth grade, and students with a ninth grade GPA above 2.5. The estimated impacts were 3, 7, and 3 percentage points, respectively. Additional analyses suggest that Upward Bound also had positive impacts on postsecondary outcomes for some other subgroups defined by student- and project-level characteristics.