Final Evaluation Report
2011-2012
Lipan Independent School District
Texas 21st Century Community Learning Centers/Texas ACE
July 21, 2012
Dr. Sharon Mills, Coordinator
Lipan ISD
211 N. Kickapoo Street
Lipan, Texas 76462
254 646-2266
Prepared by
Patricia Williams, Ph.D.
Independent Evaluator
1505 Treeline Drive
DeSoto, Texas75115
Telephone: 972 296-9445
E-mail:
TABLE OF CONTENTS
evaluator information...... 5
Purpose of Program...... 5
Program Description...... 5
Philosophies Employed...... 6
Program Objectives...... 6
Description of Program...... 7
Fiscal Management...... 7
Program Structure...... 7
Characteristics of Participants...... 8
Staff...... 9
Professional Development...... 10
Activities...... 10
Special events...... 11
Program Integration with the Regular School Day...... 11
Partners...... 12
Outcomes12
Evaluation Design...... 12
Methods...... 12
Fidelity of Implementation...... 14
Academic Achievement...... 14
Student Behavior/Attitudes...... 21
21st CCLC Cycle 7 Performance Measures...... 23
Parent Involvement/Participation...... 24
Program Feedback...... 25
Analysis...... 30
Successes...... 30
Opportunities for Improvement...... 32
Next Steps/Recommendations...... 32
List of Tables and Figures
Tables
table 1 Program Objectives...... 6
table 2 Fiscal Management...... 7
table 3 Program Structure...... 7
table 4 Total L5 ACEStudents per Site...... 8
table 5 Total Students/Regular Students by Site...... 8
table 6 Number and Percentage of Regular Participants by Grade...... 9
table 7 Staff Details...... 9
table 8 Professional Development...... 10
table 9 Activities Offered...... 10
table 10 contributions of Partners...... 12
table 11 Student Survey Respondents...... 13
table 12 End-of-Course Exam Scores for Regular Students by Subject...... 14
table 13 English End-of-Course Exam Scores for Regular Students by Grade Level...... 14
table 14 Math End-of-Course Exam Scores for Regular Students by Grade Level...... 15
table 15 Science End-of-Course Exam Scores for Regular Students by Grade Level...... 15
table 16 Social Studies End-of-Course Exam Scores for Regular Students...... 15
table 17 Writing End-of-Course Exam Scores for Regular Students by Grade Level...... 15
table 18 Changes in Classroom Grades...... 16
table 19 Percentage of School Days Attended...... 21
table 20 Student Behavior Rated by Classroom Teachers...... 21
table 21 ACEStudent Behavior Incidents...... 22
table 22 ACEStudent Criminal Incidents...... 23
table 23 21st CCLC Cycle 7 Performance Measures...... 23
table 24 ACE Parent Participants...... 24
table 25 Survey Items Related to Parent Involvement...... 25
table 26 Survey Items Related to Parent Participation...... 25
table 27 Student Survey Items Related to Program Safety and Satisfaction...... 26
table 28Student Items Related to Program Activities...... 26
table 29 L5 ACETeacher Program Feedback...... 27
table 30 Comparison of Regular and Non-Regular Participants...... 30
Figures
Figure 1 Number of Regular Students by Site...... 8
Figure 2 Comparison of Regular and Non-Regular Students’ Semester Reading Grades...... 16
Figure 3 Comparison of Regular and Non-Regular Students’ Semester Math Grades...... 17
Figure 4 Comparison of Regular and Non-Regular Students’ Semester Science Grades...... 18
Figure 5 Comparison of Regular and Non-Regular Students’ Semester Social Studies Grades...... 18
Figure 6Student Responses to Grades and School Work...... 19
Figure 7 Promotion Rates for ACE Students PK -11...... 20
Figure 8 Graduation Rates for ACETwelfth Grade Students...... 20
Figure 9 Student Responses to Homework and Behavior at School...... 22
Figure 10 Program Participants Grading of L5 ACEProgram...... 27
EVALUATOR INFORMATION
Dr. Patricia Williams was contracted by the Lipan ISD 21st Century Community Learning Centers (21st CCLC) program to serve as an external evaluator for the program at a cost of $2,000. The scope of the evaluation consists of evaluating the implementation of the grant, analyzing the progress toward stated goals, determining the grant’s impact on student achievement, and making recommendations for continuous program improvement.
Dr. Williams has experience evaluating 21stCCLC, Early Reading First, Even Start Family Literacy, Prekindergarten Early Start, Latino Youth Mentoring, Title V Population Based Grants, Title I, and Project GREAT (Getting Results Educating Adults in Texas) Centers. Evaluation training was gained through doctoral course work, on the job training and workshops provided by the Evaluation Group while contracted for three years to evaluate Early Reading First grants, and by participating in American Evaluation Associations’ conference sessions, workshops, and e-Study workshops.
PURPOSE OF PROGRAM
21st Century Community Learning Centers Program Description
The 21st Century Community Learning Centers (21st CCLC) are funded by a grant from the U. S. Department of Education through the Texas Education Agency. The program provides support for creating learning centers that offer 1) academic enrichment opportunities during non-school hours to help students who attend high-poverty and low-performing schools meet State and local student academic achievement standards in core subjects; 2) programs and activities that reinforce and complement the academic program; and 3) programs and activities to improve the literacy and educational development of families of the participating students.
Lipan ISD 21st CCLC Program (L5 ACEProgram)
Lipan ISD was awarded a five-year 21st CCLC grant in August 2011. The 2011-2012 year was the first year of the grant which targeted students in grades prekindergarten through twelve. A needs assessment was conducted through focus groups made up of administrators, teachers, students, parents, local organizations, and businesses. Participants used district and campus improvement plans and goals, school performance reports with graduation rates, promotion rates, discipline data, attendance rates, student test results, and lists of available services and curriculum. Students and parents completed surveys, and school staff members were interviewed. Because Lipan is a small rural school district in an economically depressed community that has no afterschool services available for the students, one of the identified needs of the community was for quality, free afterschool programs. Specific needs are to 1) increase the academic success of the participating students in core subjects: reading, mathematics, science, and social studies, 2) increase the number of families of participating students that show gains in literacy and educational development as well as involvement in school-related education activities relevant to their children’s school, 3) improve the citizenship and character education as measured by attendance reports for both school and ACE program days, office referrals, and the number of activities that demonstrate student responsibility and an obligation to the school and community through projects such as community service and service learning opportunities, and 4) increase the academic success of all students by passing all TAKS tests, being promoted to the next grade level, and graduating from high school.Both sites will provide sixteen (16) or more hours of services weekly during the fall and spring semesters and twenty (20) hours or more weekly in the summer that provide activities for the four components: 1) Academic Assistance, 2) Enrichment, 3) Family and Parental Support Services, and 4) College and Workforce Readiness.
Philosophies
Lipan ISD ACE(L5 ACE) program employed the philosophies adopted by Lipan ISD: to ensure that all graduates are prepared, capable, lifelong learners who face challenges well. We will accomplish this by providing an innovative, well balanced, and enriching curriculum, guided by a qualified, flexible team of professionals, with an uncompromising commitment to excellence in an environment conducive to learning.
PROGRAM OBJECTIVES
The Texas 21st CCLC/Texas ACE objectives are to improve 1) the academic performance, 2) attendance, 3) behavior, 4) promotion rates, and 5) graduation rates for students. The outcomes for these objectives will be explored in the Outcomes and Analysis sections of this report
Table 1: Program Objectives
L5 ACEObjectivesObjective / Measurement Protocol / Status
- Improve Academic Performance
Fall Semester to Spring Semester. End-of-Course Exams for Grades 9-11. / Percentage of all ACE Students who increased grades from fall to spring:
Reading=23%
Math=16%
Science=36%
Social Studies=34%
- Improve Attendance
(Average % of Days Present; Regular students who improved from fall to spring) / Average % of Days Present:
Regular Students=95.2%
All ACE Students=94.5%
Improved from fall to spring=14.8%
Perfect attendance both semesters=4.8%
- Improve Behavior
(Average #)
LA Teachers survey of Students Behavior / Regular Students=.28 average # behavior incidents
All ACE Students=.28 average # behavior incidents
Regular Students=.00 average # criminal incidents
All ACE Students=.00 average # of criminal incidents
Improved on Homework Completion=65%
Improved on Class Participation=66%
Improved on Behavior in Class=64%
- Improve Promotion Rates
All ACE Student=100%
- Improve Graduation Rates
All ACE Students=93.3%
DESCRIPTION OF PROGRAM
Table 2 lists the budget for 2011-2012.
Table 2: Fiscal Management (as of July 19, 2012)
2011-2012Description / Budget Code / Total Budget / Total Expended
Payroll Costs / 6100 / 349,866 / 357,256.36
Professional and Contracted Services / 6200 / 13,300 / 10,162.87
Supplies and Materials / 6300 / 83,300 / 79,411.10
Other Operating Costs / 6400 / 10,700 / 9,261.83
Capital Outlay / 6600/15xx / 7,000 / 7,228.76
Total Budgeted Costs / 464,166 / 463,320.92
Cost per Child / $463,320/292 students=$1,586 per child
Table 2 shows that payroll costs accounted for 77% of the budget, and the cost per child was $1,586.
.
Table 3 shows the number of hours and number of participants for each session. The fall session had the highest number of participants.
Table 3: Program Structure
L5 ACEProgram StructureTerm / Hours per week / # of Days Per Week / # of Students Enrolled
High
School / Elem.
School / High
School / Elem.
School / High
School / Elem.
School
Fall / 17.00 / 16.25 / 5 / 5 / 115 / 163
Spring / 16.25 / 16.25 / 5 / 5 / 106 / 158
Summer / 22.50 / 20.50 / 4 / 4 / 23 / 96
The L5 ACE program has two sites with a site supervisor for each, a Family Engagement Specialist, and a District Project Director to manage the program. Each site is required to meet four weeks in the summer, fifteen (15) in the fall, and twenty (20) in the spring.
Characteristics of Student Participants
Table 4: Total L5ACE Studentsper Site
Data Source: TX21st
Number and Percent of Students by Gender and EthnicitySchool / Total ACE
students / Male / Female / African
American / Asian / Hispanic / White / American
Indian/
Alaskan / Two or More
Lipan Elem. / 174 / 54% / 46% / 1% / 0% / 12% / 83% / 2% / 2%
Lipan High / 118 / 51% / 48% / 2% / 0% / 11% / 85% / 1% / 2%
Table 4 presents the total number of students served per site, their gender, and their ethnicity.
Figure 1: Number of Regular Students by Site
Source: TX21st
Figure 1 shows the number of regular students served at each site. Regular students have at least thirty (30) days of attendance at the L5 ACE program. Lipan Elementary exceeded its goal of serving one hundred (100) regular students, and Lipan High exceededits goal of serving forty-five (45) regular students.
Table 5: Total Students/Regular Students by Site
Data Source: TX21st
School / # Total Student Participants / # Regular Student Participants / % of Regular Students to TotalLipan Elementary / 174 / 128 / 74%
Lipan High / 118 / 70 / 59%
Table 5depicts the number and percentage of regular students at each site. Regular students are students who attended 30 or more days at ACE. Lipan Elementary had the highest percentage of regular students to total students with 74%.
Table 6: Number and Percentageof Regular Participants by Grade
Data Source: TX21st
Lipan ElementaryGrade / PreK / K / 1st / 2nd / 3rd / 4th / 5th / 6th / Total
# / 14 / 13 / 18 / 17 / 17 / 13 / 21 / 15 / 128
% / 11% / 10% / 14% / 13% / 13% / 10% / 17% / 12% / 100%
Lipan High
Grade / 7th / 8th / 9th / 10th / 11th / 12th / Total
# / 15 / 20 / 16 / 8 / 3 / 8 / 70
% / 22% / 29% / 23% / 11% / 4% / 11% / 100%
Fifth grade students comprised the highest number and percentage of regular students at Lipan Elementary, and eighth grade students comprised the highest number and percentage of regular students at Lipan High.
Staff
L5 ACE program was staffed with a Project Director, two SiteCoordinators, and a Family Engagement Specialist.Most of the teachers were certified and employed at the school that theirstudents attended. Most of the support staff was also employed in the students’ schools.
Table 7: Staff Details
2011-2012 StaffElementary
Staff / # Paid / Volunteer / In-Kind
Fall / Spring / Fall / Spring / Fall / Spring
Site Coordinator / 1 / 1
Certified Teachers / 5 / 8
College Students / 3
High School Students / 5
Parents / 1
Community Members / 1
Nurses / 1
Paraprofessionals / 2 / 1
High School
Staff / # Paid / Volunteer / In-Kind
Fall / Spring / Fall / Spring / Fall / Spring
Site Coordinator / 1 / 1
Certified Teachers / 5 / 7
College Students / 2
High School Students / 5
Parents
Community Members / 1
Nurses
Paraprofessionals
Totals for both Sites / 14 / 24 / 0 / 11 / 0 / 2
Professional Development
The L5 ACE staff attended valuable professional development and training sessions to increase their knowledge and skills for working in an after school program. Table 8 lists the professional development that the staff attended.
Table 8: Professional Development
2011-2012 Professional DevelopmentDate / Topic / Staff
July 2011 / Pre-Director Training / Project Director
August 2011 / Director Training/State Conference / Project Director
August 2011 / State Conference / Site Coordinator, Family Engagement Specialist, Project Director
August 2011 / Beginning of Year Training / All L5 ACE Staff
October 2011 / Archery Training / Family Engagement Specialist, Staff
October 2011 / Introduction to Evaluation / Project Director
November 2011 / BYO iPad for Education / Site Coordinator, Family Engagement Specialist, Project Director, Staff
December 2011 / Project Director and Family Engagement Specialist Training Webinar / Project Director, Family Engagement Specialist
February 2012 / Advising College Bound Students / Family Engagement Specialist
February 2012 / Family Engagement Specialist Training / Family Engagement Specialist
February 2012 / Project Directors Spring Training / Project Director
March 2012 / Intentional Activity Planning / Site Coordinators
May 2012 / Fish School / Site Coordinators, Family Engagement Specialist
Activities Offered
The L5 ACE program offered a variety of activities to enrich, reinforce, and complement the students’ academic school day. The activities supported four components: Academic Assistance, Enrichment, Family and Parental Support Services, and College and Workforce Readiness. Because the list of activities for both sites is extensive, selected activities that are representative of the program are listed in Table 9. A few of the activity names encompass more than one activity or a related activity. For example, Art Introductionalso includes Art Enrichment.
Table 9: ACE Activities Offered
Selected L5 ACEStudentActivitiesActivities / Related to Objectives
Improve Academic Performance / Improve Attendance / Improve Behavior / Improve Promotion Rates / Improve Graduation Rates
Art Introduction* (2) / x / x
Breakfast Club / x / x / x
Challenge Course* (2) / x / x / x
College Entrance Exam Preparation; College Prep 101 / x / x / x
Activities / Related to Objectives
Improve Academic Performance / Improve Attendance / Improve Behavior / Improve Promotion Rates / Improve Graduation Rates
Community Service / x / x
Cooking Introduction / x / x
CPR/ Babysitting Training / x / x
Drivers Education / x
ELA Tutorials / x / x / x
FFA Meeting / x / x
Fitness * (2) / x / x
Golf / x / x
History through Film / x / x / x
History/Social Studies Tutorials / x / x / x
Hunter Education / x / x
Kickapoo Kids / x / x / x / x / x
Language Arts Enrichment / x / x / x
Life Skills Class / x / x
Math Tutorials / x / x / x
Music Enrichment* (2) / x / x
NHS Meeting / x / x / x
Power Hour / x / x / x
Rodeo Club / x / x
School Spirit / x / x
Science Tutorial * (2) / x / x / x
Self Confidence Building* (5) / x / x / x
Spanish Tutorials / x / x / x
Team Building / x / x
Theatre Production / x / x
There’s an APP for That / x / x / x
*Includes other related Activities (# of sessions)
L5 ACE supported adult education by providing Parent Pre-GED Study, GED Study, and Financial Peace University. Additional activities for parents are listed in the Special Events and the Parent Involvement/Participation sections.
Special Events
Some of the special events provided by the L5 ACE program were the Pumpkin Extravaganza, Fall Festival, Pocahontas Play, High School Dinner and Play, Princess Tea Party, Cowboy Cookout, and Book Fair Family Night. Some parents and children participated in Go Far, a ten week program in which they prepared to run in a 5K race. Go Farparticipants ran in the Zoo 5K and in the Lipan L55K run. Also, parents and children performed community service by picking up trash along a designated highway in November and April.
Program Integration with Regular School Day
Because most of the teaching and support staff worked in the same school that their children attended, they were aware of the homework needs, current classroom concepts and lessons, and many individual student needs. Either an English or Math teacher work in the Power Hour activity that is held the first hour of the program. Because the school is small in size, the teachers have formal and informal discussions about students’ needs.
Table 10: Contributions of Partners
Contributions of PartnersPartner / Contribution / In-Kind Financial Contribution
Grace Tarpley / Programming/Music Instruction / $250
Granbury ISD Sign Language Class / Programming/Goods/Materials / $200
Parks/Recreation District / Programming/Hunter Safety Education / $640
Tarleton State University / Programming/Goods/Materials / $500
The Good News Club / Programming / $550
Weatherford College / Programming/Staff / ($4,000 Paid through grant)
L5 ACE collaborated witha college, university, school district, parks and recreation district, and a faith-based organization for programming and staff. Although Lipan is in a rural area, two colleges/universities are within driving distance.
OUTCOMES
Evaluation Description
The purpose of the evaluation is to determine to what extent various components of the project were implemented and their impact on student achievement. L5 ACE strived for the following outcomes: 1) program implemented as written in grant application, 2) improved academic performance, increased attendance, decreased behavior referrals, increased promotion rates, and increased graduation rates.
Key Evaluation Questions
- Is the L5 ACE program being implemented with fidelity?
- How successful do students, parents, and ACE staff perceive the project to be?
- What is the relationship between program attendance and academic performance?
- What is the relationship between program attendance and promotion and graduation rates?
- What is the relationship between program attendance and school attendance?
- What is the relationship between program attendance and student behavior?
- To what extent do parents participate in the program?
- In what ways is the community involved?
Methods
Both quantitative andqualitative data were collected. Quantitative data included ACE teacher surveys, parent surveys, student surveys, End-of-Course state test scores for grades 9-11, student grades, student attendance, student behavior incidents, classroom teacher ratings of students, promotion rates, and graduation rates. Qualitative data included interviews,focus groups, open-ended questions from surveys, and observations during site visits.