To: Participants of the Adding Value to the MSP Evaluations Conference

October 16, 2003

M E M O R A N D U M

To: Participants of the Adding Value to the MSP Evaluations Conference

From: Norman Webb, Principal Investigator

Subject: Summary of the September 18-19, 2003 Adding Value Conference

The second conference of the Adding Value to the Mathematics and Science Partnership (MSP) Evaluations Project was held on September 18-19, 2003 at the Wisconsin Center for Education Research, University of Wisconsin-Madison. MSP evaluators and participants present were:

Terry Ackerman, University of North Carolina-Greensboro

William Badders, Cleveland Municipal School District

Kathleen Bocian, University of California-Riverside

Christopher Broughton, Cleveland Municipal School District

Maria Cormier, University of Wisconsin-Madison

Paul Dillenburg, University of Wisconsin-Madison

Jim Dorward, Utah State University

MaryAnn Gaines, Texas A & M University-Commerce

Tina Heafner, University of North Carolina-Charlotte

Daniel Heck, Horizon Research, Inc., Chapel Hill, NC

Bryant Hutson, University of North Carolina-Greensboro

Daphne Minner, Educational Development Center, Newton, MS

Colleen Orsburn, Vermont Institutes Evaluation Center

Paulette Poncelet, Cleveland Municipal School District

Stephanie Schneider, University of California, Irvine

Iris Weiss, Horizon Research, Inc., Chapel Hill, NC

Attendees from the Wisconsin Center for Education Research Adding Value Evaluations Project were Rob Meyer, Norman Webb, and Paula White. This memo summarizes progress made at the conference.

Introductions and Site Round Robin

Norman Webb, Principal Investigator of the Adding Value to the MSP Evaluations Project, gave an introduction pointing out the goals of the conference including establishing a learning community among the MSP evaluators, analyzing evaluation issues, and providing technical assistance on evaluation designs. Following the introduction, each participant identified key evaluation issues of their MSP.

Norman Webb on SCALE: Four action goals for the project include comprehensive education, immersion units, preservice teachers in higher education, and equity. These goals are being reconceptualized as the project develops. The evaluation is being developed to fit the planned and emerging needs of the project.

MaryAnn Gaines on Texas A&M: long time to implement, responding to NSF, higher ed faculty

Colleen Orsburn on Vermont: multiple components that are being revisited, working with five individual schools to work on professional development, indirect intervention, partnerships, revising pre-service curriculum, funding removed from math initiative

Terry Ackerman on North Carolina: daunting project, 17 counties, goal to improve math and science/narrow the achievement gap, identified 24 facilitators to work with 17 counties, 3 weeks of training, now identifying lead teachers to develop inservice days for math and science teachers, difficulties are lateral entry teachers and getting them certified, developing a science test for grades 5 and 8

Christopher Broughton on Cleveland: use of Porter/ Smithson alignment index and HLM to look at student outcomes

Pauline Poncelet on Cleveland: partnership with three local universities, working on evaluation design, quality of partnership, implementation, and alignment

William Badders on North Carolina: middle school mentoring project started last February, teacher quality grant intended to provide content and faculty development, working with teachers’ union to put mentoring into contract, changing testing structures

Dan Heck on Indiana: working with nine districts, pre-service component and working with middle school plus high school modeling

Iris Weiss on Rochester: developing assessment tools in science, starting with professional knowledge and moving to student achievement, working with teacher leaders in Rochester to bridge a gap between the university and districts

Kathy Bocian on UC-Riverside: target project involves one district in math, successful completion of Algebra by 9th grade, began professional development, matching achievement data with teacher data, conducting one case study of one school, have district benchmarks that informs project

Stephanie Schneider on UC-Irvine: grant involves 100,000 students in 3 school districts, various levels of partnerships, goals haven’t shifted, activities are based on threat of losing infrastructure for grant, had to renegotiate activities, seeking to improve teaching and the number of incoming teachers

Session I: Case Analysis – Texas A&M

MaryAnn Gaines, external evaluator from the Texas A&M MSP, presented the primary goals and key evaluation issues of her MSP. The following hand-outs were distributed:

·  Visualizing Complexity in Science Classroom Learning Environments (2003) by Carol L. Stuessy, Department of Teaching, Learning, and Culture, Texas A&M University College Station, Texas

·  Summary of AIMS PK-16 MSP (2003)

·  AIMS PreK-16 Instructional Content Assessment (2003)

Four key goals of the Texas A&M project in the content area of math, working with nine districts, preK-12:

1)  Professional development through increased math content knowledge

2)  Challenging curriculum for all students, focusing on Algebra with goals of increased enrollment in math and increased SAT and ACT scores over a five year period

3)  Use of technology with math instruction. Contracted with university to instruct teachers on how to use Sketchpad

4)  Conduct research on effectiveness of intervention based on data from surveys and classroom observations

The group proceeded by asking questions to understand the intervention. Norman Webb captured the intervention by drawing a simple logic model of the Texas A&M MSP. (See next page.) Issues were raised about the intervention and the evaluation. Some of these are listed below.

·  Variation in treatment of schools, observation forms could be telling

·  Gain from professional development determined through a survey that address math content and number of hours of professional development

·  Need a pre-measure of teacher content knowledge and then could look at growth on all teachers

·  Implementation consideration – are there contextual barriers within schools that help or hinder? Will ask this in interviews

·  A lot of observation data that provides possibilities for looking at other aspects

·  For analysis plan, need a grid that maps the research questions and the available data

·  Important to determine how the test scores can be linked to the intervention

The following questions and comments were raised regarding the Texas A&M MSP:

·  Might need an institutional link

·  Why are the students not taking higher level classes?

·  Main intervention is at 9th grade level

·  Are any other measures built in besides the 9th grade? Yes, state standards

·  Constraints on data available, is it possible to look at growth over time? Yes, breakdown of students

·  Professional development intervention - teachers are given survey at end of professional development workshop plus classroom practice analysis observation

Measures of evaluation were identified:

·  TAKS/TEKS – Predicted scores

·  SAT/ACT – Baseline

·  Texas A&M classroom practices analysis

·  Classroom observation forms, state standards

·  Survey use of technology

Strengths of evaluation design:

·  Many links in evaluation model

Small Group Discussion: Case Generalizations

Group 1: Similarities to other projects:

·  Logic model, professional development leading to changes in practice and student attainment

·  Difficulties in getting program management to share information such as classroom observations

·  Teacher content tests not agreed to or allowed

Challenge to projects:

·  New state standards for math and science, change in control groups

·  Measurement of teacher content knowledge

·  Linking content knowledge and practice

·  If the intention of the intervention is to change teacher content knowledge in math and science, relating intervention with change in practice

Group 2: Similarities to other projects:

·  Struggle in developing a classroom observation aligned with professional development

·  Use upcoming year as a pilot and make changes based on recommendations learned

·  Survey aligns with standards for math teachers but doesn’t assess content

·  Focus is on teachers and professional development as the main factors influencing student achievement

·  Projects are in the initiation phase, teacher content knowledge is a concern with teachers working out of their field

Challenges to projects:

·  Higher education math faculty and the need to change how they teach, vertical alignment holds some potential

·  In North Carolina, have rural, multicultural areas and difficulties in taking that into consideration

·  Struggling with the different curriculum programs across the districts, difficult to develop a common method to assess student achievement, we have state-wide achievement for math but not for science

·  A need for common assessment measures, professional development is aimed at different styles of teaching which leads to different student achievement which could raise an alignment issue

·  Alignment issues regarding professional development and student achievement -

Differences among projects:

·  Size, some with small districts and others with much larger

·  Contextual effects that affect implementation such as unions vs. no unions and access to data

Session II: Instrumentation

Paula White of the Adding Value Team introduced the session and identified the following objectives of the session:

1)  To gain more familiarity with existing instruments to use in each MSP’s evaluation context

2)  To think about what constitutes a good instrument with regard to surveys, interviews, and observations

The following items were distributed:

1)  A booklet of Instrumentation Items for Evaluation containing twenty evaluation instruments including surveys, interview and classroom observation protocols.

2)  A table of twenty-five different evaluation instruments identifying the subject area, level, and primary purpose.

Small Group Discussion

Group 1: Surveys and Questionnaires

·  Need reliability and validity data, important to pilot test the instruments and learn who will receive the results

·  Do not use instruments verbatim

·  How to define success?

·  Could be problematic to get too many responses since difficult to analyze, some survey questions might be better used in interview format

·  Avoid use of buzzwords, need more neutral questions

Group 2: Interviews

·  Importance of pilot testing the interviews

·  Important to get to the matter quickly and efficiently and use of probes

·  Focus groups important to test interview questions

·  Time-consuming and costly if interviews are too lengthy

·  Be careful of the language, use non-biased terms

·  Use of different interviews depending on the school

·  Very personal – gear the interview to the specific population and context

Group 3: Classroom Observation Protocol

·  Observations are the most expensive strategy

·  Timing of observations is important and should be related to delivery of professional development

·  What kind of data do you want to get out of observation? To inform about the students or the teaching strategy?

Session III: Incorporating Equity Analysis

Norman Webb reviewed equity issues related to evaluation and distributed the David Ramirez article:

·  A Framework for Designing Diversity Responsive Educational Technology Evaluations by J. David Ramirez (2003), Center for Language Minority Education and Research, California State University, Long Beach

The article raises the idea of empowering school culture and social structure to help students relate what they are learning to their lives and life experiences.

Norman Webb raised the question of what strategies the MSP evaluators were using with regard to equity in their evaluations?

Colleen Orsburn: aggregate student data by poverty, don’t lump everyone together in conclusions, don’t confound English understanding and understanding of math

Iris Weiss: don’t color the data with stereotypes

Kathy Bocian: dealing with what to call different minority groups, everyone in California is a minority, now called Underrepresented groups, throw out items not fair to certain groups

Jim Dorwood: provide a network of technical assistance, offer a yearly evaluation workshop

Reporting out on Current Evaluation Issues

Norman Webb engaged participants in a discussion of current evaluation issues they would like to raise. The following questions and comments were made:

·  Are there instruments that can be used across the MSPs? Would that be helpful? Use of the rating scale rather than the whole instrument; collect data by grade level would be better. MSPs are fairly set at this point. Fit joint instruments into the study rather than as an add-on. Nearly all MSPs are conducting observations.

·  The next cohort of MSPs will need instruments, it’s an evolving process

·  Program officers may ask for new data

·  How to measure development of partnerships?

·  How to measure change at the university level?

Session IV: Everything You Ever Wanted to Know About HLM But Were Afraid to Ask

Rob Meyer of the Adding Value Team presented on HLM using some of the analysis of the SAGE program as a concrete example. The following hand-out was distributed:

·  Program Evaluation with Multilevel Data: Statistical Methods for Evaluation Math and Science Programs (2003) by Robert H. Meyer, Wisconsin Center for Education Research

How do we get school productivity effects? Work with HLM at two levels:

1.Student level

2. School level

·  Control for contextual variables

·  HLM acknowledges that there is variability

·  Value-Added – helps explain why some schools are more productive

·  In SAGE, took 30 schools and looked at the average accumulation across grades and time

·  Calculated the means tests between SAGE schools and control group – important to control for SES, race

·  Important to consider effects of other items besides the program

·  Examine interventions across grades

Session V: Evaluation Chat

This session served as an opportunity for participants to raise issues important in the evaluation of their MSPs that had not yet been covered.

Questions and comments raised:

·  Does Rob have data sets that we can use for HLM?

·  Would like more discussion on logic models, links, common scales, implementation, and analysis

·  Proof of concept

·  What are people planning to do for analysis?

Analysis Plans:

SCALE: looking at multiple levels of analysis, student achievement, teacher practice, school context and the growth that schools have made – using value-added, HLM to compare the degree of participation

Vermont: using multiple regression analysis

North Carolina: this year for baseline data collection to measure teacher performance on standards to improve instruction

UC-Irvine: gap analysis, looking at closing the gap for individual groups

Indiana: looking at the affect of the treatment, looking at student achievement, controlling for student demographics, variations in productivity according to students, also looking at treatment teachers by using weekly teacher logs, we identified 4 characteristics of good teaching and tracking design

Rochester: tracking teachers to determine effect of intervention, have a student survey indicator of student outcomes