The New Superintendents Induction Program

Annual Evaluation Report FY 2011

The New Superintendents Induction Program (NSIP) is a joint initiative of the Massachusetts Association of School Superintendents and the Massachusetts Department of Elementary and Secondary Education’s Office for Targeted Assistance

July 2011

NSIP Annual Evaluation Report: FY 2011 / Contents

Contents

Contents......

I.Introduction

II.Program Objectives

III.Curriculum and Content Days

IV.Work Assignments

V.Coaching Services

VI.Impact on Superintendent Capacity and Performance

VII.Impact on District Improvement

VIII.Participant Satisfaction

IX.Suggestions for Program Improvement

X.Conclusion

UMass Donahue Institute
Research and Evaluation Group
NSIP Annual Evaluation Report: FY 2011 / Introduction

I.Introduction

The Massachusetts Association of School Superintendents (MASS) and the Massachusetts Department of Elementary and Secondary Education (ESE) have entered into an innovative new partnership intended to develop and sustain district leadership and, by extension, enhance district capacity to support school improvement. This strategy aligns with ESE’s theory of action for district wide school improvement and manifests in the form of the New Superintendent’s Induction Program (NSIP).The University of Massachusetts Donahue Institute serves as the program’s external evaluator.

NSIP seeks to enhance superintendents’ effectiveness through an induction process centered on “The Massachusetts Way.” The Way is a specific approach to practice in relation to four broad activities that are considered essential to effective district leadership: strategic instructional leadership, district leadership team development, human resource management, and supervision and evaluation. The program delivers its curriculum through a series of Content Days and supports implementation of key practices through related work assignments and intensive coaching by a team of former superintendents who are themselves receiving extensive professional development in relation to the NSIP curriculum. The primary role of the coach is to support superintendents’ implementation of that curriculum.

NSIP is conceptualized as a three-year induction program designed to meet the needs of the Commonwealth’s new superintendents, with a cohort of newly hired, first-year superintendents entering the program each July. The first NSIP cohort (Cohort 1) entered the program in July 2010 and consisted of 25 superintendents.Cohort 2 is expected to include approximately 30 superintendents, who will begin the program in July 2011, even as Cohort 1begins NSIP’s second year curriculum. While Cohort 1 was limited to districts in Level 3 or Level 4 accountability status as of spring 2010, Cohort 2 is also open to Level 1 and Level 2 districts, on a fee basis.

Year One Evaluation Goals and Data Sources

This report summarizes key findings and lessons learned through the first year of NSIP operation, which might best be described as a fully implemented pilot year. It focuses on core findings emerging from surveys and individual and group interviews conducted over the course of FY 11. Owing to its still early phase of implementation, these findings emphasize formative feedback—some of which has previously been shared and addressed—as well as participant satisfaction and some preliminary indications of program effectiveness. This emphasis is consistent with the goalsdefined for the first year of the NSIP evaluation, which wereto:

  • Ensure that implementation and refinement of NSIP is well supported by timely and direct feedback from program managers, coaches, and participating superintendents.
  • Contribute knowledge that will support NSIP effectiveness, thereby enabling new superintendents to function more effectively in their role as leaders of district and school improvement.
  • Develop an understanding of preliminary NSIP impacts on superintendents’ practice and integration of The Massachusetts Way into their district.
  • Develop a long term NSIP evaluation plan that retains a formative focus, but expands to assess intermediate and later term impacts on district leadership practice, district capacity to support school improvement, and student achievement.

Following is a brief description of individual data sources contributing to the development of findings highlighted in this report:

  • Content Day Surveys: Each of the eight Content Days in FY 11concluded with administration of a participant survey. These surveys gathered superintendent and coach perspectives on each session’s effectiveness in relating key concepts and onthe relevance of described concepts and tools to practice.
  • IndividualInterviews: In January-February 2011, individual interviews were conducted with eight (8) superintendents on site, in their district office. A parallel set of in-person and phone interviews were conductedwith their NSIP coaches. Key program leaders and facilitators were also interviewed.
  • Group Interviews: In April 2011, a group interview was conducted with members of the coaching team not previously interviewed. A larger group discussion of lessons learned involving all coaches was held during the May Content Day, with a similar discussion held with superintendents that same day.
  • Participant Observation:Throughout FY 11, the evaluator observed and participated in Coach Training and Content Days, thereby gaining a rich understanding of how these program components function from a logistical, curricular, and cultural perspective. Coach Training Days, in particular, provided significant exposure to reflective discussion and strategizing in relation to program implementation.
  • NSIP Annual Superintendents Survey:At the conclusion of the fiscal year, a web-based survey was administered that examinedparticipating superintendents’ experiences with NSIP, established baseline measures of superintendents and district practice, and explored preliminary program outcomes. Completed by 21 of 25 superintendents at the time of this analysis, results will be linked to future surveys containing parallel measures as part of long term implementation and outcomes assessment.

Together, these sources offer substantial insight into superintendents’ experiences with, and reactions to, NSIP’s curriculum and implementation. Although self-report data are inherently subjective, they are also potentially accurate and informative, with confidence in findings increasing as sources are triangulated across informants and data sources. Such has been the case in this past year, during which evaluation data revealed both the complexity of the undertaking and the consistency of challenges affecting superintendents’ uptake of key practices associated with the Massachusetts Way.

Report Organization

The NSIP FY 11 Annual Evaluation Report utilizes the framework of the NSIP Annual Survey as a basis for its organization. This survey, the most comprehensive and systematic data collection effort undertaken in the evaluation’s first year, consists of seven distinct topical sections, as listed below, while the report also features program improvement suggestions and a conclusion.

  • Program Objectives
  • Curriculum and Content Days
  • Work Assignments
  • Coaching Services
  • Impact on Superintendent Capacity and Performance
  • Impact on District Improvement
  • Participant Satisfaction
  • Program Improvement
  • Conclusion
UMass Donahue Institute
Research and Evaluation Group / 1
NSIP Annual Evaluation Report: FY 2011 / Program Objectives

II.Program Objectives

NSIP is focused on core objectives that reflect program designers’ understandings of how to enhance district capacity to catalyze school improvement. The first year curriculum addressed four of these five objectives,[1]which included:

  • Building instructional leadership
  • Developing effective leadership teams
  • Developing collaborative relationships with key constituencies
  • Strategic management of resources
  • Developing effective systems for supervision and evaluation (not addressed)

As presented in Table 1, about two-thirds of 2011 Annual Superintendent Survey respondents generally believed that the amount of time focused on building instructional leadership and developing effective leadership teams was “just about right.” In comparison, a similar proportion believed that “too little time” was focused on strategic management of resources. Results in relation to the amount of time focused on developing collaborative relationships with key constituencies (i.e., the school committee and union, business, and community leaders) was mixed, but leaned towards the opinion that insufficient time was devoted to this objective as well.

Table 1: Time Spent on Core Objectives

N = 21 / Just about right / Too much time / Too little time
Building instructional leadership / 71% / 5% / 24%
Developing effective leadership teams / 67% / 10% / 19%
Developing collaborative relationships with key constituencies / 38% / 19% / 43%
Strategic management of resources / 29% / 5% / 67%

Table 2 suggests thatNSIP objectives are generally “very well” aligned with superintendents’priorities for district improvement, with the strongest alignment found in relation to building instructional leadership and developing effective leadership teams.

Table 2: Alignment of Objectives with Superintendents’ Priorities

N = 21 / Very well aligned / Wellaligned / Not well aligned / Not at all aligned
Building instructional leadership / 67% / 24% / 10% / 0%
Developing effective leadership teams / 67% / 33% / 0% / 0%
Developing collaborative relationships with key constituencies / 48% / 33% / 14% / 5%
Strategic management of resources / 43% / 43% / 14% / 0%
Developing systems for supervision and evaluation / 52% / 24% / 19% / 0%

These findings regarding alignment to superintendents’ priorities for district improvement are consistent with interview data, whichalso suggest that the program curriculum is relevant to the work superintendents feel they must undertake in their districts. However, open-ended comments underscore the challenge NSIP faces in delivering a curriculum that is relevant to and aligned with the needs of superintendents with vastly different experiences and personal strengths, in districts with their own unique attributes. Further, the timing or sequencing of topics was sometimes noted to be an issue, as NSIP and district schedules for addressing specific topics were sometimes out of alignment.

  • I enjoyed the program and found it very helpful. Because my strength is in instructional leadership, I feel that it could have been more beneficial for me to spend more time in areas that were more of a challenge, e.g. relationships with school committee.
  • The objectives make sense and are applicable to the work that we are asked to do. My only criticism would be centered around practical applications to the varying needs of those who participated.
  • I believe the NSIP is of great value and I feel fortunate to have participated this year. The overarching objectives of the NSIP will serve to strengthen me as I move forward in my leadership of our school system.
  • I thought the objectives were great but timing for some of them was a bit off, i.e. by the time the talk came around to budget many were already very involved in planning for the next year.

NSIP’s pilot year served to underscore the importance of developing a shared vision and priorities among superintendents and their school committees, with ongoing acknowledgment that they are sometimes not well aligned, making it difficult for superintendents to advance their vision for district improvement. To the extent that superintendents may value and pursue the core objectives of NSIP, it is then important that their school committees share in prioritizing these objectives.

Table 3 displays superintendents’ perceptions of the alignment of NSIP objectives with the priorities of their school committees. These data suggest that school committees’ priorities are generally “well”or “very well” aligned. However, their perceived priorities display a somewhat weaker alignment with NSIP objectives than do superintendents’ in relation to all objectives except strategic management of resources. Focus on the development of effective leadership teams is theobjective for which the greatest difference in priority is perceived. Comments do not illuminate the reason for this difference, but it may be that it is perceived as an operational capacity that is outside the direct influence of the school committee.

Table 3: Alignment of Objectives with School Committee Priorities

N = 21 / Very well aligned / Wellaligned / Not well aligned / Not at all aligned
Building instructional leadership / 48% / 29% / 14% / 10%
Developing effective leadership teams / 19% / 67% / 5% / 10%
Developing collaborative relationships with key constituencies / 38% / 38% / 19% / 5%
Strategic management of resources / 52% / 24% / 24% / 0%
Developing systems for supervision and evaluation / 38% / 43% / 10% / 5%
UMass Donahue Institute
Research and Evaluation Group / 1
NSIP Annual Evaluation Report: FY 2011 / Curriculum and Content Days

III.Curriculum and Content Days

At NSIP’s July 2010 kick-off event, program leaders shared a grid outlining the planned scope and sequence of the NSIP curriculum. Over the course of the year, the specific approach to delivering that curriculum was defined and refined in an iterative process in response to the perceived needs of superintendents. As observed, Content Day facilitators, Drs. Rachel Curtis and Elizabeth City, would collaborate with program manager Joan Connolly to develop aninstructional plan for each Content Day in the weeks and days leading up to the event. The day prior to each Content Day was a Coach Training Day, at which the 15 coaches would meet with Curtisand Connolly and gain exposure to the content that was to be delivered to superintendents the following day.[2]

As observed, these meetings served a dual purpose. First, and by design, these gatherings familiarized coaches with key concepts and tools, some of which were new to them. In this way, coaches received the training they needed in order to support their assigned superintendent(s) implementation of the NSIP curriculum. Interviews revealed very high satisfaction among coaches with the orientation and support they received through these events. A second key purpose of these meetings was to allow coaches an opportunity to help shape content and delivery, and an active culture of sharing emerged, such that coaches were very comfortable suggesting changes toproposedContent Day plans based on their understanding of superintendents’ emergent learning needs.

During personal interviews in February 2011, some superintendents related concerns that while the focus of content was generally of great interest to them, itscoherence and delivery could bear improvement. In particular, some suggested that the curriculum was not sufficiently wellarticulated, leading to uncertainty as to the connection between Content Day topics/activities and the broader goals of the program year. In response, the facilitators worked to more actively connect activities to program year goals during subsequent Content Days and planned a more transparent approach to communicating the curriculum in the next program year as well.

Annual Superintendent Survey results, displayed in Table 4, appear to corroborate earlier interview feedback and suggest that while content was for the most part presented in a clear and coherent fashion with connection to year-end goals, there is room for improvement.The sequencing of content seems to be an area of particular concern.Explanation for sequencing concerns are more fully articulated in the next section, which focuses on NSIP work assignments.

Table 4: Presentation of NSIP Curriculum

N = 21
NSIP content was… / Strongly Agree / Mostly Agree / Mostly Disagree / Strongly Disagree
Integrated effectively as part of a coherent curriculum / 14% / 71% / 14% / 0%
Sequenced in a manner that supported learning / 10% / 57% / 33% / 0%
Clear in its connection to NSIP’s year-end goals / 14% / 71% / 14% / 0%

Table 5 provides additional insight regarding superintendents’ reaction to Content Days and to selected approaches to delivering content. The overall value of attending Content Days was generally considered “good” or “excellent.” Content Day facilitation was also very well received,however, a few respondents rated it as “poor” or “fair,”with on expressing concern that it was delivered in a condescending fashionand another asserting that delivery was not geared to adult learning styles.

Small group work—typically involving groups of superintendents and coaches convening to share information, discuss a topic or complete an exercise—was the modality of choice among superintendents. Meanwhile, reaction to panel discussions—a strategy employed in discussions of union and school committee relationships—were less well received. While many interview respondents expressed a great appreciation of “Strategy in Action”—the text supporting the NSIP curriculum—some felt that assigned readings could have been connected more clearly to the Content Days and others expressed an interest in reading from a broader range of sources.

Table 5: Content Day Ratings

N = 21 / Excellent / Good / Fair / Poor
Quality of facilitation / 48% / 38% / 5% / 10%
Effectiveness of panel discussions as a format for learning / 5% / 52% / 33% / 10%
Effectiveness of small group work as a format for learning / 71% / 24% / 5% / 0%
Connection of “Strategy in Action” readings to Content Days / 33% / 38% / 24% / 5%
Overall value of attending Content Days / 38% / 43% / 14% / 0%

In the context of positive overall feedback regarding the value of Content Days, superintendents offered a range of constructive feedback to inform Content Day planning in the future.

  • The content days were more engaging and relevant as the months went by.
  • The greatest benefit of these days was the ability to collaborate with coaches and other new Superintendents to share information and discuss best practices.
  • Content did not always align with what was happening in my work in the district. The big picture wasn't always clear to me, and although I now understand why we were doing what we were doing, at the time it felt like there was a disconnect.
  • It was often not the content that was a problem although at times we were focusing on things that met no felt immediate needs of the participants. More, it was the flow and flexibility of the work that may have made it difficult to connect to at times.
  • [The] best learning was the collaborative working between different Supers and Mentors. At times I felt that the assumption was that the New Supers did not know anything - I think that you need to know the new Supers a bit better and try to tap into some of their experience as well as the mentors.
  • I rated the effectiveness of the panel discussions as fair. I do not think that the large group setting was conducive to all sharing or asking questions. Usually only the most vocal would dominate the forum. I would restructure to have burning questions submitted ahead or time by participants or break up into smaller groups with the panel members in a round table discussion setting.