Bellevue Middle iChange Collaborative Unit Evaluation
December 7, 2012

Submitted to:Created by:

Michelle SpringerSamara Orkin

Middle School Principal

Bellevue Middle School

Cathy Hancock

Educational Specialist

Bellevue Middle School

II. Evaluation Introduction

A. Organizational Context of the Evaluation

Two years ago, Bellevue Middle School (BMS) gave their students a survey called CSCI report. When BMS got the scores back, they noticed that 79% of students did not identify positively for social and emotional security and 55% of students did not identify positively regarding feeling physical safe in the school building. In addition, only 57% of the student body admitted to having respect for diversity. The questions and implementation of the survey are unknown, but the results that most students did not feel physically, socially, and emotionally safe caused the administration to want to act. This evaluation tool used the Sense of Social and Emotional Security tool to guide much of the evaluation(Appendix A).

BMS is a Metro Nashville Public School with 700 students and 37 teachers serving 5th grade through 8th grade. BMS is in good standing according to NCLB requirements. Therefore, it is a school of choice for schools in the district labeled as “failing” or “high-priority” because they have enough children passing the state test demonstrating proficiency in grade level content. As a result, children are bussed to Bellevue from North Nashville (an extremely impoverished and under-educated area: 60% have a high school diploma or less, and the average income is $12,000).

The ultimate challenge is to find a program to help improve the number of students who feel safe in the school building through bringing attention to student differences and educating children on those differences especially with the number of students being bussed in increasing.Dr. Michelle Springer is the principal of the school. This is her third year with BMS (1st year as principal), and this is an issue that is very close to her. She feels very strongly about running a school where students feel safe and comfortable being themselves. Dr. Springer also states, “cultural awareness was an area that students and staff addressed in a survey last year, and we know that cultural awareness for all stakeholders will be an intricate part of losing the achievement gap.”

BMS identified lack of knowledge on cultural differences to be a main source as to why students do not feel socially or emotionally safe. As a result, BMS connected with Martha Caldwell for support. The iChange Collaborative Unit is the unit on the cultural awareness curriculum that BMS will be using in an adapted form specifically for them. The program was originally created and adapted by Martha Caldwell and Oman Frame for their school in Atlanta.

The nature of work for this evaluation is to see if a curriculum called “Inquiry into Identity: A Middle School Curriculum for Race, Class and Gender Issues” from the iChange Collaborative Unit is able to improve student’s sense of emotional and social security in the school environment. This evaluation will determine if the educationalprogram can create awareness in cultural differences that will eventually and hopefully help students feel safer at school.

B. Objectives of the Inquiry into Identity: A Middle School Curriculum for Race, Class and Gender Issues iChange Collaborative Unit

The objectives of this program are to educate children on their own cultures as well as those of other students with culture involving race, class, and gender. Through open discussion, reflection, and activities, students are exposed to conversation and awareness in an explicit and safe manner. This program has 10 lessons that the students will attend in their Owl Time, which is every held every two to three weeks on Friday afternoons. The lessons cover a wide range of topics:

Table 1:Lesson Topics

Lesson Topics
Lesson One: Introduction to Owl Groups
Lesson Two: Who Are You?
Lesson Three: Discovering the Power of Emotions
Lesson Four: Guidelines for Listening and Sharing
Lesson Five: What is Learning?
Lessons Six and Seven: Family History Interview Projects
Lesson Eight: What’s Great and What’s Not about Being a Girl
Lesson Nine: What’s Great and What’s Not about Being a Boy
Lesson Ten: Talking about Oppression

The key component in achieving these objectives is through the teachers who must follow the lessons and curriculum. Additionally, the school administration must support the teachers in this endeavor. Finally, many students leave early on Fridays and that is when the lessons are taught. Therefore, buy-in from all stakeholders will be important in providing a worthwhile context for the evaluation.

The ultimate objectives:

  1. Increase student sense of social and emotional security by 10%.
  2. Ensure transfer of knowledge of cultural awareness across grades and settings by an increase of 10%.

III. The Evaluation Process

A. Evaluation Methodology, Stakeholders, Key Questions and Logic Model:

i. Purpose of the Evaluation

The purpose of this evaluation is to see if the iChange Collaborative Unit actually creates a high sense of security, a more open-minded student population, and transfer of knowledge into social and academic settings. Being that the curriculum is only given to the 8th graders this year, the 8th graders are essentially piloting it. The evaluation will provide information to show if this program can yield success and if it is worth the investment, time and energy that will go into the evaluation and the unit. Looking closer at the key questions listed below, the evaluation will answer those questions and the answers will indicate the next steps for BMS to take to create a more socially and emotionally safe place for students.

ii. Key Questions Guiding the Evaluation

From conversations with Dr. Springer and Cathy, the main concerns are bolded below. Using the main concerns, questions were created that the evaluation plan will answer.

  1. Sense of Security: Did the IChange Collaborative Unit increase students emotional and social security at school?
  1. Transfer of Knowledge: Does the iChange Collaborative Unit transfer to more than just the students being taught the unit & does it transfer to different settings?

iii. Evaluation Method

After analyzing the needs and goals of the iChange Collaborative Unit and learning the purpose of the implementation, Kirkpatrick seems to be a great fit for the model of evaluation. With Kirkpatrick, the true reaction, learning, behavior, and results can be evaluated. When working with children, it is important to see reactions to what they learned as well was the actual content learned. With a curriculum about content, the reaction of the students will be a key component to measure because reaction can help to understand the learning. Also, behavior is valuable with middle school children to see if they can actually change behavior based on learning in the program. Finally, the results will be key to see how this unit impacts the school as a whole to help look to see if it should be repeated and what could be improved. Kirkpatrick will allow a true picture of the success of the unit because it is thorough in looking at immediate reaction, what is learned, and what changes, to then look at bigger results proving that this model of evaluation is a strong choice. Below in Table 1 is the outline of the approach that will be used with Kirkpatrick.

Table 2: Evaluation Methodology

iChangeCollaborative Unit Kirkpatrick Evaluation Methodology
Level 1-Reaction /
  • How do the kids react? How do the teachers react? How was this unit perceived by participating parties? Was there relevancy in the context?

Level 2-Learning /
  • What is learned about culture? Has knowledge and awareness increased?

Level 3-Behavior /
  • Have social and emotional behaviors changed? What kind of behaviors?

Level 4-Results /
  • Do students feel a stronger sense of social and emotional safety in the building?

Quasi-Experimental: Because there was no Treatment, but measurements were taken. Two years ago the survey called CSCI was given to the students showing that students did not feel socially and emotionally safe, which is where the need for a treatment came from. Now there is a treatment and then more in depth measurements to determine if students can feel safer.

iv. Stakeholders

The table below outlines the different stakeholders for the iChange Collaborative Unit. From talking with Bellevue Middle School administration, the stakeholders are only those that have significant influence in the success of the program. Whether it is be directly giving or receiving the unit or benefiting from the results and helping to build buy-in and support the program, the stakeholders that will be most affected are listed below.

Table 3: Stakeholders

Stakeholders
Stakeholder / Role
Michelle Springer / BMS Principal
8th Grade Teachers/Unit Teachers / Ones who are teaching the unit
8th Grade Students / Recipients of unit
Parents / Recipients of improved awareness and safety from children
BMS other teachers/staff / Observe the difference in the 8th graders to other grades (related arts/special teacher—music/art/Spanish/technology/etc.

1 | Page

v. BMS Logic Model

Table 4: Logic Model

B. Evaluation Design:

Using Kirkpatrick’s design, the evaluation’s design will attempt to provide BMS with the appropriate information to determine if the effects of the unit are changing students identifying themselves as feeling safe at their school. Addressing the administration’s key questions in how the students and teachers perceive the unit (reaction), what is actually learned in the lessons (learning), how behaviors change (behavior), and actual impact and results the unit has on the students and school as a whole (results). See section C. Data Collection Plan and Instruments for specific questions associated with each level of the Kirkpatrick model, collection instruments used to assess for them, and the timing for administration.

PLEASE NOTE: All evaluation instruments will be surveys due to time constraints in the school setting.

The following are key components & factors impacting the evaluation’s design - Figure 3 provides a summary view:

Structure / Timing

  1. The evaluation design looks at the iChange Collaborative Unit as whole unit containing 10 detailed lessons with activities within the lessons. The teaching of the lessons will spread throughout a semester.
  2. The evaluation design uses a mixture of tools to evaluate the effectiveness of the unit. This design will provide Bellevue Middle School with quantitative and qualitative data.
  3. Reaction Survey: mid-way through (5 lessons in) and at the conclusion
  4. Learning Survey: After every two lessons
  5. Behavior Survey: After completion of unit (pre-post assessment)
  6. Result Survey: After completion of unit (post assessment)

Sampling

  1. 100% of 8th grade students (learning, reaction, result)
  2. 100% of those who teach the unit (learning, reaction, result)
  3. 100% of Related Arts/Specials teachers and 8th grade homeroom teachers (behavior)

Use of Results

  1. All answers to the surveys and assessments will track trends from the responses to determine the effects of the unit. Section D. Coding Plan and Data Analysis provides details regarding the analysis of instrument results.
  2. The results each evaluation tool will be compiled into a single report for the administration of Bellevue Middle School for reviewing
  3. The results will be used to align with the key questions and make programming decisions about repetition, population to give the unit to, and safety in the building.

Summary of Evaluative Design’s Annual Treatment Cycle

M1 Behavior Assessment (Pre-test) TL1TL2 M2 Learning Survey1

TL3TL4 M3 Learning Survey2  TL5 M4 Reaction Survey TL6 TL7 M5 Learning Survey3 

TL9TL8 M6 Learning Survey4 TL10 M7 Learning Survey5 & M8 Reaction Survey

M9 Behavior Assessment (Post-test)+ M10 Results Survey

T = Component Treatment; L = Lesson; M = Measurement

C. Data Collection Plan and Instruments:

i. Data Collection Plan

Table 5: Data Collection Plan

Evaluation Level / Who (Data Source) / Objective Questions / Evaluation Instrument / Timing / Responsible Party
Reaction-
The reaction of the student and teacher participants? / 8th Grade Student participants /
  • How did participants perceive this unit?
  • Do participants foresee using the content?
  • What is missing from the lessons?
/ Reaction Survey / After 5 lessons & after all 10 lessons / Principal, Unit Director, Unit Teachers
Teacher’s Teaching Unit /
  • How do the teachers react? How is it perceived?
  • What is missing from the lessons?
  • Was there relevancy in the context?
/ Reaction Survey / After 5 lessons & after all 10 lessons / Principal, Unit Director, Unit Teachers
Learning-
The knowledge learned and retained after the lesson / 8th grade Student participants /
  • Are students learning the information?
/ Learning Survey / Immediately after every one or two lessons.
L1L2 together
L3L4 together
L5L6L7 together, L8L9 together, L10 alone / Principal, Unit Director, 8th grade homeroom teachers
Behavior-
The behaviors in comparison to other grades not receiving treatment and the behaviors that are transferring outside of the classroom the environment where the treatment is given. / Related Arts/Specials Teachers /
  • How do students behave with social issues involving culturally different students?
  • How do the behaviors compare to the other grades taught?
/ Behavior Assessment / Before the unit starts & at the conclusion of the unit 1 month after / Principal, Unit Director
8th Grade Teachers (teachers of participants) /
  • How do students behave with social issues involving culturally different students?
  • Are behaviors consistent in different settings demonstrating transfer of knowledge?
/ Behavior Assessment / Before the unit starts & at the conclusion of the unit 1 month after / Principal, Unit Director
Results-
The results in comparison to the CSCI results from two years ago to see if students are feeling socially and emotionally safer in the building / 8th Grade Student participants /
  • Do students feel a stronger sense of social and emotional safety in the building?
/ Results Survey (based on the survey used to establish the need for this kind of a unit) / One month after the conclusion of the unit / Principal, Unit Director

ii. Instrument 1 - Reaction Survey

Objective:To gauge the perception/reaction all of the students and teachers who either taught or were a participant for the iChange Collaborative Unit

Responsible Party:Principal, Unit Administrator, Unit Teachers,

Given to:All those directly in contact with the treatment, 8th grade students at BMS & Teacher’s teaching the iChange Collaborative Unit, 80-100% response goal

Timing of Administration:The same survey immediately following:

  1. Lesson 5
  2. Lesson 10

Collection Method: Hardcopy pen/paper survey

Survey Design

This survey will be administered using a 5-point Likert scale, which should be attached to each of the survey statements below. There is one survey designated for teacher’s and one survey designated for students.

Scale

Strongly Agree ☐ Agree ☐ Neutral ☐ Disagree ☐ Strongly Disagree ☐ N/A ☐

Survey Statements

Which Lesson did you just complete? (Check One)

____ Lesson 5(half-way point for the unit)

____ Lesson 10 (completed the unit)

Student Statements:

  1. I know more about cultural difference than I did before.
  2. The lessons helped answer questions that I had about cultural differences.
  3. Information was not left out.
  4. I already plan to use these lessons in my own life.
  5. These lessons help me to better understand my peers.
  6. These lessons help me feel safer at BMS.
  7. Understanding more about culture, also taught me about my own culture.
  8. These lessons will help other students feel safer at BMS.
  9. These lessons will get rid of some of the bullying that goes on at BMS.
  10. I think that these lessons will help create a safer school.

Teacher Statements:

1.The lessons helped answer questions that students had about cultural differences.

2.There was not any missing content.

3.I can foresee that students will use these lessons in their own lives.

4.These lessons helped students to better understand their peers.

5.These lessons will help students feel safer at BMS.

6.These lessons will help to get rid of some of the bullying that goes on at BMS.

7.Teaching these lessons is worth my time.

8.The lessons are appropriate for the students.

9.These lessons are good for BMS students.

10.I think that these lessons will help create a safer school.

1 | Page

iii. Instrument 2 – Learning Assessment

Objective: Gauge the extent participants retained content from the treatments by measuring how much they actually learned.

Responsible Party: Principal, Unit Director, 8th grade homeroom teachers

Given to:All program participants, 8th grade students at BMS, 80-100% response goal

Timing of Administration:Six different tools will be given after various grouping of lessons (groupings based on content pairs in unit):

  • Learning Assessment 1 after Lessons 12
  • Learning Assessment 2 after Lessons 3 & 4
  • Learning Assessment 3 after Lessons 5, 6, and 7
  • Learning Assessment 4 after Lessons 8 & 9
  • Learning Assessment 5 after Lesson 10

[Please Note: There is no additional cumulative assessment necessary for the learning assessment section of instruments. Cumulative measures will take place in the results and behavior surveys.]

Collection Method: Hardcopy pen/paper survey

Survey Design: Each Learning Assessment has 5 Questions with guidelines above in the Timing of Administration section explaining when to give the assessment. Questions can be presented on a screen to the class, and students can write answers on paper numbered 1-5 with lesson numbers at the top and turn it in.

Learning Assessment 1 (Lessons 1 & 2):

  1. The purpose of the Owl Time is to explore various aspects of identity. (True or False)
  2. Which one is NOT a way that you can contribute a high quality of work?
  3. Commitment to excellence
  4. Communication with teachers
  5. Communication with classmates
  6. Commitment to technology
  7. In Owl Time we will learn about how race, class, gender, family background, religion and learning styles affect only one level of society. (True or False)
  8. Honesty, openness, courage, caring, commitment, respect, and trust will contribute to a safe place during Owl Time. (True or False)
  9. We can only talk about the lesson plans in class. (True or False)

Learning Assessment 2 (Lessons 3 & 4):