Abstract
This research illustrates the James Madison University (JMU) assessment process through an examination of the University’s Alternative Spring Break (ASB) program. The two-hour a week training program for ASB trip leaders emphasizes leadership, interpersonal skills, problem-solving, and civic responsibility. Trip leaders are administered assessments on three separate occasions. Comparisons are made between the Pre-test, Pre-trip, and Post-trip results for each of the subscales. In addition, trip leaders are compared to trip participants. Findings and implications for program improvement are discussed.
The Assessment Process
Program Outcomes
1. To improve leaders’ personal competence:
- Leaders will develop confidence in their leadership skills.
- Leaders will improve interpersonal skills.
- Leaders will improve problem-solving skills.
2. To increase leaders’ civic responsibility:
- Leaders will develop an appreciation for diverse perspectives.
- Leaders will develop a commitment to future service.
- Leaders will increase in their awareness of social justice issues.
Methods
Participants and Procedure
•Leaders are defined as those students who volunteer to lead an ASB trip and participate in the leadership training course.
•Trip participants are those who volunteer to participate in an ASB trip led by an ASB leader.
•In 2006-2007, 35 leaders and 69 trip participants completed the assessment.
•In 2008-2009, 32 leaders and 39 trip participants completed the assessment.
•Data was collected at three time points for leaders and two time points for trip participants.
•Leaders were administered the assessment within 2 weeks of beginning training, the week before spring break and immediately after returning from their spring break trip.
•Trip participants were administered the assessment 3 weeks prior to the trip and immediately following the trip.
Instrument
In 2006-2007, the ASBQ was a multiple-choice instrument with 42 items. Two of the interpersonal relationships subscale items were removed, resulting in 40 items on the 2008-2009 assessment. The test consists of five subscales, two of which were adapted from the Civic Attitudes and Skills Questionnaire (Moely, Mercer, Ilustre, Miron & McFarland, 2002). The Civic Action subscale and the Interpersonal and Problem-Solving Skills subscale. The items on these two subscales were answered using a 5-point Likert-like scale. That is, the respondents could choose from five options, ranging from “Strongly Disagree” to “Strongly Agree.” The remaining three subscales, addressing Interpersonal Relationships, Personal Competency, and Social Justice, include items adapted from the Student Service-Learning Course Survey (Wang, Ye, Jackson, Rodgers & Jones, 2005). Students respond to the Interpersonal Relationships subscale using a 7-point Likert-like scale with choices ranging from “Strongly Disagree” to “Strongly Agree”. The Personal Competency and Social Justice subscales were presented on a 7-point Likert-like scale, with choices ranging from “Completely Disagree” to “Completely Agree.” In the 2008-2009 assessment, all items were on a 7-point Likert-like scale and all items used the “Strongly Disagree” to “Strongly Agree” response choices.
Results
In 2006-2007 and 2008-2009, leaders showed significant change on the Personal Competency and the Social Justice subscales.
2006-2007: Personal Competency subscale: significant increase from pre-test to pre-trip (t (34) = -5.38, p < .001), pre-test to post-test (t (34) = -7.69, p = .003), and pre-trip to post-test (t (34) = -3.25, p <.001). Social Justice subscale: significant increase from pre-test to pre-trip (t (34) = -4.47, p <.001) and pre-test to post-trip (t (34) = -3.89, p <.001).
2008-2009: Personal Competency subscale: significant increase from pre-test to post-trip (t (31) = -6.176, p <.001) and pre-trip to post-trip (t (31) = -6.134, p <.001). Social Justice subscale: significant increase from pre-test to post-trip (t (31) = -6.016, p <.001) and pre-trip to post-trip (t (31) = -5.690, p <.001).
Other findings: In 2006-2007, leaders showed significant change on all subscales except the Interpersonal Relationships subscale. There was also a significant interaction of time by group (leaders, trip participants) for the Personal Competency subscale (F (1, 102) = 9.18, p = .003) and for the Social Justice subscale (F (1, 102) = 13.804, p < .001). In 2008-2009, leaders and participants did not change at different rates across the two times points (pre-trip and post-trip) for any of the subscales.
Discussion
•In 2008-2009,the ASB program moved to a 7-point scale to provide more opportunity for variance in leader responses. This change makes it difficult to compare across the two cohorts (2006-2007, 2008-2009) on the Civic Action subscale and the Interpersonal and Problem-Solving Skills subscale, which were previously on 5-point scales.
•In 2008-2009, the ASB program moved to using “Strongly Disagree” and “Strongly Agree” response choices. This change makes it difficult to compare across the two cohorts (2006-2007, 2008-2009) on Personal Competency and Social Justice scales, which previously used “Completely Disagree” and “Completely Agree”.
•Leaders in both the 2006-2007 and 2008-2009 cohorts report significant increases in both personal competency and social justice, supporting the effectiveness of program leadership training in meeting two of its primary objectives.
Limitations
•In 2008-2009, the ASB program moved from a 5-point scale to a 7-point scale and standardized the response choices across the various subscales.
•In 2008-2009 the percentage of ASB leaders and trip participants who completed the ASB assessments at all relevant time points dropped, possibly impacting the representativeness of the sample.
Future Research
•JMU’s ASB program will continue to collect data using the 7-point scale and standard anchors, allowing for future comparisons across cohorts.
•JMU’s ASB program is embedding assessments within mandatory leader training activities to ensure leaders complete the assessments at all three time points.