Annual Report on Assessment of Degree Programs, AY 2008-09
Name of Program: / Sociology / College: / CoASPrepared by: / Linda Shaw, Kristin Bates, Sharon Elise,
Richelle Swan, Jill Weigt / Date: / 5/21/09
Department Chair/Program Coordinator
Email Address: / / Extension: / 8026
PART A (Required by May 21, 2009 – last day of Spring semester)
1) / Please describe the student learning outcomes you focused on for assessment this year, the assessment activities you used to measure student learning in these areas, and the results of your assessments. Please also comment on the significance of your results.
Student Learning Outcomes
Rather than assessing two Student Learning Outcomes (SLOs) for our 2008-2009 assessment, the Departmental Assessment Team sought to gain a more comprehensive perspective on how our SLOs were being integrated into our courses (see Appendix A for a listing of SLOs from the 2008-2010 University Catalog for the Sociology and Criminology and Justice Studies majors as well as the Master of Arts in Sociological Practice (MASP) degree, along with a listing of the core courses in which the SLOs should appear). Before embarking on successive years of assessments to determine our success in imparting our SLOs to students, we thought that it is important to conduct a baseline assessment to determine where and how our SLOs are being communicated and taught to our students. Thus, our intention was to develop a better understanding of our successes in and challenges to incorporating departmental SLOs into our work that would provide a foundation for the assessments we will conduct in subsequent years. Specifically, the assessment efforts for this year focused on if and how our SLOs were apparent in our course syllabi.
Assessment Activities
Team Formation: We formed an assessment team comprised of five faculty members: Linda Shaw, Kristin Bates, Sharon Elise, Richelle Swan, and Jill Weigt.
Evidence: We based our assessment on syllabi to gain an overview of the extent to which SLOs were being communicated, delivered, and assessed in our courses. Using syllabi for all undergraduate (including both the Sociology and Criminology and Justice Studies majors) and graduate courses in the MASP program taught during the Fall 2008 and Spring 2009 semesters, we evaluated whether the appropriate SLOs were present and whether instructors articulated how the SLOs were being covered and assessed in each course. We evaluated syllabi for each course taught during this time period unless the class was taught both Fall and Spring by the same instructor. In these cases, we reviewed the Spring syllabi.
Assessment Tool: The Assessment Team developed a rubric that helped us evaluate our syllabi with respect to the integration of our SLOs. We created a rubric based on collective discussion (see Appendix A) which required the evaluator to: 1) list the SLOs appropriate to each course based on those listed in the University Catalog; 2) determine whether a given SLO was mentioned in the syllabus under review; 3) identify how the SLO was delivered (via lecture, readings, discussions, movies, etc.) and whether that determination was made explicitly or implicitly in the syllabus; and 4) identify how the SLO was assessed in the course (by quizzes, tests, papers, discussion posts, etc.) and whether that determination was made explicitly or implicitly in the syllabus. We also collected data on other SLOs included in the syllabi but not explicitly delineated in the departmental SLOs.
Norming Process: In order increase the reliability and validity of our assessment, we met for a norming session before carrying out the review of the syllabi. Each member of the team evaluated the same two syllabi and discussed their respective results. This process helped us to hone the assessment instrument and to achieve consensus about how we would apply the categories of analysis in a consistent manner during the next stage of the assessment.
Assessment: In the next stage of the process, we divided the syllabi between the team members. Each team member specialized in evaluating one or two particular types of courses: core courses in the Sociology and Criminology and Justice Studies majors, the MASP program, or elective courses. We assessed the syllabi for a total of 94 sections of courses offered during the 2008-2009 academic year: 27 sections of core Sociology courses, 18 sections of core Criminology and Justice Studies courses, 40 sections of electives courses for both majors, and nine MASP courses offered over the two semesters.
Data Analysis: After we reviewed the syllabi, we created a database to determine the number and percentage of course syllabi which mentioned each of the required SLOs and which linked SLOs to assessment strategies. Findings from this analysis are presented below.
Faculty Reflections: In order to disseminate our findings to the faculty and in conjunction with our Program Review process, we held an all-faculty forum at the end of the Spring semester that included both tenure-track and lecturer faculty to share the results of our analysis. Since one purpose of the forum was to respond to challenges that faculty might have in incorporating SLOs into their syllabi, we also asked faculty to analyze their own syllabi using the same rubric that the Assessment Team had used. We then talked in small groups and to the group as a whole about the experience (see Appendix C for transcript of the large group discussion). This opportunity to reflect resulted in important suggestions for revisions of our SLOs and for ways that faculty can support one another as we refine the process of communicating and assisting our students to master the Departmental SLOs (see Question # 3 for suggestions for future action that came from faculty reflections during the All-faculty Forum).
Results and Discussion of Syllabi Analysis
The Extent to Which Core Courses Reflect Corresponding Student Learning Outcomes
Table 1 reports the absolute numbers and percentages of syllabi that mentioned each of the required SLOs for our two undergraduate majors.
Table 1: Number and Percentage of Syllabi for Core Courses in the Sociology and Criminology and Justice Studies Majors that Mentioned Required SLOs for Academic Year 2008-2009
SLO / Sociology
N = 27 / Criminology and Justice Studies
N = 18
Mentioned / Not Mentioned / Mentioned / Not Mentioned
Diversity of Social Experience / 18
94.7% / 1
5.3% / 14
100% / 0
0%
Theory / 16
69.6% / 7
30.4% / 12
92.3% / 1
7.7%
L/A/A/C Scholarship / 4
50% / 4
50% / 4
50% / 4
50%
Research strategies / 9
90% / 1
10% / 7
100% / 0
0%
Construct theory / 1
10% / 9
90% / 0
0% / 4
100%
Ethical and sj implications / 17
65.4% / 9
34.6% / 12
92.3% / 1
7.7%
Advocate for social change / 14
77.8% / 4
22.2% / 6
85.7% / 1
14.3%
While there were 27 sections of core courses for Sociology and 18 sections of core courses for Criminology and Justice Studies, no single core course requires all of the SLOs. For this reason, the N for each SLO is less than the overall number of sections of core courses offered. For example, the number of sections of Sociology core courses that should have mentioned the first SLO is 19 (18 mentioned this SLO, 1 did not). The number of sections of Criminology and Justice Studies core courses that should have mentioned this first SLO is 14 (all 14 did include this SLO).
Sociology
From Table 1, we can see that some SLOs are better represented in the syllabi than others. The first SLO, “Analyze and interpret the diversity of social experience using a sociological perspective, especially as they relate to race, class, gender, age, sexual preference, religion, and nationality,” was mentioned in 14 of the 15 required courses (94.7%). The fourth SLO, “Understand and employ a range of research strategies—quantitative and qualitative—and their applicability to particular research questions, theoretical orientations, and social contexts,” was mentioned in nine of the 10 required courses (90%). Several of the SLOs were mentioned in a majority of the courses in which they were required but not with the frequency of SLO 1 or SLO 4. The second SLO, “Assess the merits of competing theoretical approaches to formulate empirically researchable questions about social life” was mentioned 69.6% of the time. The sixth SLO, “Understand the ethical and social justice implications of sociological inquiry,” was mentioned 65.4% of the time. And the seventh SLO, “Apply sociological theory and empirical research to advocate for positive social change,” was mentioned 77.8% of the time. Finally, two of the SLOs were mentioned much less often. The third SLO, “Locate, analyze, assess, and communicate sociological scholarship,” was only mentioned in 50% of its required courses, and the fifth SLO, “Construct informed theories of social behavior from systematic observation of social life,” was only mentioned 10% of the time.
Criminology and Justice Studies
As with Sociology, SLOs in the Criminology and Justice Studies major were represented better in some syllabi than in others. Two of the SLOs were represented 100% of the time in its required courses--SLO 1, “Analyze and interpret the diversity of social experience associated with criminology and social justice issues, especially as they relate to race, class, gender, age, sexual preference, religion, and nationality,” and SLO 4, “Understand and employ research strategies and their applicability to particular research questions, theoretical orientations, and social contexts.” Three SLOs were represented well but not 100% of the time. SLO 2, “Assess the merits of competing theoretical approaches to formulate empirically researchable questions about criminology and social justice concerns,” and SLO 6, “Understand the ethical and social justice implications of criminology and social justice inquiry,” were mentioned 92.6% of the time. SLO 7, “Apply criminology and justice studies theories and research to advocate for positive social change,” was mentioned 85.7% of the time. Finally, as with Sociology, two of the SLOs (the same two) were mentioned much less often. SLO 3, “Locate, analyze, assess, and communicate criminology and social justice scholarship,” was mentioned in 50% of its required courses, and SLO 5, “Construct informed theories of social behavior associated with criminology and social justice from systematic observation of social life,” was not mentioned in a single course in which it should have been mentioned.
Master of Arts in Sociological Practice
Table 2 shows the results for the evaluation of MASP syllabi for their effectiveness in mentioning SLOs.
Table 2: Number and Percentage of Syllabi for Courses in the Masters in Sociological Practice that Mentioned SLOs for Academic Year 2008-2009
SLO / MASP
N = 9
Mentioned / Not Mentioned
Develop Critical Reasoning in Service Delivery / 2
100% / 0
0%
Application of Social Theory to Social Policy / 2
100% / 0
0%
Learn Effective Communication Skills / 2
100% / 0
0%
Acquire Advanced Quantitative and Qualitative Research Skills / 4
100% / 0
0%
Appreciate the varying perspectives, experiences, and value differences of those from different gender, race, ethnic, class, and age groups. / 4
66.7% / 2
33.3%
Because the MASP program does not have its SLOs explicitly outlined in the University Catalog, and therefore does not have core courses specifically linked to its requisite SLOs, the analysis of the MASP SLOs must be understood differently. Table 2 reports that the first four SLOs, 1) develop critical reasoning in the area of the delivery of human and criminal justice services; 2) practically apply social theory to the development and appraisal of social policies and programs; 3) learn effective communication skills to become strong advocates for sociologically informed decision making; and 4) acquire advanced research skills in applied and evaluation research, using both quantitative and qualitative methods were mentioned 100% of the time; and the fifth SLO, “Appreciate the varying perspectives, experiences, and value differences of those from different gender, race, ethnic, class, and age groups,” was mentioned 66.7% of the time. However, these results are really only accurately reporting the number of courses that included these SLOs in their syllabi, NOT the number of courses that should have included them but did not. Until the MASP SLOs are linked more specifically to each of the courses, it is impossible to completely evaluate which courses have and have not satisfactorily incorporated their required SLOs.
In summary, the Sociology and Criminology and Justice Studies major core course syllabi were quite good at articulating some of the SLOs, while there is much room for improvement articulating others. In the instance of one of the SLOs for both majors (SLO 5) the representation was so poor that the faculty may wish to revisit this SLO to determine if it does, indeed, represent a core concern or value for each of the majors. The MASP program needs to fully articulate its SLOs in the University Catalog and link them more explicitly to its core/required courses.
The Extent to Which Syllabi Report that SLOs are Explicitly Assessed in Core Courses
Tables 3 and 4 report the absolute numbers and percentages of core courses in Sociology, Criminology and Justice Studies, and the MASP program that explicitly or implicitly reported assessing the requisite SLOs. While determining the extent to which SLOs are assessed (i.e., how faculty determine what students know and are able to do upon completion of courses)
cannot be fully evaluated by examining syllabi, we thought it may provide a preliminary way