Santa MonicaCollege
California chemistry diagnostic test
ADDENDUM to
Request for Approval for the Renewal of the CCDT
January 2004
Esau Tovar
Faculty Leader, AssessmentCenter
ADDENDUM to
Request for Approval for the Renewal of the CCDT
The information presented below is submitted following the memorandum (Poggio & Gasnapp, 2003) written to Edward Karpp of GlendaleCommunity College and Conrad Amba on December 15, 2003. In the report, Santa MonicaCollege was referenced as having problems in the areas of bias, cut score validity, and disproportionate impact. To this end, the following actions were taken and are fully discussed in this addendum:
Bias:Clarifying that all Caucasian students used were female; an additional 3 African American students were recruited to review the CCDT, and their responses added. None of the three African American students indicated any problems/concerns with bias.
Cut Score Validation:
Per Poggio and Glasnapp’s (2003) observations and a review of new consequential-related data, it is recommended that the current cutoff score of 22 points be decreased to 20 points. This will alloa an additional 9% of students to gain entry into Chemistry 11, while still having a reasonable opportunity to complete the course successfully.
Disproportionate Impact:
Although Santa MonicaCollege was listed in the report as having a problem in this area, a narrative of the problem was not included. However, in light of this potential problem, and given the proposal to decrease the cutoff score to 20 points, a new analysis suggests that this action alone will result in increasing the number of impacted students (i.e., females and minorities). Additionally, it is re-affirmed that the Chemistry Department will convene a panel of diverse faculty, staff, and students to fully study this issue and issue recommendation to go into effect in fall 2004.
Test Bias Assessment
A diverse panel of 23faculty and students was convened to assess the CCDT for biases in content and language. Committee members were instructed to rate each test item and instructions by answering the following question: Does the test item contain content or language that would be deemed unfair or distasteful to a particular group of examinees (e.g., ethnicity, gender, age, disability, culture, spoken language)? Members were provided with a rating form and asked to indicate if the item in question was “free of bias,” “unfair,” or “offensive.” Furthermore, they were instructed to provide a brief explanation if they believed an item was unfair, offensive, or otherwise presented some type of concern.
Panel Composition
The panel was composed of ethnically diverse full-time and adjunct chemistry faculty and faculty from other disciplines (counseling, assessment, and disabled students), staff, and students of variant ages.
Ethnicity & Gender
The specific ethnic composition of the group was as follows:
- 10 Caucasian/White Females
- 5 Asian Females
- 3 Hispanic Males
- 2 Hispanic Females
- 2African American Males
- 1 African American Female
Evaluation Results
Table 8 presents a summary of the test bias evaluation conducted by the panel members. Of the 44 CCDT items: 35 were found to be free of biases, nine (#2, 14, 27, 28, 30, 32, 35, 37, and 39) were found to be “unfair,” and no items were found to be “offensive.” Listed below are the reasons why nine items were rated unfair:
- Item 2:
- Item 14:
- Item 27:
- Item 28:
- Item 30:
Student: [Item 30, option B] is misspelled.
Chemistry 11 Student: #30 is unfair because not everyone knows what a lump sugar or fine powder is.
- Item 32:
- Item 35:
Student: In my opinion, it is unfair to ask the student to determine the length in the units of the ruler, since the distance between each two line segments changes.
- Item 37:
Chemistry 12 Student: Perhaps students not familiar with metric system are at a disadvantage.
Chemistry 11 Student: Some international student may not know the exact meaning of the word “width.”
Chemistry 11 Student: In the US, we aren’t taught the metric system from a young age, so it might be difficult for some to determine the width in centimeters.
Chemistry 11 Student: #37 is unfair because Chem 11 students don’t have to know the approximate width of the paper.
Chemistry 11 Student: In my opinion it is unfair to ask the student to estimate the page’s width in metric units, especially if he is not used to the metric system. Students’ ability to approximate lengths should not be tested.
- Item 39:
Taken together, the comments above do not sufficiently raise a “red flag” so as to prevent the college from continuing to use the CCDT in chemistry placement. The comments do not specifically indicate that the language or content is offensive to any particular group of students. While Item 37 may presume that test-takers should be familiar with the metric system, it is the faculty’s expectation that students taking science courses, including chemistry, should be familiar with it at even the most basic level. As for Item 35, which was deemed by one rater as hard to read, the CCDT is available in large print format for individuals that request it or have difficulty reading the “normal” test.
Table 8. CCDT Bias Analysis SummaryItem / Rating / Item / Rating
Free of Bias / Unfair / Offensive / Free of Bias / Unfair / Offensive
1 / 23 / 0 / 0 / 23 / 23 / 0 / 0
2 / 22 / 1 / 0 / 24 / 23 / 0 / 0
3 / 23 / 0 / 0 / 25 / 23 / 0 / 0
4 / 23 / 0 / 0 / 26 / 23 / 0 / 0
5 / 23 / 0 / 0 / 27 / 22 / 1 / 0
6 / 23 / 0 / 0 / 28 / 22 / 1 / 0
7 / 23 / 0 / 0 / 29 / 23 / 0 / 0
8 / 23 / 0 / 0 / 30 / 20 / 3 / 0
9 / 23 / 0 / 0 / 31 / 23 / 0 / 0
10 / 23 / 0 / 0 / 32 / 21 / 2 / 0
11 / 23 / 0 / 0 / 33 / 23 / 0 / 0
12 / 23 / 0 / 0 / 34 / 23 / 0 / 0
13 / 23 / 0 / 0 / 35 / 21 / 2 / 0
14 / 20 / 1 / 0 / 36 / 23 / 0 / 0
15 / 23 / 0 / 0 / 37 / 17 / 6 / 0
16 / 23 / 0 / 0 / 38 / 23 / 0 / 0
17 / 23 / 0 / 0 / 39 / 22 / 1 / 0
18 / 23 / 0 / 0 / 40 / 23 / 0 / 0
19 / 23 / 0 / 0 / 41 / 23 / 0 / 0
20 / 23 / 0 / 0 / 42 / 23 / 0 / 0
21 / 23 / 0 / 0 / 43 / 23 / 0 / 0
22 / 23 / 0 / 0 / 44 / 23 / 0 / 0
Cut-Score Validation
In line with the observations provided by Poggio & Glasnapp (1993) and findings related to the consequential validity study conducted in fall 2003 an analysis was conducted to determine if by decreasing the current cutoff score of 22 to 21, 20, 19, and 18 would significantly impact chemistry placement and success rates.
Taking into account the consequential related findings from fall 2003, it was determined that of the 36 Chemistry 10 students indicating they should have been placed into a higher-level class, 89% completed their fall 2003 Chemistry 10 class successfully. Only two students withdrew from it, one obtained and “F” and another, a “D”. However, it was only students with a CCDT score of 20 or 21 whom obtained at least a “C” grade. These findings once again indicate that on average, students completing the CCDT test tend to be more successful overall compared to all Chemistry 10 students college-wide.
Chemistry 10 Outcomes for Students Indicating Wanting a Higher Level PlacementGrade / Frequency / Percent / Valid Percent / Cumulative
Percent
Valid / Fail / 1 / 2.8 / 2.9 / 2.9
D / 1 / 2.8 / 2.9 / 5.9
C / 7 / 19.4 / 20.6 / 26.5
B / 9 / 25.0 / 26.5 / 52.9
A / 16 / 44.4 / 47.1 / 100.0
Total / 34 / 94.4 / 100.0
Missing / Withdrew / 2 / 5.6
Total / 36 / 100.0
Based on these new findings and on Poggio and Glasnapp’s observations, it is recommended that the cutoff score of 22 points be lowered to 20. An analysis of all CCDT placement records to-date suggest that by lowering the cutoff, an additional 9% of students testing can be expected to gain entry into Chemistry 11.
Disproportionate Impact
The original CCDT report submitted in November 2003 for review indicated that in the immediate future, the Chemistry Department would convene a panel of diverse individuals to fully assess disproportionate impact findings at SMC and to develop a plan of action to address the disproportionate number of Latino and African American students failing to obtain the minimum of 22 points needed for Chemistry 11 placement. However, in view of the Poggio and Glasnapp’s (2003) recommendations and the new findings discussed in the previous section; i.e., the lowering of the CCDT cut score to 20 points, it is estimated that the disproportionate impact for ethnicity could be lowered from a 21.6% difference to 17.1%. The new cutoff of 20, however, will only reduce the Chemistry 11 placement difference between males and females by a fraction of a point (from 4.98% to 4.6%). While these differences are not substantial enough, a reduction in the cutoff score is but one of possibly other actions we must conduct to address disproportionate impact. Our current plan is to convene a panel of faculty, staff, and students in spring 2004 to begin studying this issue further and set in motion plans to address disproportionate impact in fall 2004 and beyond.
CCDT Disproportionate Impact Assessment for EthnicityEthnicity / Descriptives / Chemistry 10
Cutoff 22 / Chemistry 11
Cutoff 22 / Chemistry 10
Cutoff 20 / Chemistry 11
Cutoff 20
White/Asian
(majority group) / N / 111 / 231 / 87 / 255
% within course placement / 32.5% / 67.5% / 25.4% / 74.6%
% of Total / 27.1% / 56.3% / 21.2% / 62.2%
African American/
Latinos
(minority group) / N / 46 / 22 / 39 / 29
% within course placement / 67.6% / 32.4% / 57.4% / 42.6%
% of Total / 11.2% / 5.4% / 9.5% / 7.1%
Total / N / 157 / 253 / 126 / 284
% within course placement / 38.3% / 61.7% / 30.7% / 69.3%
% of Total / 38.3% / 61.7% / 38.3% / 61.7%
EEOC Guideline Percentage / 54% / 59.7%
Difference / 21.6% / 17.1%
CCDT Disproportionate Impact Assessment for Gender
Gender / Descriptives / Chemistry 10
Cutoff 22 / Chemistry 11
Cutoff 22 / Chemistry 10
Cutoff 20 / Chemistry 11
Cutoff 20
Male
(majority group) / N / 72 / 162 / 49 / 165
% within course placement / 40.4% / 58.5% / 38.9% / 58.1%
% of Total / 15.8% / 35.6% / 12.0% / 40.2%
Female
(minority group) / N / 106 / 115 / 77 / 119
% within course placement / 59.6% / 41.5% / 61.1% / 41.9%
% of Total / 23.3% / 25.3% / 18.8% / 29.0%
Total / N / 178 / 277 / 126 / 284
% within course placement / 100.0% / 100.0% / 100.0% / 100.0%
% of Total / 39.1% / 60.9% / 30.7% / 69.3%
EEOC Guideline Percentage / 46.5% / 46.5%
Difference / 4.98% / 4.6%
1