SPSCC College Wide Abilities: Ethics- Academic

Reliability Analysis of Evaluation Rubric

April 27, 2010

SPSCC Office of Institutional Research

1

“Ethics” Rubric

Reliability Analysis

Five individuals from academic disciplines evaluated five different samples of student work using the ethics rubric. The work samples were chosen to represent understanding of ethics in an academic (non technical) manner. The samples included two assignments from Cultural Anthropology and three student essays from members of the SPSCC Psychology Club. The assignment was a response to the circumstances surrounding the Psychology Club’s denial of permission to travel in March 2010 to the Western Psychological Conference in Cancun, Mexico.

The evaluation rubric contains four possible outcomes. With five evaluators, the total number of outcomes available for scoring was 20. There was moderate disagreement between the evaluators as to which outcomes should be scored, so just 60% of the outcomes were used in the reliability analysis. Table 1 below shows the inter-item correlation matrix for the evaluators of the rubric. The scores given by Evaluator 1 were negatively correlated with Evaluators 2 and 3, which indicates a substantial level of disagreement. The rest of the Evaluators’ scores were positively correlated. Table 2 shows the Cronbach Alpha for the ethics rubric to be .640, which is just below the acceptable level of .70.

Table 1. Inter-Item Correlation Matrix

Table 2. Reliability Statistic for Five Evaluators

Given the negative correlation of scores between Evaluator 1 and Evaluators 2 and 3, some additional analysis was conducted. Table 3 shows what the value of Cronbach’ Alpha would be if Evaluator 1’s scores were eliminated. In this case, the value rises to .718, which is above the acceptable level. It is appropriate to consider Evaluator 1 an outlier and move forward with just the scores from Evaluators 2-5; however, when the analysis is actually conducted again there are further problems. The strength of the internal consistency does not hold using just four sets of scores as the missing values become a greater factor, and the Cronbach Alpha score drops to .597, which is lower than the original.

Table 3. Cronbach’s Alpha with Removal Scenarios

The conclusion from this reliability study is that the rubric may have some weaknesses. This could be exposing flaws with the particular work that was selected for evaluation, or a simple difference in personal opinion, or a lack of specificity within the outcomes. It is important to note that this study did not include an evaluation of ethical understanding in technical fields. An additional study using only technical course material is recommended to further refine the strength of the rubric as a college wide evaluation tool.

1