Critique of Test 1

Running head: CRITIQUE OF A VIRGINIA STUDIES TEST

Critique of a Fourth Grade Virginia Studies Test

Ginny Wilburn

College of William and Mary

Educational Policy, Planning, and Leadership

The Test

The test used for this critique was created by a fourth grade teacher that solely teaches Virginia Studies to the grade level. The creator of this test has been teaching for three years. I decided to collaborate with her on this project because of my interest in the Social Studies curriculum. I explained my task to her and requested a test that she routinely uses to assess her students. During our conversation about the assignment, she explained the difficulty of teaching the Virginia Studies curriculum to fourth grade students. The Teacher explained her frustration at teaching the Virginia Studies curriculum after the students have been learning nothing but the ancient civilizations in grades one, two, and three. She feels that the Social Studies curriculum builds upon prior knowledge through third grade and then starts an entirely different curriculum in fourth grade with the history of Virginia.

The intended use of the test is to assess the knowledge of fourth grade students on the VS.7 Standard of Learning. The teacher explained that tests of this nature are given at the conclusion of each unit for assessment purposes before a new unit is initiated. She uses primarily select-response questions for the majority of assessments to align with the format of the SOL tests. The assessments are evaluated and remediation efforts are made for those students that demonstrate below grade-level expectations, which is considered a 70% and below. The teacher infers a high degree of consequential validity with this test. She asserts the assessment is used to improve learning due to the remediation efforts made. In order to determine if remediation efforts are successful, another assessment is given to test the same information. She records the assessment data of each of her students on an assessment chart which tracks the percentage of students that fell below the 70% expectation level and the progress they made after remediation.

The format of the test is comprised of 29 select-response questions and a supply-response question offered as a bonus question. The test has a very simple construction. The subject and SOL are referenced at the top of the test, VS.7. The directions are clear and free from misinterpretation. Each question is directly stated in the stem or as an incomplete statement. There is an error in the numbering of test items. Two questions on the test are numbered 11, so they have been referenced in the Table of Specifications as 11a and 11b. Most of the questions with similar intended learning objectives are grouped together. Many of the test items share the same answer choices, and therefore, are not independent of one another. An answer on a previous question may aid the student in answering the next question.

Intended Learning Outcomes and Construct Validity

The standard, VS.7, assesses the student’s knowledge of issues that divided our nation and led to the Civil War. The issues embedded in the standard include the identification of events and differences between northern and southern states that divided Virginians and led to secession, war, and the creation of West Virginia. It also includes the student’s ability to describe Virginia’s role in the war and the major battles that took place in Virginia. Standard VS.7 includes Essential Skills that are comprised of higher cognitive levels, such as drawing conclusions, interpreting ideas and events, and determining cause and effect relationships. The teacher explained the intended learning objectives are based on some, but not all of the cognitive levels listed in the Essential Skills section of the standard. The test items also address the Essential Understandings and Essential Questions of VS.7. The intended learning outcomes are taken directly from the curriculum standards, which are identical to the Virginia Standards of Learning. The instructional materials used to facilitate student learning come directly from the Enhanced Scope and Sequence, provided by the Virginia Department of Education. Trade books as well as primary and secondary source documents are used to help students gain knowledge about the intended learning objectives.

Based on the assessment and information provided by the teacher about the cognitive levels addressed, one can infer a low level of construct validity. The psychological processes addressed in the theoretical framework’s main description and the Essential Skills do not reflect an adequate sampling of the various reasoning processes. The test items correlate with the content of the objectives but the higher cognitive levels are underrepresented which compromises the validity of the assessment. The predictive effectiveness of the test is lessened due to poor sampling. The test format also lowers the construct validity of the test. One of the questions is given the same number (11), so there are actually 29 test items, rather than 28. This affects the scoring of the test. Another issue which lowers the construct validity of the test is the bonus question. The bonus question is not identified as an intended learning objective. It addresses previously learned material and is used for the sole purpose of review. This problem could have been avoided if the bonus question was listed as an intended learning objective for this particular assessment. This affects the scoring of the test and leads to erroneous inferences regarding the interpretations of test results.

Content Validity: Representative Sample of Items

A Table of Specifications was created for this assessment. For this SOL, the student is required to demonstrate knowledge of the issues that divided our nation and led to the Civil War. This objective is divided into two parts. The first two content objectives in bold print address the first part of the VS.7 SOL. The third content objective in bold print addresses the second part of the VS.7 SOL. The content for each objective is listed under the appropriate column with the corresponding test items and cognitive levels. The number of test items for each objective and the corresponding cognitive level has been aggregated in bold print above the individual objectives.

Table 1. Table of Specifications for Grade 4 VS.7a and VS.7b SOL Test

Content / Bloom’s Taxonomy
Knowledge / Comprehension / Application / Analysis / Synthesis / Evaluation
Differences between northern and southern states that led to war VS.7a / X
7 items / X
1 item
Because of economic differences between the North and South, they were unable to resolve their conflicts and the South seceded from the US / X
11b, 12 / X
1
Draw conclusion, make generalization VS.1d
Economy in northern part of the US was more industrialized. / X
2
Economy in southern part of the US was agricultural and relied more on slave labor. / X
3,4
Northern states wanted new states created out of western territory to be “free” states. / X
5
Southern states wanted new states to be slave states. / X
6
Events leading to secession, war, and the creation of West Virginia
VS.7a / X
5 items / X
2 items
Nat Turner led a revolt against plantation owners in Virginia. / X
7
Content / Bloom’s Taxonomy
Knowledge / Comprehension / Application / Analysis / Synthesis / Evaluation
Abolitionists campaigned to end slavery. / X
8
Harriet Tubman supported secret route known as Underground Railroad. / X
26, 27
John Brown led a raid on the U.S. Armory at Harper’s Ferry, VA. He was trying to start a slave rebellion. He was captured and hanged. / X
9 / X
10
Interpret VS.1g
Virginians were divided about secession from the union, which led to creation of West Virginia. / X
11a
Determine cause/effect relationships VS.1b
Describe Virginia’s role in the war, including identifying major battles that took place in Virginia. VS.7b / X
X
14 items
The First Battle of Bull Run (or Manassas) was the first major clash of the Civil War. Confederate General Thomas “Stonewall” Jackson played a major role in this battle. / X
13,14, 28
General Robert E. Lee, Commander of the Army of Northern Virginia, defeated Union troops at Fredericksburg, VA / X
15,21,22,23,24
Content / Bloom’s Taxonomy
Knowledge / Comprehension / Application / Analysis / Synthesis / Evaluation
Richmond was the capital of the Confederacy. It fell to General Ulysses S. Grant and was burned near the end of the war. / X
16,17
Lincoln used the Union navy to blockade southern ports. An imported sea battle between the Monitor (Union) and the Merrimack, (Confederate), two iron-clad ships, took place in Virginia waters near Norfolk and Hampton. The battle was fought to a draw. / X
18,19,20,25

I feel this assessment has a low level of content validity. The content is not representatively sampled at the cognitive levels addressed in the intended learning objectives, listed in the Essential Skills and provided by the teacher. The majority of the test items fall in the Knowledge range and only three test items fall in the higher Analysis range. Although the teacher specifically listed drawing conclusions, interpreting events, and determining cause/effect relationships in her intended learning objectives, there is only one test item addressing each these levels of achievement. There are six content objectives that have only one test item in the Knowledge range. Student achievement can not be correctly assessed with an inadequate sample. A more appropriate sampling of test items would include at least three test items per objective. Transfer of knowledge can not be assessed by this test which is comprised mainly of lower cognitive skills. Selecting more test items that assess the student’s ability to draw conclusions, interpret events, and determine cause/effect relationships would be necessary to raise this test’s level of content validity. The sample of tasks does not reflect the cognitive behaviors the test was intended to assess.

The error in numbering the test items also lowers the content validity. The teacher used a scoring system in which each question weighs one point on a 28-point scale when there are actually 29 questions. The bonus question does not relate at all to SOL VS.7. The factors that contribute to this test’s low level of content validity are the inadequate sampling of test items, underrepresented cognitive skills, irrelevant items, and errors in the scoring procedures.

Reliability: Sources of Measurement Error

There are several examples of potential systematic error in this assessment. There are too few items for many of the objectives. There are only 3 questions out of 29 that focus on reasoning and application skills. The other 26 questions fall at the lowest cognitive skill, Knowledge. The same issues that lower the validity of this test also reduce the level of reliability. Although, the assessment results may provide consistent results with a different sample of the same type of questions, the assessment does not measure the content at the levels identified by the teacher as the intended learning objectives. Therefore, the outcome of test results would result in an inflated achievement score. The test-retest method used to remediate students who did not score at a satisfactory level is an efficient means to achieve student learning, but the content must be aligned with the behavioral objective. The retest is assessing the student’s ability to recall, rather than process and analyze the information.

It is my opinion that this assessment has a low level of reliability. The misalignment between content and cognitive skill yields incorrect inferences to student mastery of the intended learning objectives. The small number of test items for questions 1, 2, 5 -10, and 11a prevent an accurate reflection of student mastery. The sources of systematic error can be corrected to improve the effectiveness of this test. The number of test questions should be increased for the content objectives with less than three items. The test questions should be constructed in a way that assesses higher cognitive skills. The elimination of the bonus question, which does not correlate with the objectives, would yield a more reliable score. The consistency of scores would then reflect mastery of the content at the appropriate cognitive skills.

Scoring System

The scoring system of this test is very simple. Each of the 29 questions counts as one point on a 28 point scale. The misnumbering of question 11 causes the measurement error. All of the test questions are presented in the same select-response format with a low level of difficulty. The bonus question asks the student to list the four major rivers of Virginia, which is material covered in the previous unit. If answered correctly, the student could earn one point. This bonus point inflates the student’s score and translates to an incorrect presumption of mastery. The scoring procedures used for this test lower the test’s validity and reliability. The test does not measure what it intends to measure.

There are several ways to improve the scoring procedures that would increase the validity and reliability of the assessment. First, the scoring values should be printed on the test. The student needs to know how much each question is worth. Higher level test items should carry more weight than questions relating to recall of information. A wider selection of test questions, such as matching, short-answer, and essay, would increase the student’s opportunity to demonstrate their knowledge of the issues that divided the nation and led to the Civil War. For example, a matching section could assess the student’s knowledge of the Civil War battles fought in Virginia, whereas a restricted response question could focus on the differences between the northern and southern economies.