A Study of ALEKS for 8th Grade Pre-Algebra Students
ALEKS Corporation and Ken Petlock
April 10, 2009
Overview
The following is a summary of a paper in progress. The paper details a year-long study of the ALEKS software among three classes of 8th grade students at a middle school in California. This summary begins (part I) with a short description of ALEKS and the theory behind its artificial intelligence, called Knowledge Space Theory. Then, some details are given about the instructor, students, and the specific implementation of ALEKS (part II). The next section (part III) contains an analysis of the data collected in the study, which III) involves two main lines. The first is an examination of the effectiveness of ALEKS as a learning tool (part IIIA). The second is an examination of ALEKS as an assessment / diagnostic tool (part IIIB).
I. Description of ALEKS and Knowledge Space Theory
ALEKS (Assessment and LEarning in Knowledge Spaces) is a Web-based, artificially intelligent assessment and learning system. Its artificial intelligence is based on a field of study called Knowledge Space Theory (KST); for an extensive database of references, see http://wundt.uni-graz.at/kst.php. KST is unique among theories guiding assessment and learning. It allows the representation (in a computer’s memory) of an enormously large number of possible knowledge states organizing a scholarly subject. Rather than giving a score or series of scores that describe a student’s mastery of the subject, KST allows for a precise description of what the student knows, does not know, and is ready to learn next.
According to KST, a subject such as arithmetic, Algebra I, or first-year college chemistry can be parsed into a set of problem types, with each problem type covering a specific concept (or skill, fact, problem-solving method, etc.). A student’s competence can then be described by the set of problem types that she is capable of solving. This set is the student’s knowledge state. The knowledge space is the collection of all of the knowledge states that might feasibly be observed in a population. Each subject typically has 200-300 problem types and several million knowledge states.
A. The assessment
The task of a KST assessment is to uncover, by efficient questioning, the knowledge state of a particular student. Because the number of problem types for a typical subject is large, only a small subset of them can be used in any assessment. So, the successive questions must be chosen cleverly.
At the beginning of a KST assessment, each knowledge state is given some initial probability. A question (problem type) is chosen, and based on the student’s answer, the probabilities are updated: if the student answers correctly, then each knowledge state containing that problem type is increased in probability, and if the student answers incorrectly, then each of those states is decreased in probability. The next question is chosen so as to be as informative as possible (according to a particular measure), and the process continues until there is one knowledge state with a much higher probability than the others. This is the knowledge state assigned to the student.
The process usually takes from 25 to 35 questions. The assessment’s efficiency is significant, considering that the scholarly subject has about 200-300 problem types and its knowledge space has millions of feasible states. This efficiency stems from the many inferences made by the system via the knowledge space.
We note that, because all problem types have either open-ended responses or multiple choice responses with a large number of choices, the chance of a lucky guess is negligible. This is another important contrast with many testing devices.
B. The fringes
Each knowledge state is uniquely specified by two much smaller subsets, the inner fringe and the outer fringe of the state. In everyday terms, the inner fringe is the set of problem types that represent the high points of the student’s competence, and the outer fringe is the set of problem types that the student is ready to learn next. These fringes are typically rather small, and they completely specify a knowledge state that may contain dozens of problem types. The economy is notable. Moreover, the summary is more informative to a teacher than is a grade or a percentile and certainly more useful. For example, the students’ outer fringes can be used by the teacher to gather students in small, homogeneous groups ready to learn a given problem type, fostering efficient learning. Note that the outer fringe of a state can be seen as a formal implementation of the zone of proximal development in the sense of Vygotsky. [reference]
C. The learning mode
The assessment gives precise access to further learning. After being assigned a knowledge state by the assessment, the student may choose a problem type from her outer fringe and begin learning. Once the system determines that she has mastered the problem type, it is added to her knowledge state, and she may choose another problem type to learn from her outer fringe. Subsequent assessments update her knowledge state.
II. Description of the instructor, students, and implementation of ALEKS in the current study
The study took place during the 2007-2008 school year at a Middle School in Northern California which will be referred to as “The Observed Middle School”.
A. The instructor
The instructor, Mr. Petlock, was in his eighth year as a full-time credentialed teacher and his third year teaching with the ALEKS software. He held a multiple subject teaching credential with a supplemental authorization in Mathematics. He was hired about two months into the school year (October 5) to replace a long-term substitute.
B. The students
In this district being studied, 70% of students participated in the free or reduced-price lunch program, compared to the state average of 51% (Source: CA Dept. of Education, 2007-2008). The majority of students were White (57%).
Students at the observed middle school were ability grouped into math classes starting in the 7th grade and distributed into six classes. One of the classes was comprised of the brightest students from the previous year’s 6th grade classes. This class matriculated into an Algebra I class in 8th grade. Students in the remaining five classes were placed in a blocked (two period) Pre-Algebra course. At the end of the 7th grade year, another cut of the brightest students was made to fill one class. These students entered an Algebra IA course in their 8th grade year and then completed Algebra IB during their 9th grade year in high school. The remaining 7th graders, about half of the entire 7th grade class, were then recycled into another Pre-Algebra course the following year (8th grade).
This latter group consisted of the lower half of the (non-special education) 8th grade class, as determined by grades and California Standards Test (CST) scores. These were the 59 students studied.
C. The implementation
Each class period was 43 minutes. Most instructional time occurred in the computer lab. The lab was equipped with a digital projector and a large white board in the front of the room.
Each class had a majority of students using the ALEKS Pre-Algebra product and a minority using the ALEKS Essential Mathematics product. The instruction consisted of a mixture of whole class and small group direct instruction, self-directed learning and peer tutoring, all done chiefly using the ALEKS software. The direct instruction, for example, was initiated by the ALEKS report giving the “outer fringe” of a student, which are the topics the student is deemed “ready to learn” by ALEKS. Using this report, the instructor gave specified instruction on a particular topic to small groups or individuals, allowing for highly differentiated learning. As a result of the diverse learning needs of the class as a whole, the majority of direct instruction occurred in small groups or on an individual level. This instruction was facilitated by a wireless InterWrite Pad [http://www.einstruction.com/products/interactive_teaching/pad/index.html] and the Vision Classroom Management Software [http://www.netop.com/products/education/vision6.htm].
Soon after ALEKS was fully implemented, homework was rarely assigned.[1]
Students’ course grades were given using the following weights: 60% Assessments (taken via ALEKS), 20% daily learning progress (amount of ALEKS topics learned per day), and 20% from other assignments.
In addition to ALEKS, Mr. Petlock used STAR Math, a component of Accelerated Math (by Renaissance Learning), as an independent measure of math ability, knowledge and growth throughout the year. STAR Math is discussed more in section IIIB of this write-up, in which the effectiveness of ALEKS as an assessment tool is discussed.
D. Note about scientific controls in the study
In reviewing this write-up, the reader will note that the students studied fell into two mutually exclusive groups: (i) those who were taught by Mr. Petlock and used ALEKS, and (ii) those who were not taught by Mr. Petlock and didn’t use ALEKS. In other words, from this study, it is difficult to separate the effect of Mr. Petlock from the effect of ALEKS.[2] Rather, what can safely be concluded from the study is that systematic use of the ALEKS software by a conscientious teacher can give the results detailed in the write-up.
III. Data Analysis
There are two lines of data analysis for the study. The first looks at the usefulness of ALEKS as a learning tool during the 2007-2008 school year. The second examines the effectiveness of the ALEKS assessment as a diagnostic tool for predicting performance on the California Standards Test (CST).[3]
The CST is taken each year by nearly all students in California public schools. There is a different CST for each grade level, but each has 65 multiple choice questions and is scored on a scale of 150 to 600. Though the scoring scale is the same for each grade, the tests are criterion-referenced, in that each score is based in part on the assessed mastery of the California state standards for that grade.[4] As a result, scores from the same grade and different years (e.g., 8th grade 2007 and 8th grade 2008 scores) can be compared directly, but scores between grades (e.g., 7th grade and 8th grade scores) cannot. For more information, see the California Department of Education’s website, especially http://star.cde.ca.gov/star2008/help_comparescores.asp .
A student’s score gives rise to his performance level. There are five performance levels: Far Below Basic, Below Basic, Basic, Proficient, and Advanced. The goal of the California Department of Education is for all students to score at the Proficient or Advanced levels, so the percentage of students scoring at these two levels is of special interest.[5] We include this percentage in our analysis below.
A. Learning Improvement Using ALEKS
We first compare the CST results of Mr. Petlock’s students with those of the Pre-Algebra students at the observed middle school the year before, in 2007.[6] Figure 1 shows the mean CST scores (in red) for the 8th graders in Pre-Algebra at the middle school in 2007 and 2008. In 2008, these 8th graders were Mr. Petlock’s students; in 2007, these 8th graders were a group taught by another teacher who did not use ALEKS. This latter group is a reasonable group to which to compare results.
For reference, also shown in Figure 1 are the relevant CST score means for the District, County, and State.
Looking at Figure 1, we see that Mr. Petlock’s students scored much better than the 8th grade Pre-Algebra students from the year before at the observed middle school. In particular, the previous year’s students scored well below the District, County, and State means, and Mr. Petlock’s students scored well above these means, even with increases in these means from 2007 to 2008.
Figure 1: Comparison of 2007 and 2008 8th Grade CST Mean Scores
A (two-tailed) t-test was performed to compare the mean CST scores for the observed middle school’s 8th grade Pre-Algebra students in 2007 and 2008. For this t-test, it was not assumed that the variances of the two populations were equal[7] [reference]. This test is summarized in the table below.
Sample Size / Mean CST Score / Standard Deviation of CST Scores / Results2007
Observed Middle School
Pre-Algebra
(no ALEKS) / 41 / 304 / 35 / t ≈ 3.644
df = 96
p ≈ 0.0004
(two-tailed)
2008
Observed Middle School
Pre-Algebra (ALEKS) / 58 / 337 / 55
So, there was a significant difference in the mean CST score between the 2007 8th grade Pre-Algebra students at the observed middle school (who did not use ALEKS) and those in 2008 (who did use ALEKS).
Note in Figure 1 that the mean for the entire State increased slightly from 2007 (mean=316) to 2008 (mean =320). So, it may be argued that the test in 2008 was “easier” than in 2007, and that this should be considered when comparing the two sample groups. We have chosen to do two separate t-tests, one comparing the 2007 observed middle school’s Pre-Algebra group to the State mean in 2007, and the other comparing the Mr. Petlock’s 2008 Pre-Algebra group to the State mean in 2008. These t-tests are summarized in the table below.