Audit of ACT’s Career Programs Assessment Test ATB Program ED-OIG/A03-B0024
ED-OIG/A03-B0024
Dr. John Roth
Principal Consultant Placement Programs
American College Testing, Inc.
P.O. Box 168
Iowa City, IA 52243-0168
Dear Dr. Roth:
This Final Audit Report (Control Number ED-OIG/A03-B0024) presents the results of our audit of American College Testing’s Ability-to-Benefit (ATB) policies and procedures for its Career Programs Assessment Test (CPAt).
A draft of this report was provided to ACT. In its response, ACT concurred with our recommendations for Finding No. 1 through Finding No. 5, and noted that it has either already implemented or will be implementing actions to address the issues raised in these findings. ACT also appears to concur with the basis for Finding No. 6, but provided an alternate corrective action plan for its resolution. We summarized ACT’s response after each finding, and a copy of the complete response is provided as an attachment to this report.
AUDIT RESULTS
We found that American College Testing (ACT) was not in compliance with its agreement with the United States Department of Education (ED), for the approved use of the CPAt, and with applicable laws and regulations.
Under 34 C.F.R. § 668.145(e) —
The approval of a test may be withdrawn if the Secretary determines that the publisher violated any terms of the agreement . . . or that the information the publisher submitted as a basis for approval of the test was inaccurate . . . .
Based on the number and substance of our audit findings, if ACT does not implement all of this report’s recommendations for findings 1 through 5, we recommend that ED withdraw its approval of the CPAt test for ATB testing purposes.
We also found that the agreement and procedures for ACT’s Assessment Test, which is also approved by ED for ATB testing purposes, do not meet Federal requirements.
Finding No. 1 ACT’s Analysis of CPAt ATB Test Scores Was Not Submitted Timely
We found that ACT did not submit its required three-year analysis of CPAt ATB test scores to ED in a timely manner. ACT was required to submit the analysis to ED no later than October 1999. In November 2000, we were informed by an ACT official that ACT had not prepared an analysis of student CPAt ATB test scores and that it would begin to do so. ACT submitted its analysis to ED in January 2001.
Under 34 C.F.R. § 668.150(b)(8) —
The agreement between a test publisher and the Secretary provides that the test publisher shall . . . [t]hree years after the date the Secretary approves the test and for each subsequent three-year period, analyze the test scores of students to determine whether the test scores produce any irregular pattern that raises an inference that the tests were not being properly administered, and provide the Secretary with a copy of this analysis . . . .
An ACT official stated that the report was not prepared timely because ED did not provide guidance for conducting the evaluation and reporting its results.
Without timely analysis of CPAt ATB test scores, ACT cannot ensure that instances of improper test administrations are identified or that prompt corrective actions are taken. Consequently, a high volume of ATB testing abuses may occur that result in funds being awarded, under Title IV of the Higher Education Act of 1965, as amended (HEA), to ineligible students.
Recommendation:
1. We recommend that the Chief Operating Officer (COO) for Federal Student Aid (FSA)[1] require ACT to strengthen its management controls to ensure that any future analysis of student CPAt ATB test scores required under 34 C.F.R. § 668.150(b)(8) is prepared and submitted to ED in a timely manner.
ACT’s Reply
ACT concurred with this recommendation. In its response, ACT asked about the date that it must submit the next analysis of student ATB test scores for the CPAt and about the period of testing dates that the next analysis should include.
OIG’s Response
ED approved ACT’s CPAt test for ATB purposes on October 25, 1996. Consequently, under 34 C.F.R. § 668.150(b)(8) —
§ ACT’s first analysis of students’ CPAt ATB test scores should have covered the three-year period ended October 24, 1999.
§ ACT’s next analysis of students’ CPAt ATB test scores due to ED should cover the three-year period from October 25, 1999, through October 24, 2002.
ACT should contact the appropriate program officials at ED regarding the required submission date for this analysis.
Finding No. 2 ACT’s Analysis of CPAt ATB Test Scores Did Not Contain the True CPAt ATB Population
ACT’s Director of Statistical Research informed us that ACT could not identify directly, based on its data, the universe of CPAt ATB test scores for its three-year analysis of CPAt ATB test results, covering the period January 1997 through December 1999. Instead, ACT identified the universe by identifying the universe of CPAt examinees who —
§ Received a valid test score for all three CPAt testing sections, and
§ Indicated on the CPAt answer sheet that they had not graduated from high school.
As a result, the universe used for ACT’s three-year analysis improperly excluded CPAt ATB examinees who —
§ Did not complete the educational background question on the CPAt answer sheet;
§ Indicated, on the educational background question, that they have at least a high school education; or
§ Did not receive a valid test score on all three testing sections.
In addition, we found that the universe may have incorrectly included test results for non-ATB CPAt examinees with less than a high school education who received a valid test score on all three testing sections.
Federal regulations applicable to this finding are in 34 C.F.R. § 668.150(b)(8), as cited above for Finding No. 1.
ACT’s Director of Statistical Research stated that, at the time the universe for the three-year analysis was determined, the data needed to identify CPAt ATB examinees directly was not readily available.
We concluded that, since there is not an appropriate universe, there is no assurance that ACT’s three-year evaluation of CPAt ATB test records is reliable.
Recommendation:
2. We recommend that the COO for FSA require ACT to initiate appropriate action to ensure that ACT bases any future analysis required under 34 C.F.R. § 668.150(b)(8) on a complete and accurate universe of test data.
ACT’s Reply
ACT concurred with this recommendation.
Finding No. 3 ACT Does Not Monitor CPAt ATB Retest Administrations
ACT does not have processes in place to identify and inform institutions of CPAt ATB administrations conducted in violation of its retesting procedures. Our review of ACT’s file documentation for a sample of 25 examinees from the universe of 629 examinees who retested on the same CPAt form at the same institution in consecutive months during the period July 1, 1997, through June 30, 2000, revealed that 20 (80 percent) were retested improperly. Also, our analysis of the universe of 9,179 examinees who were administered a retest of the CPAt examination during the period July 1, 1997, through June 30, 2000, identified approximately 400 instances in which examinees were improperly tested twice on the same CPAt form, during the same month, at the same institution.[2]
The CPAt Test Administration Manual allows an examinee to retest if his or her performance on the test was affected by conditions other than ability. For example, an examinee would be able to retest if his or her performance was affected by an illness or by an interrupted test session, or if there is reason to believe a significant change occurred in the examinee’s level of knowledge and skills. According to the CPAt Test Administration Manual, if retesting is warranted —
An alternate form of the CPAt (a form other than the one most recently administered) is used for the retest (i.e., retesting must “cycle through” all CPAt forms available from ACT). If all CPAt forms available from ACT have been administered to the examinee, a minimum of 30 days must pass between administrations of the same CPAt form.
Federal regulations state —
§ “The agreement between a test publisher and the Secretary provides that the test publisher shall . . . [c]ertify test administrators who have . . . [t]he necessary training, knowledge, and skill to test students in accordance with the test publisher’s testing requirements . . . .” 34 C.F.R. § 668.150(b)(2)(i).
§ “The agreement between a test publisher and the Secretary provides that the test publisher shall . . . [d]ecertify a test administrator for a period that coincides with the period for which the publisher’s test is approved if the test publisher finds that the test administrator . . . [h]as repeatedly failed to give its test in accordance with the publisher’s instructions . . . .” 34 C.F.R. § 668.150(b)(3)(i).
§ “An institution may use the results of an approved test to determine a student’s eligibility to receive Title IV, HEA programs funds if the test was independently administered and properly administered.” 34 C.F.R. § 668.151(a)(2).
§ “The Secretary considers that a test is properly administered if the test administrator . . . [a]dministers the test in accordance with instructions provided by the test publisher, and in a manner that ensures the integrity and security of the test . . . .”
34 C.F.R. § 668.151(d)(2).
ACT’s Manager of Production Services explained that ACT does not monitor CPAt ATB retesting activity because its certified independent test administrators (ITA’s) are responsible for conducting retests in compliance with its rules.
ACT’s failure to identify and report retest administrations not conducted in accordance with its established procedures may result in invalid ATB determinations, improper admission of students, and disbursements of Title IV, HEA program funds to ineligible students. Also, because ACT does not monitor its certified ITA’s retesting activity, ITA’s who should be decertified may continue to perform CPAt ATB testing.
Recommendations:
We recommend that the COO for FSA require ACT to —
3.1 Strengthen its management controls to ensure that ITA’s follow the already established procedures for conducting a CPAt ATB retest.
3.2 Establish a system to monitor and review the testing practices of each certified ITA, and decertify any ITA who repeatedly violates or compromises its approved testing procedures.
3.3 Establish a system for identifying and reporting retest errors to ensure that institutions have accurate and timely information at the time that eligibility determinations are made.
ACT’s Reply
ACT concurred with the recommendations and stated that on May 1, 2002, it will activate a computerized process for applying retest rules to each CPAt ATB answer sheet it scores. ACT’s reply described the process that ACT planned to implement, to identify and report retest errors.
OIG’s Response
ACT’s planned actions must ensure it will establish a system for identifying and reporting violations of the retesting guidelines for CPAt ATB tests described in Attachment A of its response to the Draft Audit Report. The CPAt ATB retest guidelines contained in Attachment A have not yet been reviewed and approved by ED.
Finding No. 4 ACT’s CPAt Database Contains Incomplete and Inaccurate Test Date Information
The test date field maintained in ACT’s CPAt database is a four digit numeric field that includes only the month and year data for test administrations. Our review of ACT’s file documentation for 95 of 73,455 CPAt ATB tests conducted during the period July 1, 1997, through June 30, 2000, revealed that the test date recorded on 9 CPAt ATB test answer sheets (9.5 percent) did not agree with the test date in the CPAt database. In seven instances the wrong test month was recorded in the CPAt database, in one case the wrong test year was recorded, and on another occasion both the wrong test month and year were recorded.
Data from CPAt test answer sheets that are scanned to file, for review and update to the CPAt database, includes the examinee’s name, Social Security number, date of birth, race, educational background, and item responses for the CPAt’s Language Usage, Reading, and Numerical Skills Tests. Although each examinee records the full testing date on the CPAt (including the month, day, and year the test was administered), that full date is not recorded in a format that can be scanned to file.
ACT uses a “Site Header Sheet” to update the CPAt database with the general date (month and year only) for each group of CPAt ATB answer sheets received from an ITA. The “Site Header Sheet” is completed by ACT staff, and it is also used to collect the state code, test center code, number of answer sheets to be scored, and any applicable special codes. Special codes are used to designate CPAt ATB tests, un-timed tests, and the ITA.
The erroneous test date records in ACT’s CPAt database for nine of the ATB tests we reviewed appear to be the result of human error on the part of ACT staff, when completing the “Site Header Sheets.” Without complete (month, day, and year) and accurate test date records, ACT cannot adequately monitor CPAt ATB administrations for compliance with its retesting procedures.
Recommendation:
4. We recommend that the COO for FSA require ACT to initiate appropriate action to ensure that complete and accurate CPAt ATB test dates are maintained in its CPAt database.
ACT’s Reply
ACT concurred with this recommendation.
Finding No. 5 ACT Needs to Improve Procedures Used to Certify CPAt ITA’s