QA Data Analysis
Page 1
QA Data Analysis
2015-2016 Award Year
Slide 1:
This recorded Microsoft LiveMeeting presentation provides the post-conference training promised during the Pre-conference in Las Vegas, late last year.
I am Holly Langer-Evans and I am joined by DavidRhodes, Together, we will provide your school guidance on analyzing your school’s 2015-16 data which we promised in November.
Instructions for down loading slides and talking notes – I assume the slides and transcript/talking notes will be posted on the website itself and NOT otherwise linked to the recording
Slide 2:
The second slide of this presentation specifies the goal of this training exercise,
The goal of this training is to help every QA school identify their own school verification criteria that add the most value in terms of correcting FAFSA inaccuracies unaddressed by federal verification.
The purpose of the analysis exercise is to help QA schools identify the school criteria that corrected need-based aid awards that the federal edits didn’t. We hope that schools choose to retain some of these as discretionary criteria after the QA Program and its exception from verifying records selected by the CPS end.
Slide 3:
As indicated on slide 3, this training will present a “how-to” for the transitional analysis we encourage all QA schools to conduct. The analysis uses the Pell Improper Payment report to identify school criteria that add the most to federal verification. The goal of the analysis is to allow your school to consider retaining its most proficient school criteria, in this regard, as discretionary criteria after the conclusion of the QA Program.
Slide 4:
The six steps to this transitional analysis are presented on slides 4 and 5.
Most schools, hopefully, completed the first three steps last fall, but just as a reminder, you do need to: create an ASCII file identifying records that were selected for school verification during the 2015-16 award year and the specific school verification criteria each record met; load those ISIRs into the Tool; and validate the records within the Tool before you can analyze your data in the manner we describe in this training.
Slide 5:
It’s the next three steps, steps 4 through 6, listed on slide 5 that this training focuses on.
By running the Pell Improper Payments Report and focusing on the records not selected for federal verification your school will be able to make a more informed choice about which school verification criteria it may want to retain after the end of the QA Program.
Slide 6:
Starting on slide 6, this presentation provides illustrated instructions on how to run and use the Pell Improper Payments report in the manner David just described.
After logging into the ISIR Analysis Tool, with your 2015-16 data loaded, select analyze data. From the domain tree on the left of your screen select IA Tool 15-16; Reporting Objects; and finally Pell Improper Payments.
Clicking on Pell Improper Payments will launch the report. Depending on your browser settings, you may get a message about the screen reader “JAWS.” If you do, simply click on “OK” to proceed to the report. Again, depending on your browser settings, you may not see this message. In which case, you will proceed directly to the report.
Slide 7:
Slide 7 is a screen shot of the Pell Improper Payments report’s initial interface, after it loads, That is, after selecting it and selecting “OK” on the JAWS dialog box if presented.
To run the report simply click on the “Run” icon located on the top left of the browser window.
Slide 8:
Slide 8 presents the result of clicking “Run.” Note the results are in the form of an active report and populate a new browser window.
Depending on the browser program you are using and your settings, this “Results” window may be embedded within Report Assistant (as in the example on slide 8) or be located in a separate browser window. This distinction affects your ability to save the report as an html file for future use, but these differences do not affect how you use the report to conduct the recommended analysis.
Therefore, we will continue the training on how to use the Pell Improper Payments report immediately after running it, which is the same in all browser environments. David will begin to cover that topic on the next slide.
Slide 9:
Thanks, Holly.
Slide 9 begins our instructions on how to use the Pell Improper Payment Report to identify school criteria that you may want to retain after your participation in the QA program ends.
In order to focus on the records not selected by the CPS for verification, we will need to apply a filter.
Step #1 in this process is to click on the small triangle immediately to the right of the column heading “SELECTED_BY_CPS” within the Active Report’s browser window.
Doing so will generate the larger dialog box to the far left of slide 9. In this dialog box perform step #2 by moving your cursor over the word “Filter” and clicking.
This will generate the second, smaller, dialog box immediately to the right. Step #3 is clicking on “Equals” in this second dialog box. Doing so will generate a third dialog box where you will be able to specify the components of your filter selection. You may need to scroll left/right or up/down to locate this third dialog box.
After locating the third, “Filter Selection” dialog box, perform step #4, populating the value of the “SELECTED_BY_CPS” filter with ‘No.’ You accomplish this by clicking on the box with a downward facing triangle on the far right, and selecting ‘No.’
Complete the process by applying the filter by clicking on the “Filter” button indicated by step #5.
Slide 10:
Slide #10 illustrates how to begin building the table that will present the analysis of your school criteria.
Your selection of the field, “INST_VERIF_CRIT” within the “Group By (X)(COU)” dialog box will complete the specification and result in the Rollup table displayed on the next slide.
Generate this table after applying the filter selecting only those records not selected for federal verification. The instructions for performing this prerequisite filtering were provided on slide #9.
We want schools to look at the effectiveness of their own criteria only among records NOT selected by the CPS because after the QA Program ends, schools will be required to verify all records selected by the CPS.
Slide 11:
Slide 11 displays this initial rollup table.
Note that every unique combination of characters in the verification criteria column is displayed on the left and a count of the number of records with the identified combination is displayed on the right.
Slide 12:
But we are interested in more than simply a count of the number of records that met a given combination of school criteria. We are interested in the effect changes to ISIR data among the aid applicants with the given combination of codes had on student eligibility for need-based aid.
The fields “ABS_EFC_CHANGE” and “ABS_PELL_CHANGE” provide two such measures, the absolute value of changes in EFC and Pell Grant amounts between the initial and paid on transactions.
Slide 12 illustrates how to add these two fields to our rollup table.
First, you edit the contents of the roll up table by clicking on the icon with different length horizontal bars located in the top left corner of the initial results window.
Second, select “Add(Y)” in the initial dialog box that displays.
Finally, select the two desired additional measures “ABS_EFC_CHANGE” and “ABS_PELL_CHANGE” in the subsequent dialog box that displays after selecting “Add(Y).”
Slide 13:
The screen shot on slide 13 displays the initial result of adding the two absolute value measures to the roll up table.
Note the only initial consequence of adding our two fields to the table is two additional columns with the same row counts we had to start with.
In order to change the content of information in the two additional columns to the average value on the new measures among the records in a particular row, first click on the Greek letter Sigma, then select “Avg.”
Slide 14:
Slide 14 provides helpful suggestions in evaluating your results.
First, sort your results by the number of records with the displayed school verification code or codes. To do this click on the downward pointing triangle to the right of “inst_verif_crieria” and select “Sort descending.”
The average of effect on EFC and Pell are in terms of absolute values, so positive and negative values don’t cancel each other out. Both of these measures reflect aggregate effect on need-based aid, rather than exclusively potential over- or under-awards.
Consider keeping the school criteria or combination of criteria with the highest averages on these two measures. As these are averages they represent the average effect on aid eligibility of verifying each applicant who meets a given criteria or combination of criteria.
Note the number of records for each row. Make sure your school can commit to the level of effort of adding this number of aid applicants to discretionary verification.
Slide 15:
Slide 15 displays our rollup table after we have:
•Changed the measure to average. Note that the label to the right of the summation sign incorrectly indicates “Count.” This is due to a bug in the software. While, the label here will not change when the user changes the measure in the manner described on Slide 13, the output does change correctly.
•Sorted the results (in descending order) based on the number of records in each row
•Drawn a blue box around the rows with 20 or more records – the threshold we suggest using for “trusting” the results
•Inserted blue arrows pointing to the largest changes in EFC and Pell, depending on meaning of the institutional criteria listed on these two rows, these may be good candidates for retention as discretionary verification
Slide 16:
Slide 16 summarizes how we suggest analyzing your school’s results in the Pell Improper Payments Report run in the manner we described in this training.
First look for criteria associated with large average changes to EFC or Pell
Keep in mind the number of students behind the results. The more students involved increases the size of the issue and the amount of additional work involved in adding said group via discretionary verification. Again don’t get too worked up about results based on fewer than twenty students.
Slide 17:
That concludes our training on identifying the best school criteria to keep after returning to federal verification.
If you have any questions do not hesitate to contact your regional representatives
Also feel free to contact David directly if you have a question that concern solely analysis.
Contact information is provided on slide 17.
END