Departmental/Office Assessment Plan
Department/Office: Vanderbilt Institutional Research Group (VIRG)
Departmental Contact: Name: Roberta Bell Phone: 615.322.4359 Email Address:
Date Submitted: September 1, 2008
Mission Statement:
The VIRG mission is to be a central resource for decision-making support by providing empirically-based data and consultation to University Offices through the collection, analysis, and distribution of information concerning Vanderbilt and the higher education community.
Goals / Desired Outcomes / Assessment Methods and Procedures / Results / Planned Improvements Based on Assessment Results
1. Collect and organize institutional and peer data. / A. Gather and organize data from multiple internal data sources.
B. Gather and organize data from multiple external sources.
C. Administer institutional surveys to monitor student and faculty activity and satisfaction while ensuring data validity. / A. Integrate existing University databases to facilitate synthesis of information from multiple sources.
B.1.Establish and maintain data exchange relationships with outside consortia.
B.2.Stay current with changing resources and technologies for acquiring external data.
C.1. Develop and maintain surveys to streamline data collection.
C.2. Monitor survey activity and develop strategies to maximize response rates. / A.1. Integrated 18 internal data feeds with an increase of 2 over the past year. All are operational.
A.2. Created 3 internal data collection tools with increase of 2 (for Graduate School) over past year.
A.3. Managed FIS collection of faculty information from all Schools except Medical.
A.4. Held training sessions for FIS data collection resulting in moderate success. Reliable data entry is still an issue.
B.1. Maintained all data exchange relationships except with one consortium that was re-evaluated and dropped due to lack of peer institutions.
B.2. Received and processed data from external data sources including 2 new sources.
C.1.a. Maintained 9 surveys and added 1 this year. Changed 3.
C.1.b. Undertook major revision of Spring Survey. C.1.c. Evaluated PG survey and took it off-line. C.1.d. Mapped all DOS and VIRG surveys by institutional goals addressed.
C.2.a. Score tool utilized to monitor response rates and strategies adjusted accordingly.
C.2.b. Response rate for ASQ increased (automation and timing).
C.2.c. Response rate for GSS increased from 58.4% in 2007 to 61.8% in 2008 (number of mailings increased and emailed on Tuesdays when best results were achieved).
C.2.d. Response rate for CIRP survey increased from 89.7% to 92.7% (held session on Friday at end of orientation for the first time).
C.2.e. Response rate for Spring Survey increased from 33.6% in 2007 to 36.4% in 2008. / A.1. More are planned in housing and space.
A.2. To stay consistent with VIRG mission, the building of new tools may be phased out.
A.3. Incorporate the Medical School Data.
A.4. Automated rules to be built to help with reliability of data entry.
C.1. Major revision of PG survey planned. Revision will consider aligning to QEP goals and AAUDE exchange. Revisions of ASQ and GSS surveys.
C.2.c. Automate GSS communications.
C.2.e. Automate Spring survey communications.
2. Efficiently distribute institutional and peer data for use in University decision-making purposes. / A. Develop new self-service systems of information distribution.
B. Inform the appropriate decision-makers of the availability of data posted on VIRG website.
C. Maximize accessibility of data contained on the VIRG website.
D. Provide timely and accurate updated information.
E. Maintain website security protocols. / A. Develop and maintain interactive analytic web tools.
B.1. Analyze usage reports to refine distribution list and target relevant audiences.
B.2. Send out VIRG update to announce new reports available.
C.1. Ensure ease of navigation on VIRG website.
C.2. Routinely review and refine search terms.
C.3. Routinely inform users of maintenance being performed that may affect availability of information.
C4. Utilize security settings for wider distribution of reports.
D.1. Routinely review Master Calendar for scheduled reporting activities.
D.2.Create syntax to routinize data processing.
D.3. Develop templates to allow for automated production of reports.
D.4. Conduct internal review of all distributed reports before releasing.
E.1. Maintain a list of current users.
E.2. Monitor web activity. / A. Developed 5 new query tools on VIRG in the past year. Added features or elements to 4 older query tools.
B.1. Analyzed usage reports for annual report and reviewed distribution lists, adding IPEDS reports for more audiences.
B.2.a. Continued to send out VIRG blurbs on new reports.
B.2.b. ASQ delayed due to external approval process. B.2.c. Worked on consolidating blurbs and releases.
C.1. Redesigned and implemented new factbook.
C.2. Reviewed search terms on somewhat regular basis.
C.3. Used messaging feature of the VIRG website to announce upcoming and current maintenance.
C.4. Developed a faculty group.
D.1. Every staff meeting reviewed Master Calendar.
D.2. Created syntax for production of Vanderbilt Visions and GSS datasets, and created views for Admissions data.
D.3. GSS and Vanderbilt Visions Fall Benchmark group reports created in Access.
D.4. All reports and blurbs were shared before release.
E.1.a. Housed user lists in VIRG and reviewed security settings on a monthly basis.
E.1.b. Automated data checks on staff who move departments to catch changes in positions that would require changing security settings.
E.2. Monitored web activity regularly. / A.1. Continue to develop and add features as requested. A.2. Develop error reports to catch issues related to our data in our tools.
B.2.b. Intervene if delay occurs again.
C.2.Conduct search term reviews every month
C.3. As new tools are built, integrate messaging feature.
C.4. Review the faculty group to include administrators who are faculty.
D.2. Refine Spring Survey syntax and routinize Clearinghouse upload.
D.4. Target at least two people for initial review with a deadline to increase efficiency.
E.2. Set up Google Analytics for Assessment website.
3. Coordinate Vanderbilt’s external reporting. / A. Continue to be a central reporting source.
B. Ensure accurate and consistent reporting.
C. Maintain a process of planning and prioritizing to meet deadlines. / A.1. Continue to work with upper administration to serve as a central reporting source.
A.2. Educate Vanderbilt community regarding VIRG’s role in external reporting.
B.1. Continue to monitor data reliability and consistency from sources internal to Vanderbilt.
B.2. Review data for consistency in reporting to organizations external to Vanderbilt.
C.1. Routinely review Master Calendar for scheduled reporting activities.
C.2. Coordinate timely collection of data from other Vanderbilt offices. / A.1.a. Managed collection and reporting of University-wide data to many highly visible external organizations such as IPEDS, NSF, AAU, SACS, Common Data Set, college guide books, national ranking publications, higher education journals, and newspapers.
A.1.b. Played a major role in educating and advising university administration of mandatory IPEDS changes (e.g., race reporting).
A.2. Education took place informally during meetings and during collection and preparation of external reports. VIRG updates highlighted VIRG’s role in external data reporting.
B.1.a Maintained relationships and establish new relationships with various internal offices to collect ‘official’ University data to ensure reliability and consistency.
B.1.b. Intervened occasionally to ensure consistency with prior year reporting.
B.2.a. Routinely conducted final analyses for reliability and consistency of data reported to multiple organizations external to Vanderbilt.
B.2.b. Reviewed peer data reporting practices to ensure uniformity in reporting Vanderbilt data.
C.1. Monitored 30 external reports on the Master Calendar on a weekly basis.
C.2.a. Organized and drove the collection of necessary data in a timely manner from other internal offices to ensure timely reporting.
C.2.b. Met all final external reporting deadlines. / A.2. Consider distribution of VIRG newsletter.
4. Provide models and conduct analyses for use in empirically-based decision-making. / A. Develop and validate predictive models for EM.
B. Provide structured reporting focused on goals to inform decision-making.
C. Make recommendations based on data/facts and knowledge of the University as well as the existing research in higher education. / A.1. Develop predictive models.
A.2. Perform routine validation of model accuracy by testing combination of variables for highest predictability.
A.3. Compare projections to actuals on an annual basis.
B.1. Develop benchmarking schemas that track progress toward stated goals.
B.2. Consider program and institutional goals in structuring the presentation of findings.
C.1. Utilize executive summaries in reporting to aid administration in synthesizing and contextualizing information.
C.2. Perform data-mining and compose white papers when relevant to allow for more in-depth exploration of topics. / A.1.a. Developed PGPA equations for 08 Admissions cycle,.
A.1.b. Developed a new logistic regression equation based on ASQ responses.
A.1.c. Developed new academic achievement yield models.
A.1.d. Refined and improved class yield modeling tool.
A1e. Developed model to categorize geo-markets.
A.1.f. Developed a preliminary high school rating model.
A.2. Tested all models and refined for accuracy.
A.3. Predicted models compared to actual data on an annual basis and discrepancies reconciled.
B.1. Developed an EM goals analytic tool with future goals specified on a 5 yr basis.
B.2.a. Communicated EM goals to staff and when relevant EM goals were reflected in VIRG analysis.
B.2.b. Structured all QEP reporting around stated goals.
B.2.c. Revised Spring Survey to balance and align items to address QEP goals.
B.2.d. Contributed to institutional mapping of VIRG and DOS surveys onto QEP goals.
C.1. Added executive summaries to all reports created. Staff worked to improve the poignancy of executive summaries.
C.2. Performed data mining/analyses with the Parent Survey, ASQ, and QEP reporting. ASQ data mining resulted in a new ASQ Minority Report. / A.1. Plan to develop new retention model.
A.1.a. For 09 Admissions cycle, create Arts & Sciences PGPA model. Also, accelerate PGPA schedule.
A.1.d. Automate part of yield modeling tool into EMMA.
A.1.e. Refine geo-market model based on validity study (i.e., feedback from Admissions).
A.1.f. Develop an alternative high school rating model.
B1. Modify based on current data, goals met and revised goals.
C.2.a. Mine data on ASQ related to yield of African American students for fall 08.
C.2.b. Compose Spring Survey white papers that explore topical undergraduate experience issues.
C.2.c. Mine data for retention study.
5. Consult with other University offices promoting the exchange of data, knowledge, and expertise to advance institutional effectiveness. / A. Manage workload and communication to ensure continued collaboration with a broad base of clients within the university.
B. Stay informed about University activities, needs and concerns both internal and within the context of higher education.
C. Link clients to data, knowledge, and expertise both internal to VIRG and across the University.
D. Consult on statistical and data-based decisions to help offices use information resources for strategic decision-making. / A.1. Effectively manage contact with other offices by maintaining practices of promptly replying to requests, and out-of-office messaging.
A.2. Ensure appropriate prioritization of work/consultation requests through review of their relevance to VIRG’s mission.
A.3. Evaluate all new projects for adequate staffing prior to agreeing to participate.
A.4. Maintain individual, internal task lists and conduct annual reviews of completed requests and clients served.
B.1. Maintain relationships and practice active outreach to other offices.
B.2. Obtain and share relevant information regarding University needs and activities at weekly staff meetings and other methods of dissemination (i.e., email forwarding, verbal, review of University publications).
B.3. Maintain subscriptions and memberships with relevant organizations.
C.1. Maintain and reference an internal VIRG skill and topical expertise grid to link client requests with appropriate skill sets.
C.2. Maintain a working inventory of survey activity, and other knowledge and data sources available within and outside of the University.
D.1. Maintain a working knowledge of data structures and technologies that facilitate data access and flexibility.
D.2. Document data definitions and data processing procedures. / A.1. Routinely practiced prompt replies and out-of-office messaging with no negative feedback from clients.
A.2. VIRG staff consulted with Director for prioritization and relevance of ad hoc requests.
A.3. Director successfully reviewed all projects for adequate staffing.
A.4. Routinely utilized tasks list and completed annual reviews.
B.1.a. Hosted happy hours and brown bag lunches along with an open house event after relocating.
B.1.b. Collaborated with other offices on assessment projects including the Assessment website.
B.2. Held weekly staff meetings to discuss current projects and current issues regarding the University and higher ed.
B.3. Maintained memberships in AIR, SAIR, TENNAIR, AAUDE, Eduventures and subscriptions to Chronicle, Academe, and other online publications.
C.1. In progress
C.2.a. Assisted in mapping major survey questions to QEP goals.
C.2.b. Consulted with other offices to provide information found in survey data for decision making and evaluation
D.1.a. Worked with other offices to understand data processes and data entry that affect data quality (i.e. SAT Profile project, EMMA).
D.1.b. Continued to learn new technologies to facilitate data access and flexibility.
D.2. Created HELP files for University databases in VIRG warehouse. / C.1. VIRG staff will complete an expertise profile.
D.2. HELP files will be updated.
1