Top of Form

Evaluation Kit Acceptance Requirements Sandbox Survey Summaries

General summary: the EvaluationKIT sandbox, i.e. proof of concept, satisfied all criteria proposed by the EvaluationKIT vendor, as outlined below:

UNM Pre-Pilot Trial Experience
Start Date / End Date / Assigned To / Duration / Comments
1 / Test Integrations with UNM Systems - CAS / 8/8/2014 / 8/25/2014 / 12 / Successful!
2 / EvaluationKIT Provisions Account / 8/8/2014 / 8/8/2014 / EK / 1
3 / EK Provides UNM CAS Detail / 8/11/2014 / 8/11/2014 / EK / 1
4 / UNM Configure CAS / 8/11/2014 / 8/13/2014 / UNM / 3
5 / UNM Creates Test User & Provides Test User Info and CAS Inputs to EK / 8/11/2014 / 8/13/2014 / UNM / 3
6 / EK Configure CAS & Creates UNM Test User / 8/13/2014 / 8/15/2014 / EK / 3
7 / UNM Tests CAS with Login / 8/18/2014 / 8/22/2014 / UNM / 5
8 / CAS Test Complete / 8/25/2014 / 8/25/2014 / 1
9 / Test Integrations with UNM Systems - Banner / 8/8/2014 / 8/25/2014 / 12 / Successful!
10 / EvaluationKIT Provisions Account / 8/8/2014 / 8/8/2014 / EK / 1
11 / EK Provides UNM Course & User File Format Details / 8/11/2014 / 8/11/2014 / EK / 1
12 / UNM Prepares Test Course & User Files / 8/11/2014 / 8/13/2014 / UNM / 3
13 / UNM Provides Test Course & User Files to EK / 8/13/2014 / 8/13/2014 / UNM / 1
14 / EK Imports Course & User Files / 8/13/2014 / 8/15/2014 / EK / 3
15 / EK Schedules Online Desktop Share to Review with UNM - Process and Review Resulting Courses/Users / 8/18/2014 / 8/22/2014 / EK / 5
16 / Banner Test Complete / 8/25/2014 / 8/25/2014 / 1
17 / Test Integrations with UNM Systems - Blackboard / 8/8/2014 / 8/25/2014 / 12 / Successful!
18 / EvaluationKIT Provisions Account / 8/8/2014 / 8/8/2014 / EK / 1
19 / EK Provides to UNM Building Block files and instructions for Installing EK Building Blocks for Blackboard / 8/11/2014 / 8/13/2014 / EK / 3
20 / EK call with UNM to install EK Building Blocks / 8/11/2014 / 8/13/2014 / EK / 3
21 / UNM Provides to EK both Course and User Files of Test Bb Course and Test Bb User Enrolled in Course / 8/13/2014 / 8/15/2014 / UNM / 3
22 / UNM Tests Bb User Integration / 8/18/2014 / 8/22/2014 / UNM / 5
23 / Bb Test Complete / 8/25/2014 / 8/25/2014 / 1
24 / Test User Operability / 8/11/2014 / 8/25/2014 / 11 / Successful!
25 / UNM Provides EK with Test Course & User Files, with CAS Users and Bb Courses/Users / 8/11/2014 / 8/13/2014 / UNM / 3
26 / EK Creates Sample Survey, Project, and Imports UNM Test Course & User Files / 8/13/2014 / 8/15/2014 / EK / 3
27 / EK Deploys Project - Surveys Active / 8/15/2014 / 8/15/2014 / EK / 1
28 / UNM Test Users Can Access Surveys via Blackboard & CAS / 8/18/2014 / 8/22/2014 / UNM / 5
29 / User Operability Test Complete / 8/25/2014 / 8/25/2014 / 1

The next twelve pages of survey responses were gathered from sandbox participants, i.e. four groups consisting of Application Integrators, Departmental Administrators, Faculty, and Students, in order to validate sandbox results.

Application Programmer/Integrators

The respective survey responses below were gathered from the three Proof of Concept (Sandbox) application integration experts: Jared Galbraith, IT/Platforms; Linda Mayforth, IT/Applications; and Jonathan Bocock, Extended Learning. Each integration area, i.e. CAS (Central Authentication Service), Blackboard Learn, and Banner proved to integrate and function well with the EvaluationKIT application hosted by the vendor. Note: questions that did not pertain to respective areas were omitted for accuracy and brevity.

Top of Form
[print]
Single response: / Application Programmer/Integrator Evaluation Kit Acceptance CAS Requirements
Respondent id: / 1320034
Start date: / 8/29/14 10:05 AM
Completed date: / 8/29/14 10:06 AM
1. / Please enter your name in the text box below.
Jared Galbraith
2. / Please enter your email address in the text box below.

8. / Is single sign-on supported in a manner consistent with authentication and authorization methods used for other UNM applications?
1. / Yes
9. / Were you able to get support from the vendor if required?
1. / Yes
Bottom of Form /

Bottom of Form

Top of Form
[print]
Single response: / Application Programmer/Integrator Evaluation Kit Acceptance Banner Requirements
Respondent id: / 1346714
Start date: / 9/3/14 9:36 AM
Completed date: / -
Invitee email: /
1. / Please enter your name in the text box below.
Linda Mayforth
2. / Please enter your email address in the text box below.

3. / Did the survey tool integrate with Banner?
1. / Yes
4. / Did survey tool Banner-integration include:
Yes / No / Not
sure
Generating and tracking surveys for specific courses, sections, and instructors / X / - / -
Customizing survey format and content based on instructor and course characteristics / X / - / -
Collecting survey responses only from currently enrolled students / X / - / -
Integrating survey responses and summary data with course grades assigned to specific respondents / X / - / -
Motivating students to complete surveys (for example, via early, delayed, or disabled release of grades) / X / - / -
Enabling release of reports and data to instructors as soon as course grades are submitted / X / - / -
Tell us about any issues or problems that you encountered:
8. / Is single sign-on supported in a manner consistent with authentication and authorization methods used for other UNM applications?
1. / Yes
9. / Were you able to get support from the vendor if required?
1. / Yes
Bottom of Form
Top of Form
[print]
Single response: / Application Programmer/Integrator Evaluation Kit Acceptance Blackboard Learn Requirements
Respondent id: / 1319944
Start date: / 8/29/14 9:42 AM
Completed date: / 8/29/14 9:57 AM
1. / Please enter your name in the text box below.
Jonathan Bocock
2. / Please enter your email address in the text box below.

5. / Did the survey integrate with Blackboard/Learn?
1. / Yes
6. / Did the survey tool Blackboard-integration include:
Distributing and publishing announcements of survey availability
Linking surveys to specific course pages and related tools / Yes / No / Not
sure
Messaging to students regarding survey completion or lack thereof / X / - / -
Messaging to faculty regarding response rates for ongoing surveys / X / - / -
Motivating students to complete surveys (for example, via reminders,
assignment credit, and early, delayed, or disabled release of grades) / X / - / -
Tell us about and issues or problems that you encountered: / - / X / -
X / - / -
The EK Module can be configured to pop up a message that blocks access to learn until the student goes to the survey and completes it or dismisses the popup if it is configured with a "do it later" option. It can be added to both the homepage and inside courses. It will take some discussion as to how to deploy it, because allowing it to be in both places with the popup option on, it can get a little overbearing.
7. / Did the survey tool provide the following turnkey integrations with Blackboard?
Yes / No / Not
sure
Data Integration Features / - / - / X
Student Enrollment Refresh Feature / X / - / -
User Access Features / X / - / -
Gradebook Survey Points Feature / - / - / X
Export Institutional Hierarchy Feature / - / - / X
There are two building blocks. One for user access and one for data integration. We were told we weren't testing the data integration building block at this stage. The 3 "Not sure" answers above all related to that.
8. / Is single sign-on supported in a manner consistent with authentication and authorization methods used for other UNM applications?
1. / Yes
9. / Were you able to get support from the vendor if required?
1. / Yes
Bottom of Form

Departmental Administrators

As part of the IDEA Replacement Proof of Concept (Sandbox), the following survey responses were gathered from departmental Administrators from one branch campus and seven different departments across the University. The volunteer Admins responded to questions concerning an EvaluationKIT presentation that focused on application tasks and functionality that departmental Administrators may perform while using the tool or service.

Report date / Thursday, September 11, 2014 10:32:50 AM MDT
Start date / Monday, June 16, 2014 12:40:00 PM MDT
Stop date / Tuesday, September 30, 2014 12:40:00 PM MDT
Stored responses / 8
Number of completed responses / 8
Number of invitees / 11
Invitees that responded / 8
Invitee response rate / 72.73%
Question 1
Please enter your name in the text box below.
Marjorie Crow
David Sanchez
Toni Unwin
Yvette Sanchez sedillo
Laura Encinias
Alejandro Flores
Margaret Anaya
Cheryl Martin
Question 2
Please enter your email address in the text box below.


;





Question 3
Was survey question creation and management efficient, flexible, and easy-to-perform?
Question 4
Did the survey tool capture and was it able to report open-ended/qualitative comments?
Question 5
Did Evaluation Kit support surveys containing questions defined by different levels of the University such as the entire university or campus, a specific college, a specific department, a specific course coordinator, and a specific instructor?
Question 6
Did the survey tool repeat questions for multi-instructor courses? (i.e. is it able to evaluate two or more instructors within a course and produce independent evaluations).
Question 7
Did the survey tool have the ability to use older surveys as templates for newer surveys?
Question 8
Did the survey tool provide easy-to-use dashboards to monitor surveys in progress and report response rates?

Faculty
As part of the IDEA Replacement Proof of Concept (Sandbox), the following survey responses were gathered from University Faculty from one branch campus and five various departments across the University. The faculty responded to questions concerning an EvaluationKIT presentation that focused on application tasks, and functionality, that Faculty would perform while using the tool or service.
Report date / Wednesday, September 10, 2014 1:54:47 PM MDT
Start date / Monday, June 16, 2014 12:40:00 PM MDT
Stop date / Tuesday, September 30, 2014 12:40:00 PM MDT
Stored responses / 7
Number of completed responses / 7
Number of invitees / 8
Invitees that responded / 7
Invitee response rate / 87.5%
Question 1
Please enter your name in the text box below.
Kevin Comerford
Miriam Chavez
Kristopher Goodrich
Mary Margaret Rogers
Julie DePree
Christopher Holden
Aeron Haynie
Question 2
Please enter your email address in the text box below.




jdepree@unm,edu


Question 3
Was survey question creation and management efficient, flexible, and easy-to-perform?
Question 4
Did the survey tool capture and was it able to report open-ended/qualitative comments?
Question 5
Did Evaluation Kit support surveys containing questions defined by different levels of the University such as the entire university or campus, a specific college, a specific department, a specific course coordinator, and a specific instructor?
Question 6
Did the survey tool repeat questions for multi-instructor courses? (i.e. is it able to evaluate two or more instructors within a course and produce independent evaluations).
Question 7
Did the survey tool have the ability to use older surveys as templates for newer surveys?
Question 8
Was report and information output easy and efficient to generate and easy to interpret for multiple uses, but especially for instructor reviews and for course and instructional improvement?
Question 9
Did the survey tool provide easy-to-use dashboards to monitor surveys in progress and report response rates?

Students

During the sandbox, Students were also given a chance to experience the EvaluationKIT service in a mock Faculty Course Evaluation class survey, which was also conducted during the Proof of Concept (Sandbox). The following survey responses were gathered from the Students. Note: student names and email addresses have been omitted.

Report date / Wednesday, September 10, 2014 1:51:29 PM MDT
Start date / Monday, June 16, 2014 12:40:00 PM MDT
Stop date / Tuesday, September 30, 2014 12:40:00 PM MDT
Stored responses / 59
Number of completed responses / 56
Number of invitees / 85
Invitees that responded / 58
Invitee response rate / 68.24%
Question 1
Please enter your name in the text box below.
Names Omitted
Question 2
Please enter your email address in the text box below.
Email addresses Omitted
Question 3.
Which interface did you use to access the survey:
Levels
Question 4.
If you used an iPhone or Android, did you encounter any trouble downloading, installing or using the app?
Question 6.
Did you receive positive communications reminding you to respond to the evaluation?
Question 8.
If applicable, were you able to save an uncompleted evaluation and return at a later time to complete it?
Question 9.
If applicable, were you able to resume online survey completion in the event of a disconnection?
Question 11.
Was single sign-on supported in a manner consistent with authentication and authorization methods used for other UNM applications?
Question 13.
On a scale of 1 - 5 please rate the survey tool for usability:
Question 15.
Did the survey tool provide a point of contact for service related incidents, assistance, general questions and problem resolution?