Uniqurate Final Report
Project InformationProject Identifier / To be completed by JISC
Project Title / Uniqurate
Project Hashtag / #uniqurate
Start Date / 1st Sept 2011 / End Date / 28 Feb 2013
Lead Institution / Kingston University
Project Director / Graham Alsop
Project Manager / David Livingstone
Contact email /
Partner Institutions / University of Glasgow, Harper Adams University College and University of Strathclyde
Project Web URL / http://uniqurate.kingston.ac.uk
Programme Name / Assessment and Feedback
Programme Manager / Heather Price
Document Information
Author(s) / Graham Alsop, Paul Neve, Sue Milne
Project Role(s) / Project Director, Principal Developer, Consultant
Date / 14/01/13 / Filename / UniqurateFinalReport.doc
URL
Access / This report is for general dissemination
Document History
Version / Date / Comments
Draft01 / 10/01/2013 / SM
Draft02 / 14/01/2013 / GA
Draft03 / 20/02/2013 / SM
Draft04 / 21/3/13 / DL and GA
Table of Contents
1 Acknowledgements 2
2 Project Summary 3
3 Main Body of Report 3
3.1 Project Outputs and Outcomes 3
3.2 How did you go about achieving your outputs / outcomes? 4
3.2.1 Development 4
3.2.2 Client Partners 5
3.2.3 Dissemination 6
3.3 What did you learn? 7
3.3.1 Evaluation 7
3.3.2 User Testing 7
3.3.3 General 8
3.4 Immediate Impact 8
3.4.1 Partner Institutions 8
3.4.2 Wider Community 8
3.5 Future Impact 9
3.5.1 QTI Adoption 9
3.5.2 Conversion Capability 9
4 Conclusions 9
4.1 General Conclusions 9
4.2 Wider Conclusions 10
4.3 Conclusions for JISC 10
5 Recommendations 11
5.1 General Recommendations 11
5.2 Wider Recommendations 11
5.3 Recommendations for JISC 11
6 Implications for the future 11
6.1 Implications 11
6.2 Further development 12
6.3 Sustainability 12
6.4 Future activity 12
7 References 12
8 Appendices 13
8.1 Appendix A – User Manual 13
8.1.1 Getting Started 13
8.1.2 Editing Questions in Friendly Mode 14
8.1.3 Creating and Editing Tests 19
8.1.4 Editing Questions in Intermediate Mode 20
8.1.5 Expert Mode 20
8.1.6 Equation Editor 22
8.2 Appendix B – Technical Documentation 22
8.3 Appendix C – Project Evaluation Report 22
1 Acknowledgements
This project belongs to the JISC Assessment and Feedback programme, Strand C: Technology transfer, and has been funded by JISC. We are grateful for the support we have had from JISC-CETIS and from our partner institutions, the University of Glasgow, the University of Strathclyde and Harper Adams University College.
The collaboration with our sister project QTIDI has been vital in providing a delivery mechanism for the resources produced by Uniqurate, and hence enabling users to see their work in action.
The support provided by JISC at the CAA conference in July, and also by the conference organisers was very important in bringing the Uniqurate editor to a wider audience.
2 Project Summary
The main objective of this project has been to increase the usage of standards-based electronic assessment tools in Higher Education. In order to do this, we have refactored some existing tools that produce resources compliant with the Question and Test Interoperability Specification 2.1 (QTI2.1) to provide an editor in which novice users can author engaging content with flexible feedback. The functionality of the application has been designed in consultation with the project’s client partners, to enable them to construct the assessment experiences they would like to provide for their students. This includes the ability to combine several different inputs into one question and give feedback which is directly related to the student’s input. The editor also provides facilities for contextualising and re-using existing resources, and access to the XML code of the resource for those who wish to fine tune their content.
3 Main Body of Report
3.1 Project Outputs and Outcomes
Output / Outcome Type(e.g. report, publication, software, knowledge built) / Brief Description and URLs (where applicable)
Publications/events / Conference/journal papers/presentations/online demo recordings – See Section 3.2.3
Specifications / General specifications - http://uniqurate.kingston.ac.uk/
User Documentation - http://uniqurate.kingston.ac.uk/docs.html
Developer details - http://uniqurate.kingston.ac.uk/developers.html
Knowledge built / Ongoing workshop and training sessions
Knowledge built / Blog at QTI Support site and Uniqurate posts @ KU Learning Technology Research Group (http://ltrg.kingston.ac.uk ). Also Uniqurate “official” site, http://uniqurate.kingston.ac.uk and QTI Support blog at http://qtisupport.blogspot.co.uk/
Prototypes / Software output of development sprints – see 3.2.1 for details
Software (binary executable) / Web-based QTI authoring tool(s) usable by novice users, hosted at http://uniqurate.kingston.ac.uk/demo
Software (source code) / Source for above, open source licensed, available from http://sourceforge.net/projects/uniqurate
Publications / User manuals: an introductory leaflet has been prepared downloadable from https://jiscsupport-assessmentandfeedback.pbworks.com/w/file/63439499/UQIntroLeaflet.pdf; a more comprehensive manual is at https://jiscsupport-assessmentandfeedback.pbworks.com/w/file/63829219/UniqurateManual.pdf
Content / E-assessment resources produced by users in authoring tool in QTI format
https://www2.ph.ed.ac.uk/qtiworks/web/anonymous/samples/list
3.2 How did you go about achieving your outputs / outcomes?
3.2.1 Development
At the beginning of the Uniqurate project, we had two question editors available: Aqurate was too simple and Mathqurate too complex. Our challenge was to create something that was “just right” – the Goldilocks editor. We feel that we have succeeded in that aim, but along the way something unexpected has happened: Uniqurate is in some respects more complex than Mathqurate, at least in terms of the content that can be produced. This means that those new to QTI can create rich, flexible resources with good feedback without ever seeing any XML. In the long run, it is expected that this will help to encourage the adoption of QTI outside the existing community.
The full functionality of Uniqurate is described in the User Manual https://jiscsupport-assessmentandfeedback.pbworks.com/w/file/63829219/UniqurateManual.pdf (Appendix A); the technical details of the application are provided online as indicated in Appendix B - Technical Documentation.
The original intention was that components would be identified and then small, wizard-style applications would be created – so, for example, one wizard would be created for authoring a multiple choice question, another for simple maths questions, and so on. These wizards would then be tied together within a menu and/or user interface (UI) and the collection would form the overall Uniqurate application.
Discussions at the start of the year, however, led to the conclusion that this would be too limiting, primarily as it would mean that only one component could be used within a single question –this was not appropriate for some of the users’ use cases. Instead, a platform and UI was developed that would suit multiple components in a question, and which has come to be known as “Friendly Mode”. A round of feedback meetings in February 2012 confirmed the revised model and identified additional components to be incorporated into the application.
Sprint 1 resulted in an “Expert Mode” editor. In this mode, authors edit raw QTI XML and can package additional content or “media” into a question, such as stylesheets, images, video and sound. This feature also enables a question to have metadata packaged with it, currently by editing the manifest file, but ultimately the metadata editing facilities will be ported from Mathqurate and/or Spectatus.
While Expert Mode was considered a relatively low priority in terms of the overall project goal – i.e. an accessible, user-friendly application that enables QTI naïfs to author content – the reason for placing this into the first sprint was to give the user community (particularly those new to QTI authoring) a longer time to be able to “digest” the existing tools and to acquire a better grasp of their own requirements. The “expert mode” was something that the developer could produce with little involvement from the user team – thus what would otherwise have been “dead” time was utilised and created a productive output.
Sprint 2, which commenced following the UI paradigm shift, created a platform that supported a UI where users could drag and drop multiple components onto a question frame and then modify their values in situ to suit the author’s needs. While this development was a complexity that had not been anticipated, it was development time well spent. The first “components” were also added into the UI at this time, providing simple text areas within a question and also a multiple-choice component. This alone provided a significant part of one partner institution’s use case.
Sprint 3 provided the next component which is described as the “triangle”. Consider Ohm’s Law in electronics, i.e. V = IR. There are a host of similar equations across a multitude of subject areas – e.g. distance = speed X time. The “triangle” provides the ability to quickly create such a question and add subject-specific context to it by naming the variables, units and so on. The “triangle” component illustrates our approach to identifying components; specifically, we are seeking components with the widest possible cross-disciplinary utility. By focusing on such, we hope to ensure that the eventual application breaks outside of the limited subject areas who have engaged with QTI to date.
Sprint 4 added an “Intermediate Mode” along with a text entry and an ordering component. We had realised that there was also a need to modify existing questions. Resources created in Uniqurate’s Friendly Mode can be reopened and changed in the normal way. However, because Friendly Mode inserts, and depends on, markers in the code to connect the parts of a component, any question without those markers cannot be edited in Friendly Mode. Intermediate Mode allows users to edit the human-readable aspects of more complex questions that ordinarily would require Uniqurate’s Expert Mode. XML code is hidden and only the human-readable aspects of a question exposed to the user editing it. Although this means that one cannot change the inherent “logic” of questions in Intermediate Mode, one can alter its context – for example, one could change a generic maths question to one more subject specific simply by altering the question preamble text. Although this functionality was not anticipated originally, it represents an important milestone in facilitating the adoption of QTI beyond its current community. It allows for re-use and re-purposing of existing resources that ordinarily might be too generic to be appropriate, but too complex to contextualise.
Sprint 5 added a custom maths component, in response to requests from client partners. (See 3.2.2 below.)
Sprint 6 added test functionality and some component management features. Test functionality was deferred until last on the basis that QTIWorks, (the rendering solution being created by sister project QTIDI) did not render tests until October 2012.
Uniqurate can construct a test consisting of several sections each containing one or more questions. Questions can be repeated or not as required by the author, and the order in which the questions appear can be randomised at runtime. Questions can be edited within the test, using the edit button beside the question, and can be removed if no longer required.
In line with user feedback, several modifications have been made to the Friendly Mode question editor. Components can be removed, copied and moved to different positions in the question sequence by dragging. Feedback can appear either after the component to which it refers or aggregated together at the end of the question; this is particularly useful in questions which have several inputs. In multiple choice components, there is a facility to give the same feedback for all wrong responses, and the author can choose to give no feedback at all. Multiple choice components with text-only content can also be presented as a drop-down list.
3.2.2 Client Partners
We held meetings with the client partners, Harper Adams and Strathclyde, in late January and early February 2012, and again met with them in July 2012. The Strathclyde and Glasgow partners continued to meet through the summer on a number of occasions, to author resources and to provide feedback on the tools, and Harper Adams was represented at the QTI-PET workshop in September at which a number of Uniqurate modifications were suggested and a few bugs caught.
Although one of the stated purposes of Uniqurate was to “de-maths QTI”, it became difficult to ignore the cries from our partners for some degree of maths functionality. The advanced, computer algebra-based complexity of a tool such as Mathqurate was still considered undesirable, given its comparative inaccessibility to a novice user. However, our approach whereby questions that involved some kind of numeric manipulation would be covered by a range of components like the quick maths triangle proved to be too inflexible.
Thus, a custom maths component was added which allows a user to specify a simple maths expression, provide a range of possible values for any variables, and finally provide a range of feedback for different student responses. This was greatly improved by the creation of new code which will parse a maths expression and convert it into a QTI XML fragment. Potentially this could be expanded into a standalone library, as we can see applications outside of Uniqurate for anyone who is converting existing resources into QTI. In Uniqurate itself, however, it enables users to create richer content within the custom maths component: start and end values for variables need not be numbers, but can themselves be maths expressions. In keeping with the ethos of Uniqurate, the end-user sees no QTI – they simply type an expression such as a+b*c-(d/pi).
The following is a list of activities undertaken with our client partners:
Date / Activity13/10/2011 / Visit to Harper Adams UC to meet technical and academic staff involved in project
31/01/2012 / Feedback/update visit to HAUC team members
03/02/2012 / Feedback/update meeting with Strathclyde team members and QTIDI
Summer 2012 / Shazia Ahmed, Sue Barnes and Sue Milne have been meeting to create content for Maths Support using Uniqurate, leading to some useful questions and a number of feedback comments on the application.
21/01/2013 / Shazia Ahmed, Sue Barnes, Kate Durkacz from Napier University and Sue Milne met to continue creating content for Maths Support using Uniqurate
18/02/2013 / Sue Milne led a demo and workshop session for Strathclyde staff
25/02/2013 / Sue Barnes led a demo and workshop session for Strathclyde staff
In February 2013, we have run two sessions at Strathclyde University to familiarise staff with the Uniqurate and QTIDI tools. The first of these was run by Sue Milne, and the second by our “project champion” at Strathclyde, Sue Barnes. Participants have requested logins to QTIWorks, indicating that they will be continuing to use the tools.