Grant Agreement Number ECP-2008-EDU-428045
OpenScienceResources:
Towards the development of a Shared Digital Repository for Formal and Informal Science Education
Validation Plan
Deliverable number / D 6.1Dissemination level / Public
Delivery date / July 2010
Status / Final
Author(s) / Claudio Dondi, Michela Moretti, Chiara Picco / Menon Network
Marlen Goldschmidt, Franz Bogner / UBT
eContentplus
This project is funded under the eContentplus programme[1],
a multiannual Community programme to make digital content in Europe more accessible, usable and exploitable.
Summary
This report represents the first deliverable in the framework of work package 6, Trials and Validation; it aims to provide a detailed description of all aspects of the validation of the OSR project, its approach and its methodology in a wide variety of usage contexts. The validation will be very comprehensive including all major user groups interacting with the system in all foreseeable interactions and contextual settings. These trials are not only meant for validation purposes (focusing on technological issues and user experiences), but also for involving users (museum staff and museum visitors) so that they can provide direction to the project.
This document describes in detail all validation activities that will be undertaken throughout the lifetime of the project. This validation plan includes details on the theoretical approach, on the examples of users’ scenarios prepared for the OSR portal, on the methods of validation. Further, the report presents an overview of the tools that will be used for the validation, on the exact duration and purposes of the validation phases of the project.
Finally, in the last chapter of this document a detailed operational planning will be described. The implementation of the validation process will aim to ensure user acceptance of the proposed methodologies and tools, optimal adaptation to the local environments and realistic evaluation of the effects of use, since as already mentioned, the OSR consortium aims to deploy a heavily user centred approach.
Table of Content
1 Validation Context 9
1.1 VALNET framework and methodology in OSR 11
1.2 Multi-aspect dimensional analysis in OSR 11
1.2.1 Pedagogical 11
1.2.2 Organisational/institutional 13
1.2.3 Technological 14
1.2.4 Economic 14
1.2.5 Cultural 14
2 Validation Objects and Criteria 15
2.1 OSR objects for validation 15
2.1.1 OSR portal as a whole 16
2.1.2 Pedagogical model 16
2.1.3 Organisational model 17
2.1.4 Learning Objects 17
2.1.5 Educational pathway 18
2.1.6 Content sensitive search and retrieval tools 19
2.1.7 Educational metadata authoring system 19
2.1.8 Social Tagging system 20
2.1.9 Community of users 21
2.1.10 Learning experience 21
2.1.11 Authoring/teaching experience 23
2.2 Quality criteria for validation : Key criteria and key measures 24
2.3 External - What the user perceives 25
2.3.1 Searchabilty 25
2.3.2 Relevancy 26
2.3.3 Quality 26
2.4 External –what both the user and the museums perceive 27
2.4.1 Usability 27
2.4.2 Reliability 28
2.4.3 Portability 28
2.4.4 Innovation 28
2.4.5 Belonging, engagement and ownership 29
2.4.6 Cost 29
2.5 External –what the museums perceive 29
2.5.1 Sustainability 29
2.5.2 Transferability 30
2.5.3 Productivity 30
2.5.4 Added Value 30
2.5.5 Internal Variables 31
2.5.6 Capacity 31
2.5.7 Cycle Time 31
2.5.8 Conformance to Standards 32
2.5.9 Security 32
3 Methods and tools 38
3.1 Introduction 38
3.2 Validation : perspectives to take into consideration 39
3.2.1 a) Typology of access to the museum resources: 39
3.2.2 b) Frequency of the visit 39
3.2.3 c) User typology: 39
3.2.4 d) Type of learning 40
3.2.5 e) Process initiators 40
3.2.6 f) Driver of the process 40
3.2.7 g) Reason for accessing resources: 41
3.3 Different users & different perspectives/reasons/motivations (User scenarios) 44
3.4 Validation typologies 52
3.5 Validation of macro-protocols 54
3.6 Training workshops-Validation phase I 55
3.7 In-service seminars (summer school) 58
3.8 Training workshops- Validation II 59
3.9 Web-based evaluation 62
3.10 Qualitative validation 64
4 VALIDATION SOURCES 65
4.1 OSR Partners (OSR Pedagogical / Technical Experts and Museum Educators) 70
4.2 Science Teachers 71
4.3 Students (Formal Learning scenarios) 74
4.4 Schools, Training centres, Universities, Third age Universities 75
4.5 User Groups & Communities: Individual Learners, Families, Science Groups and Socio-cultural Associations 75
4.6 Affiliated Science Centres and Museums, Publishing Houses dealing with science resources 76
4.7 Ministries of Educations, Research Institutions (Private & Corporate) 77
4.8 International Standards Bodies (for example, ISO, SCORM, IMS, HR-XML Consortium, IEEE-LOM relevant to metadata and elearning model development) 78
5 Validation Timeframe 81
5.1 In service seminars (summer schools) 81
5.2 Training workshops- Validation phase I 81
5.3 Training workshops – Validation phase II 82
5.4 Validation timetable 83
6 Validation outcomes 84
7 Literature 85
Introduction
This document aims to provide a comprehensive validation plan describing all validation activities that will be undertaken throughout the lifetime of the project. This plan will include details on: the theoretical approach and methodology, the type of trials, the type of participants and the exact duration and aims of each validation activity. It describes in a very comprehensive way the OSR validation activity including all major user groups interacting with the system in all foreseeable interaction contextual settings. The validation trials are not only meant for validation purposes (focusing on technological issues and user experiences), but also for involving users (museum staff and museum visitors) so that an OSR community of users will be created and can provide direction to the project. Indeed the OSR validation activities are based on a user-centred approach. Moreover, this validation plan offers a platform for the realization of examples of User Scenario that will be explained in detail in chapter 3 of this report.
The validation activity aims to and is made for monitoring and evaluating the progress of the project, collecting data and analysis the results, involving project target groups and stakeholders in the implementation of the project and for the dissemination of the project itself.
The plan reflects the general iterative mode of continuous interaction and exchange between different OSR work packages realized in consecutive cycles, namely definition of Educational Designs (WP2), user requirements (WP3), technology integration and customisation (WP4) and validation activities (WP6).
The validation strategy is based on different steps that aim to answer the following questions:
ü “Why” it is important to conduct evaluation activities (validation aims)
ü “What” has to be validated and on the basis of what criteria (validation objects and criteria)
ü “How” the evaluation activities will be conducted (validation methods and tools)
ü “Who” will provide the required information and data (validation sources)
ü “When” the evaluation activities will be conducted (validation timeframe)
ü “How” the outcomes of validation will be used
The first chapter of the document defines the approach, the methodology and the aims of all the validation activities. In this chapter, a detailed explanation of the VALNET methodology is provided. Briefly, the VALNET methodology allows the OSR Consortium to adopt a multidisciplinary approach in order to analyse the results and the innovation value of the proposed processes and educational scenarios. The main innovation aspects will be identified taking as a focus for the analysis the following five dimensions: a) Pedagogical, b) Organisational/institutional, c) Technological, d) Economic and e) Cultural.
The second chapter provides a detailed definition of the validation objects. This is one of the main parts of the document, indeed before starting any evaluation procedures it is essential to clearly define what has to be validated. The objects of validation start from considering first of all the OSR portal as a whole and then proceeding with the validation of each different, meaningful and innovative part of OSR portal as well as the OSR project purposes. It is important to underline that these objects are based on the descriptions and the definitions stated in the two previous OSR deliverables such as D 2.1 Educational Design and D 4.1 System Specifications and Technical Design. The eleven objects for validation are the following:
1. OSR portal as a whole
2. Pedagogical model
3. Organisational model
4. Learning Objects
- Educational Pathways
6. Content sensitive search and retrieval tools
7. Educational metadata authoring system
8. Social Tagging system
9. Community of users
10. Learning experience
11. Authoring/teaching experience
Moreover, after the detailed explanation of each object, some criteria have been defined and matched with the above mentioned objects. Indeed in order to achieve performance validation, an important step has been to identify key criteria at an external level (relating to both users of OSR and the Museums) as well as at an internal level, where the internal performance of the system itself is validated.
The third chapter describes the methods through which the validation activities will be implemented and it provides also an overview of the tools which will be used for monitoring and collecting data (detailed tools will be designed in the next deliverable D 6.2). This chapter presents some user scenarios which represent a “model categorisation framework” of the different perspective/reasons that OSR users/audience may have for accessing the OSR portal. The selected set of scenarios indentifies and defines some of the more common ones that would be anticipated. Moreover, this paragraph presents the list of the tools which will be used for each phase of the validation activity and a first description of results acquired by using Google Analytics for the evaluation of some web aspects of OSR portal.
The fourth chapter defines in detail the actors that the validation activities will involve for each phase, their characteristics, their level of involvement in the OSR project and their roles in the validation activity itself.
Finally, the fifth chapter presents the operational planning of the validation activity, defining period and timing during the OSR project life time, and the last sixth chapter defines the outcomes which are expected as a result of the validation process.
1 Validation Context
The validation is a very important part of the work in the overall scheme of the project; the validation plan is based on the following two main principles:
· Validation must be performed on the basis of a common methodological framework and according to defined procedures;
· Validation and evaluation methodologies should be focused on providing new solutions and facilitating the dissemination of best practice and exploitation of results.
Due to the complexity and the ambitious nature of OSR goals, the validation plan must take into account different variables, such as:
§ usage contexts (for instance for different science centres and museums),
§ learning objects (such as all the learning materials provided by OSR Content providers),
§ target/users groups (such as teachers, students, general visitor, etc)
In the OSR project the validation activity covers really important role. Indeed, the OSR project tries to investigate on the benefits of enriching digitised scientific objects which are currently dispersed in European science museums and science centres, with well-defined semantic metadata along with social tags, so that they become more widely and coherently available, and better searchable and usable in a variety of learning occasions. The main outcome of the project will be the OSR Portal, a set of customizable learning-oriented discovery services, reliably offered through the web sites of science centres and museums, school portals, visualization environments and other online education publishing services.
The validation activity in OSR aims to contribute to the following main process outcomes:
· Provide the means and methods for data collection and analysis of the results of the OSR project;
· Accompany project management by providing updated information on project achievements and emerging critical aspects vital for the continuation of OSR.
· Generate “learning” among the parties involved
A good validation plan should have two main characteristics in order to reach its goals and to be useful and efficient:
§ It should be based on a user centred approach, because it is necessary to start from and take into account the needs and the points of view of the final users on the products and on results that must be evaluated and validated
§ It should be linked and fitted to the contextual setting where the validation activities take place
The OSR project has a wide European partnership that covers 12 different European countries plus 3 partners outside Europe (2 of them from USA and one from Taiwan). For this reason it using a methodology that allows examining also of the different settings, cultural and linguistic aspects and background of the users is essential. Indeed the validation involves all major user groups interacting with the system in all foreseeable interaction contextual settings. It is important to underline that the validation purpose is not only focused on technological issues and user experiences, but also on the ways of involving users (museum staff and museum visitors) so that they provide direction to the project and its technological and usability results. One of the main scope of the OSR validation trials and of the analysis of the data gathered is evaluating the added value that OSR project wants to provide is the exploration of social tagging and folksonomy as an accessory strategy to the educational content of the science centres and museums. It tries to perform extended comparisons between the access points in existing collections documentation and the terms that are supplied when visitors describe an exhibit along with the relevant physical phenomena.
The OSR validation framework is based on a Formative and Summative approach supported by a multi-dimensional analysis:
1) Formative approach means to accompany the development of the project and suggest changes whenever a problem can be identified for improvement through evaluation. It also means to generate learning among all the actors involved to better achieve the task at hand. The formative approach will focus on the ongoing development and implementation of services and tools; this allows to gathering systematic, structured feedback from partners/experts and potential /actual users as part of an iterative development cycle during the building of the prototype.
2) Summative approach means to check if the expected technical and methodological objectives have been achieved and an impact has been produced. In this case, the focus is on the objectives dealing with the technical functionalities, with the impact being on the prototype and, by extension, the ongoing development of the global system.