SOCIABLE – ICT-PSP No. 238891
SOCIABLE DELIVERABLE D7.1
“Results Validation Plan”
Project Acronym / SOCIABLEGrant Agreement No. / 238891
Project Title / Motivating platform for elderly networking,
Mental reinforcement and social interaction
Deliverable Reference Number / SOCIABLE_WP7_D7.1
Deliverable Title / “Results Validation Plan”
Revision Number / V2.0
Deliverable Editor(s) / AUSL
Authors / Chiara Zaccarelli (AUSL), Stelios Pantelopoulos (SLG), Paolo Mattarelli (CEDAF), Michele Fabbri (CEDAF), Roberta Annicchiarico (FSL), Francesco Barban (FSL), Olga Lymperopoulou (HYGEIA), Reyes Moliner (PREVI), Rocio Saragoza (AIJU)
Project co-funded by the European Commission within the ICT Policy Support Programme
Dissemination Level
P / Public / P
Statement of originality:
This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both.
Revision History
Revision / Author(s) / Organization(s) / Date / Changes0.1 / Chiara Zaccarelli, Stelios Pantelopoulos / AUSL, SLG / 10/01/2010 / Structure, Table of Contents and Inputs based on the SOCIABLE Protocol/Evaluation Methodology
0.2 / Chiara Zaccarelli, Stelios Pantelopoulos / AUSL, SLG / 18/02/2011 / Revision of Table of Contents
0.3 / Stelios Pantelopoulos, Paolo Mattarelli, Michele Fabbri / SLG, CEDAF, CEDAF / 22/02/2011 / Contributions from CEDAF and SLG, Addition of Forms in the Appendix
0.4 / Rocio Zargoza / AIJU / 23/02/2011 / Contribution from AIJU, Addition of Forms in the Appendix
0.5 / Reyes Moliner / PREVI / 8/03/2011 / Contribution from PREVI
0.6 / Olga Lymperopoulou
Reyes Moliner / HYGEIA, PREVI / 15/03/2011 / Comments and modification of leaker test and questionnaires
0.7 / Reyes Moliner / PREVI / 16/03/2011 / Contribution from PREVI modified, addiction forms in the Appendix
0.8 / Francesco Barban / FSL / 21/03/2011 / Contribution from FSL
0.9 / Olga Lymperopoulou,
Roberta Annicchiarico, Francesco Barban, Reyes Moliner / HYGEIA, FSL,
FSL,
PREVI / 22/03/2011 / Comments to the draft document
1.0 / Chiara Zaccarelli / AUSL / 25/03/2011 / Fine-Tuning, Executive Summary, Conclusions
1.1 / Chiara Zaccarelli / AUSL / 4/04/2011 / First final version to be Delivered
1.1 / Chiara Zaccarelli / AUSL / 14/04/2011 / Final version for quality check
1.2 / Stelios Pantelopoulos / SLG / 15/04/2011 / Final version for delivery to EC
1.3 / Stelios Pantelopoulos, Chiara Zaccarelli / SLG, AUSL / 27/10/2011 / Revised Structure according to comments received during the SOCIABLE review
1.4 / Paolo Mattarelli, Reyes Moliner / CEDAF, PREVI / 31/10/2011 / Inclusion of inputs from PREVI (on social inclusiveness) and CEDAF (on indicators for business/economic evaluation)
1.5, 1.6 / Stelios Pantelopoulos, Chiara Zaccarelli / SLG, AUSL / 04/11/2011 / Inclusion of new questionnaures for users feedback; revisions to the introduction, executive summary and conclusions
1.7, 1.8 / Stelios Pantelopoulos, Chiara Zaccarelli / SLG, AUSL / 07/11/2011 / Incorporation of partners feedback and inclusion of AIJU input on a scoring methodology for usability evaluation.
1.9 / Stelios Pantelopoulos, Chiara Zaccarelli / SLG, AUSL / 11/11/2011 / Fine-Tuning and Revisions to the Introduction, Inclusion of an additional table of the revisions according to the reviewers recommendations
V2.0 / Stelios Pantelopoulos, Chiara Zaccarelli / SLG, AUSL / 17/11/2011 / Fine Tuning and Preparation of Version for Delivery
Revisions made following the SOCIABLE Reviewers Recommendations
Reviewers Recommendation / Consortium ActionsAppropriate methodologies and indicators for measuring the impact of the project on business, usability, social inclusiveness, end user motivation dimensions, should be identified within the Deliverable 7.1. Moreover, a clear methodology and related suitable indicators should be identified by which to measure the usability benefits gained by elderly users by using a multimodal and mixed reality interactive platform. This should also be part of D7.1, to be resubmitted within 1 month following receipt of this report. / The second release (V2.0) of the deliverable addresses the recommendations as follows:
· Regarding technical/economic aspects, Section 5 has been enhanced with indicators that exploit the elderly feedback on the willingness to pay (economic aspect), as well as on the times required to complete cognitive training activities/processes (technical aspect). In order to derive these indicators, a number of questionnaires have been enhanced with questions relevant to the willingness to pay (see the Appendices of this document).
· Regarding usability aspects, section 4 lists now a scoring methodology for quantifying the usability evaluation results. The scoring methodology instructs the experts in quantifying the usability aspects.
· Regarding social inclusiveness aspects, section 4 reports the use of the LSNS test/questionnaire for the quantification of the social inclusiveness. The LSNS questionnaire has also been used in the scope of WP1.
· Regarding end-user motivation aspects, the user satisfaction questionnaires illustrated in Section 3 and included in the Appendix will be exploited.
The above-mentioned metrics and indicators will be considered in subsequent deliverables D7.3 and D7.4, which apply the methodology (according to relevant reviewers recommendation).
Abstract
The present deliverable provides a detailed plan for the evaluation methodology of the SOCIABLE results through describing the main activities to be applied for results validation, in terms of medical/scientific impact and technical/technological perspective, along with their scheduling and organization.The deliverable illustrates and describes in detail how the pilot operations will be conducted, in line with the SOCIABLE clinical protocol prepared by the consortium’s medical experts, along with specifying the study population and the detailed description of the study design, and establishing the evaluation activities for the project results.
The SOCIABLE evaluation methodology foresees the involvement of a control group which will be compared to the experimental groups of elderly users that will be enlisted and participate in SOCIABLE programs.
The deliverable illustrates the methodologies and metrics evaluating the impact of the SOCIABLE service on the users, from different perspectives.
To this end, the deliverable specifies in particular the instruments and metrics to be used to measure the effectiveness of SOCIABLE program to improving cognitive functioning and social activation in elderly users, along with the impact on quality of life and mood. Also, methodologies to evaluate the wider impact of SOCIABLE system on health professionals involved and how to evaluate the usability of the platform are described.
Following the 2nd technical review of the SOCIABLE project (held in October 2011), the deliverable has been revised in order to include quantitative metrics and indicators associated with several evaluation dimensions including social inclusiveness, business anmd usability aspects. Such indicators are indeed included in the present version of this deliverable.
PROPRIETARY RIGHTS STATEMENT
This document contains information, which is proprietary to the SOCIABLE Consortium. Neither this document nor the information contained herein shall be used, duplicated or communicated by any means to any third party, in whole or in parts, except with prior written consent of the SOCIABLE consortium
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
Table of Contents
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
Revision History 2
Revisions made following the SOCIABLE Reviewers Recommendations 3
Abstract 4
Table of Contents 5
List of Figures 6
List of Tables 6
Executive Summary 7
1. Introduction 9
2. Planning of the Experimentation Activities 13
3. Overview of the SOCIABLE Methodology for Clinical Validation 14
3.1 Study Population 15
3.3 Eligibility Criteria 15
3.3 The Study Design 18
3.4 Statistical analysis 20
3.5 Study Assessment Procedures 20
3.6 Instruments and tests for the experimentation 22
4. Validation and Evaluation of the SOCIABLE System Usability, Impact on Social Inclusiveness and Wider Impact of the System 28
4.1 Overview of Approach – Reception of Feedback from all stakeholders 28
4.2 Validation of Impact to health professionals 29
4.3 4.2 Validation of Impact to elderly users (inlcuding end-user motivation) 30
4.4 4.3 Usability Evaluation 30
4.4 Validation of Impact on Social Inclusiveness 33
4.5 Validation of Wider Impact to the Medical Community 34
5. Technical Evaluation of the SOCIABLE System 35
5.1 Technical Evaluation Aspects 35
5.2 Technical Evaluation Groups and Modalities 36
5.1 Indicators for Technical/Economic Evaluation 37
5.1.1 Execution time of the evaluation process 37
5.1.2 Costs of the evaluation process 39
5.1.3 Potential Customer Satisfaction (through the Questionnaires) 40
6. Conclusions 41
7. References 43
Appendices 45
Appendix 1: Satisfaction Questionnaires 45
Appendix 23: Questionnaires and templates for Usability Evaluation 62
7.1 Appendix 4: Questionnaires for Technical/Technological Evaluation 72
Appendix 5: SOCIABLE SATISFACTION Questionnaires (CAREGIVER/MEDICAL EXPERT VERSIONS) 76
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
List of Figures
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
Figure 1: The SOCIABLE Evaluation Framework 10
Figure 2:Timing of the steps entailed in the medical evaluation process 13
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
List of Tables
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
Table 1: Timing of the Cognitive, Functional and Affective Assessments for the four segmented SOCIABLE groups 19
Table 2: Standardized Tests to be used in SOCIABLE 21
Table 3: Metrics to be used by the Technical Experts in the scope of the usability evaluation 33
Table 4: Metrics to be used by the Medical Experts in the scope of the usability evaluation 33
Table 5: Evaluation Times to be used for the Technical Evaluation 39
76/76
SOCIABLE: Motivating platform for elderly networking, mental reinforcement and social interaction
WP7- Pilot Operations
Deliverable D7.1: “Pilot Operations Plan”
Executive Summary
The main objective of the SOCIABLE project is to introduce and pilot a novel ICT based approach to the cognitive training and social activation of elderly individuals. The approach will be pilot in seven different sites, across four European countries.
Each site will undertake the process of enlisting users for the pilots, while at the same time assessing their cognitive, functional and affective status. The assessment will take place during the elderly registration/enlistment in the SOCIABLE pilot program, as well as at later time instants as part of the SOCIABLE medical-related evaluation. In order to support the above-mentioned processes, the project has established scientifically sound criteria and methodologies for selecting users and assessing their cognitive, functional and affective status. Criteria and methodologies have been defined by the medical experts of the consortium, following a consensus process that took into account the project needs, as well as the peculiarities (e.g., availability of neuropsychological tests) of the various participating counties. The adoption of a common agreed uniform approach to user selection and assessment is expected to boost the SOCIABLE pilots, given that all partners will have to apply a common process that will serve as a reference point for all the pilots.
This deliverable reports on the planning of a common methodology for evaluating and validating the results of the SOCIABLE project, in terms of medical/scientific evaluation and technical evaluation.
The purpose of this deliverable is to describe the main aspects of the results validation plan.
In terms of medical/scientific evaluation, the SOCIABLE formal pilot operations will apply the SOCIABLE clinical protocol in terms of the cognitive, functional and affective assessment of the elderly, as well as their participation in play sessions and leaker tests, in order to evaluate the effect of the SOCIABLE program on the elderly users. Also, social aspects will be investigated through the use of specific questionnaires to be applied before and after the SOCIABLE programme.
The wider impact of the SOCIABLE system to the elderly users, in terms of effect on their quality of life, along with the impact on the health professionals involved and on the Medical/Scientific Community in general have been identified.
Finally, the evaluation methodologies from a technical/technological perspective have been described, in terms of the technical evaluation of the SOCIBLE platform and services and usability.
Following the review and assessment of the document from the SOCIABLE reviewers (during the project’s second technical review in October 2011), the experts have recommended revisions to the document with a view to quantifying several evaluation aspects including social inclusiveness, business, usability and economic aspects. In order to address these recommendations, the consortium has defined metrics and quantitative indicators, which will permit a more quantifiable and objective judgment/assessment of the SOCIABLE services.
Based on the activation and application of different aspects of the SOCIABLE evaluation methodology, the project will produce the rest WP7 deliverables, which will detail and analyze all the evaluation aspects at two different time instants within the pilot operation period (leading to an interim and a final version for all the remaining WP7 deliverables). The following table illustrates how the different evaluation aspects and methodologies of this deliverable will be used to drive the results of the coming WP7 deliverables.
SOCIABLE Deliverables / Scope and Evaluation ResultsD7.2a Interim Assessment of the SOCIABLE Platform and Services / This deliverable will apply the medical/clinical methodology of the project in order to evaluate the SOCIABLE approach. The interim version will consider and analyze the set of elderly users that will have completed their participation in the SOCIABLE pilot operations at the time of the interim evaluation.
D7.2b Final Assessment of the SOCIABLE Platform and Services / This deliverable will apply the medical/clinical methodology of the project in order to evaluate the SOCIABLE approach. The final version will consider and analyze the full set of elderly users that will participate in the SOCIABLE pilot operations.
D7.3a Interim Techno-Economic Evaluation / This deliverable will report the evaluation results associated with the technical/technological evaluation aspects, including evaluation of technical experts and usability evaluation. The interim evaluation will provide an intermediate assessment at the time where the interim evaluation will be carried out.
D7.3b Final Techno-Economic Evaluation / This deliverable will report the evaluation results associated with the technical/technological evaluation aspects, including evaluation of technical experts and usability evaluation. The final evaluation will provide the final assessment of the SOCIABLE services from a techno-economic perspective.
D7.4a Interim Evaluation from Elderly-Users / This deliverable will analyze the satisfaction questionnaires received from elderly users participating in the pilot operations. At the same time it will report feedback from other stakeholders (such as medical experts and caregivers). The interim version will report the analysis of the feedback received up to the point of the interim evalution.
D7.4b Final Evaluation from Elderly-Users / This deliverable will analyze the satisfaction questionnaires received from elderly users participating in the pilot operations. At the same time it will report feedback from other stakeholders (such as medical experts and caregivers). The final version will report the analysis of the feedback from the full set of elderly, medical experts and caregivers participating in the evaluation.
1. Introduction