Developers' Checklist for ethics compliant products and services
This checklist may serve as a guide within and beyond AEGIS project. It may be useful for all user organisations conducting research and testing with users with disabilities (and elderly persons).
This checklist refers to developers/integrators that intend to develop solutions and ensure that incorporate in an ethics compliant way the targeted user groups into their specific project. In order to make this easier for the reader and give a quick reference for the following checklist, the AEGIS UCD Plan summary figure is provided below. UCD plan, followed also in AEGIS, is being proposed as a guide to any developer wishing to conduct ethics compliant development.
AEGIS UCD Plan
Issue No / Issue Description / Rationale / Checklist points / Comment /1 / Select your Persona (s) [existing or new one(s)]. / According to AEGIS user modeling phase (see UCD plan phases at D1.1.1: “User groups' and stakeholders' definition and UCD Implementation plan”; downloadable from the web site; http://www.aegis-project.eu) one of the basic items to start your work with in order to ensure that you address sufficiently users is Personas.
You may use existing Personas (from AEGIS or ACCESSIBLE; http://www.aegis-project.eu) or build new ones.
In case that there is a need for new Personas, you should follow the UCD Phase 1 (Gathering User Needs) proposed approaches (contextual inquiry, interviews, questionnaires, task analysis) in order to build new ones that will comply with real and up to date user needs.
In case the scope of the targeted application is not possible to be covered by Personas, and further individualisation and personalisation needs to take place, you should be aware that this is beyond the scope of AEGIS and should refer to on-going cloud technologies based initiatives that deal with that. / AEGIS Personas used: Names/ID's:
ACCESSIBLE Personas used: Names/ID's:
Other Personas used from other reference; please provide info:
New Personas built; Please define how many and provide short description:
Individual User Profiles used; provide number, short description and source:
No Personas or user profiles used; explain in short the reason: / Please comment if you have selected existing Personas from AEGIS or ACCESSIBLE projects or you have built new one(s) and if yes, if you have followed the AEGIS UCD Plan as a guide.
If you have selected to move forward with individual user profiles, please describe in short the research process and reference.
If you have not used any Personas or user profiles, please provide reasoning.
2 / Build your conceptual models, Use Cases and User Scenarios in order to map user needs with design requirements. / It is important to outline the basic requirements and conceptual design of your solution in advance and in a way that will ensure that will encompass the mapping to your previously identified user needs (interpreted as explained in Issue 1 in Personas or user profiles).
Thus, following the approaches proposed in AEGIS UCD Plan Phase 2: “Specifying user requirements” (use cases, user scenarios, conceptual design through expert evaluations and focus groups), you need to define your basic use cases, scenarios and conceptual models and map them with the Personas of the previous Issue. The format and guide for content of all the above can be found at AEGIS D1.1.3: “Use cases and application scenarios” (downloadable from http://www.aegis-project.eu). / Built Use Cases; please provide number and short description and how they emerged:
Built Conceptual models; please provide number and short description and how they emerged:
Built Use Scenarios; please provide number and short description and how they emerged:
Other format/way used to map user needs to design requirements; please provide reference and short description:
Verification of use cases or other with users and experts; please provide short info (where, how, with how many and which type of users and experts):
No Use Cases, conceptual models or other format has been used to map user needs to design requirements; explain why: / Please comment on the process you have actually followed for the specification of your work requirements and for which reasons you have selected the approaches and formats you did.
Please also focus on if you have verified these requirements (in whatever format you have followed) with users and experts.
3 / Following your requirements mapped with Personas (see previous issues), please design and schedule your evaluation with users and experts in an iterative matter and upon concise experimental plans, defining specific research hypotheses, targets, evaluation approaches and measuring tools and success thresholds for each phase. In each iteration, the level of maturity of the solution towards evaluation should be explicitly defined together with the scope of the evaluation iteration. / This is a necessary follow-up in order to ensure ethics compliant development and ensure that users (all types, addressed directly and indirectly) will be involved from the first phase of design until the final release of the solution.
Again, in the AEGIS UCD plan, the Phase 3: “From Concept to Working Prototype” and Phase 4: “ Field trials” proposes a series of approaches to proceed from quick & dirty testing to HiFi Prototype redesign (see D1.1.1, downlodable from the web site, or above figure for a quick overview).
Please make sure that the users that you will involve will satisfy your Personas (or user profiles) and that the evaluation tasks/scenarios will be in compliance with your previously built conceptual models, use cases and scenarios.
Please also make sure that besides experts, you need to involve in the evaluation indirect stakeholders (i.e. tutors, carers, etc.) as well as experts in the field through focus groups and interviews.
Performance testing, getting more complex as the maturing of the solution is being evolved from one phase to another, is necessary, whereas it is also necessary to define in advance both subjective (interviews, ...) and objective (logging, service diaries...) approaches and measures.
Especially for the last phases of evaluation, please make sure that you conduct a strong experiment with valid results [AEGIS recommendation; Power of the study: 0,8 and Statistical significance of the results (α level): 0,05].
Specifically, for the recruitment of the users to be engaged, you need to apply in advance specific criteria (see again D1.5.1 as a guide; downlodable from the web site) in order to be sure that you take into account gender and other issues. AEGIS also recommends that you keep a pool of same users across all phases in order to have a reference and keep users in the loop.
For all the above, you may use as a guide AEGIS D1.5.1: “Novel framework and supporting material for the inclusive evaluation of ICT” (downloadable from http://www.aegis-project.eu) / Conduct of evaluation; please define type:
Iteration evaluation framework and experimental plans built; please provide it in short, responding also the following:
Iterative nature; please define number and type of iterations:
Representation of targeted users (satisfaction of Personas/user profiles); please provide type and number;
Representation of other experts and stakeholders; please provide type and number:
Plan for subjects' recruitment; please outline criteria:
Anticipated to keep a common subjects' pool across all iterations
Interaction scenarios used; please provide number and short description:
Interaction scenarios responding to initial requirements (use cases, …)
Specific design plan built (with research hypotheses, indicators, …); please define the level of detail:
Performance testing incorporated; please provide details:
Other approached used; please provide details:
Subjective measures and tools incorporated; please define:
Objective measures and tools incorporated; please define:
Statistical strenght of your experiment; please provide power and statistical significance of the results: / Please specify how many iterations you have foreseen and provide a short and concise experimental plan, which will make obvious the validity and strength of your evaluation and the approaches that you will encompass providing a reasoning.
4 / In parallel to user testing, make sure you leave room for consequent iterative technical validation (lab or other testing) and optimisation from one phase to another. In order to do so in an efficient manner, please follow a concise way of feedback to the development team. / As defined in UCD Plan, but also in D1.5.1, intermediate technical validation and optimisation (fed by both technical validation and user and experts evaluation) is recommended.
It is also important to make sure that you interpret each evaluation round results in specific guidelines/checklist in order to keep track of the optimisation level achieved in each case and also that the development team will have a clear idea of the users' needs and wants. As a guide, you may follow the AEGIS checklists for developers (reported in D1.5.2: “Consolidated evaluation report”; see in AEGIS web site http://www.aegis-project.eu). / Conducted intermediate technical validation
Scheduled a specific experimental plan for technical validation; please provide details:
Conducted intermediate optimisation:
Defined a specific way of capturing user needs and wants from testing and development team response; please provide details (i.e. if you have used AEGIS format or other): / As above.
5 / Make sure that all evaluation being conducted is ethics compliant and schedule ethics controlling mechanism following your evaluation iterations. / Complementary to D1.5.1, D5.6.1 and D5.6.3 (AEGIS ethics manual, downloadable from the AEGIS web site) should serve as a guide for the recruitment, engagement and interaction of users during evaluation. Issues like personal data protection and appropriate consent forms are especially important for an ethics compliant testing (and consequently research and development). Please go through the checklist provided in the previous chapter for checking against each item of the ethics policy (please note that depending the local/regional/national policies of the sites your testing will take place), the ethics policy provided herein may need adaptation.
As an outcome of this process, you need to define ethics controlling reports (see as an example D5.6.2: “AEGIS Ethics controlling report”, which is downloadable from AEGIS web site). / Defined ethics controlling mechanism:
AEGIS one:
Other; please define:
Taken into account local/regional/national ethics laws and regulations
Yes
No; please explain why:
Consent forms use:
Yes
No; please explain why:
No ethics controlling mechanism and measures taken; please provide justification / Please explicitly define if you have used the AEGIS ethics policy or established a new one and provide as a result the completed checklist of previous chapter for each iteration as a complement of your ethical controlling report, making sure that you mention problems and revisions you were forced in.
6 / Provide sufficient training before testing. / It is important that you provide complete training courses for your solution(s) to the end users you are going to involve in your testing in advance.
AEGIS recommends a blended approach , consisting of face to face training and on-line training before evaluation. Please visit D5.2.1: “Training platform and users guide” and D5.2.3: “Training course for pilots participants” (http://www.aegis-project.eu) for a guide of your training and also the AEGIS on-line platform at http://aegis.bluepoint-it.ro/login.php.
Please note that the training phase should be prior to the consent forms, in order to leave room to users decide if they are interested or not in evaluation your solution. / Provided training courses:
Following AEGIS approach (blended approach)
Only on-line training
Only physical training
Following other approach; please define:
Used AEGIS training courses; please provide names:
Used other training courses; please provide reference:
No training courses provided; please explain why: / Please describe in short the training approach you have followed and provide in short the feedback.
7 / In order to make sure that the specific application/service is applicable for a specific user group, and prior to any demonstration, please assess in an objective manner the accessibility of your solution.
Please note that this can be also an intermediate step across your iterations. / Depending the nature of your solution, you should select a tool (standard and valid) to assess your solution.
You can also make use of the ACCESSIBLE project assessment tool (DIAS), which simulates the experience of a user with desktop (and mobile) applications. DIAS is addressing users with visual, upper limb and language impairments. A developer may directly benefit from that, whereas, will be also assisted by the recommendations provided by the tool (this is valid for Java Swing applications through Netbeans). See DIAS in ACCESSIBLE web site (http://www.accessible-eu.org/). / Checked accessibility of solution(s); please define at which stage:
Used ACCESSIBLE DIAS for this purpose
Used other tools for this purpose; please provide reference:
No type of accessibility check performed; explain why: / Please mention in specific which tools you have used for the accessibility assessment of your solution and what has been the outcome.
8 / Demonstrate in public your final solutions. / Open and wide public demonstration (physical or electronic) is an added value. If your solution is Open Source, you can directly share with the Open Source Community and incorporate a short feedback form for all types of audience (developers, users, etc.).
Next to that, you may also provide your solution for a considerable amount of time to users to play with freely before making the final adjustments. / Open Source solution:
Yes
No
(If Open Source) shared with Community; please define latest url and when you started sharing:
(If Open Source) and did not share with Community; explain the reasons:
(If shared with Community) got feedback
(If shared with Community) did not get feedback
Other type of public demonstration/sharing (i.e. workshop, …) close to final release; please define:
Provided solution(s) for some time to users to play freely with them
No demonstration and sharing; please explain why: / Please provide in short the overall impression and feedback.