BUILDING A PRIVACY GARDIAN FOR THE ELECTRONIC AGE
Project title / PISA
Deliverable type / Public Usage
Deliverable number / D1.124
Release number / 01
Contractual date of delivery / June 30, 2003
Actual date of delivery / June 30, 2003
Title of deliverable / D24-1: Testing of User Interfaces for the Privacy Agent
Work package / 5.4
Nature of the deliverable / Report
Author(s) / Andrew S. Patrick ......
Cassandra Holmes ...... / Organisation
NRC
Carleton University
Abstract
This is the second report on the human-computer interface (HCI) aspects of the PISA project and it contains the results of a preliminary usability test of portions of the interface design prototype. The purpose of the testing was to determine if the design concepts in the prototype were successful in constructing an agent technology that (1) users can use, (2) users can understand, and (3) users can trust with secure and private information. This testing involved laboratory sessions where subjects interacted with the software system on a computer and answered questions about the features and performance that they experienced. The results indicated that the prototype worked fairly well and was reasonably easy to navigate, but it had poor visual appeal. Users generally understood the concept of a personal assistant who could provide services, and most of them understood the major functions (create, modify, track, get results). The most notable problem was that users had trouble understanding what information was being protected when they completed the privacy preferences portion of the data entry screens. The users also had to rely heavily on rollover help information to understand the privacy preference terms, such as “retention period” and “require tracking”, and they were frustrated by having to set their preferences three times.
Keyword list / user interface, human factors, human-computer interaction, trust
Page 1 / 37
D24-1: Testing of User Interfaces for the Privacy Agent
Executive Summary
This is the second report on the human-computer interface (HCI) aspects of the PISA project. This report contains the results of a preliminary usability test of portions of the interface design prototype. The purpose of the testing was to determine if the design concepts in the prototype were successful in constructing an agent technology that (1) users can use, (2) users can understand, and (3) users can trust with secure and private information.
Fifty people were recruited for the study from the Carleton University community through posters, the Introductory Psychology notice board, and also by approaching individuals in common areas. The usability testing involved sessions where subjects interacted with the software system on a computer and answered questions about the features and performance that they experienced. After completing the usability test, participants were given a 16-item questionnaire to assess the overall usability of the prototype. In addition, this questionnaire enquired about their attitudes towards the trustability of the Internet in general, Internet services, and the PISA prototype that they had just tested.
It should be noted that the participants were not representative of the general population, or of the likely user group for a real PISA service. Instead, the participants were young, Canadian college students, who may have very different attitudes towards WWW site design, Internet services, job searching, or privacy preferences than older adults or European citizens. This important limitation should be kept in mind when interpreting the results.
The usability testing of the prototype PISA interface revealed a number of strengths and weaknesses. Notable findings were:
Result 1.users were able to complete or understand an average of 17.5 out of a possible 27 (65%) major concepts or functions in the interface
Result 2.the average ease-of-use rating was 4.7 on a 7-point scale, corresponding to a rating of "slightly easy"
Result 3.the average ease-of-navigation ratings was 3.64 on a 5-point scale, corresponding to a slightly positive rating
Result 4.the average rating of proper operation of the prototype was 3.62 on a 5-point scale, corresponding to a slightly positive rating
Result 5.30% of the users attempted to login without registering first
Result 6.38% of the users did not like the graphics, describing them as copied from another site. Others thought the graphics looked "professional" and "cute"
Result 7.42% of the users did not like the colours in the interface, describing them as "bland" and "boring". Others described them as "soothing" or "relaxing"
Result 8.50% of the users thought that the visual appearance was unlike other sites that they think highly of
Result 9.88% of the users liked the fonts in the interface because they were familiar
Result 10.92% of the users understood the concept of a personal assistant created for them
Result 11.some users questioned the need to login as a second step after a successful registration
Result 12.users were generally able to understand the major functions presented in the prototype, such as "create task", "modify task", etc.
Result 13.users were sometimes confused by the term "task" when used to describe a set of instructions given to the assistant
Result 14.50% of the users could not understand why they needed to name a task being created
Result 15.users frequently did not understand the terms "job sector" or "company type"
Result 16.users sometimes did not know what units to use for input fields like "desired salary" or "desired location"
Result 17.users sometimes did not understand why they were asked to provide their "current employer"
Result 18.64% of users failed to associated the privacy preferences with the job search parameters
Result 19.users often did not feel that job search parameters required privacy protections
Result 20.users often had difficulty understanding the terms used in the privacy preference interface. Rollover help did provide assistance, but it should not have been necessary
Result 21.many of the users failed to notice or use the preset buttons available for setting privacy preferences
Result 22.the function of the check boxes in the privacy preferences was often unclear, such as whether checking "other allowed purposes" had the effect of allowing the purpose or not
Result 23.users were often unclear about how to complete the "retention period" field
Result 24.understanding of the privacy preference screens increased when contact information and resume information were entered
Result 25.a Just-In-Time Click-Through Agreement (JITCTA) to seek explicit confirmation when processing "union membership" information failed to appear reliably, but when it did reaction was mixed. Some users appreciated the warning about the sensitive information, while others ignored the message completely.
Result 26.a JITCTA to seek final confirmation before the task agent is launched also had mixed reactions, with some users finding the pop-up box redundant and annoying
Result 27.results from the trust questionnaire revealed that, whereas only 54% of participants were willing to send personal information on the Internet at large, 84% would provide their resume to the prototype, 80% would provide their desired salary, and 70% would provide name, address, and phone number
Result 28.whereas only 34% thought that Internet services at large acted in their best interest, 64% felt that the prototype service would act in their best interest.
Result 29.66% of the users agreed that they would "feel comfortable depending on the privacy provided by [the prototype]"
Result 30.only 48% of the people characterized the prototype service as "trustworthy", with 44% of the users being "undecided".
These findings have led to a number of recommendations for further development of the interface:
Recommendation 1.improve on the terms used throughout the interface. Small usability tests can be conducted on the terms to ensure that potential users share the meaning that was intended by the developers.
Recommendation 2.consider using more polished and professional looking graphical images
Recommendation 3.consider a brighter, more attractive color scheme
Recommendation 4.keep to standard fonts, such as Times Roman
Recommendation 5.improve the registration process so that it is clear that users must register before they can use the service
Recommendation 6.integrate registration with login so that users are entered into the system automatically after they register
Recommendation 7.continue to highlight when information is transferred securely
Recommendation 8.the mental model of a personal assistant being created for the user is working, and users understand that the assistant can be given tasks
Recommendation 9.look for a word other than "task" to label the process of creating a job for a personal assistant to do
Recommendation 10.move the step of naming a task to the end of the creation process, and make it clear why users must name their task (so they can differentiate them later)
Recommendation 11.remove or replace the terms "job sector" and "company types"
Recommendation 12.use selection menus instead of text boxes whenever a list of items is appropriate
Recommendation 13.make units of measurement clear, such as for "desired salary" or "desire location"
Recommendation 14.make the reason of entering the current employer clear at the time that it is entered, and make sharing the resume with the current employer a separate and explicit step
Recommendation 15.integrate the privacy preferences screen into a single screen that is completed once, after all the personal information is collected
Recommendation 16.collect the contact information first, since the need for privacy protection is clearest with this information
Recommendation 17.continue to use rollover help, but make it more prominent and easier to discover
Recommendation 18.change preference parameters so they are all yes/no choices, not check boxes whose function is unclear
Recommendation 19.make the privacy presets more prominent and a clear alternative to customized choices for each parameter
Recommendation 20.make it clear that using a privacy preset will erase any custom parameter settings
Recommendation 21.fix the implementation of the presets in the interface code
Recommendation 22.fix "retention period" so it is either a duration for keeping the information or a date
Recommendation 23.change the JITCTAs to be integrated into the interface screens instead of being distant, separate interface windows
Recommendation 24.fix the implementation of the JITCTAs for the union membership field
Recommendation 25.improve the wording for the sensitive information JITCTA
Recommendation 26.fix the "start over" feature so that previously entered information is saved and can be edited
Overall, the results indicate that users can use the major features of the interface, such as creating a job-searching agent. However, some of the specific features, such as controlling specific privacy preference parameters, are in need of more attention. Concerning understanding, the results clearly indicate that users have difficulty understanding the privacy preference terms used in the interface, and this is the most important characteristic to improve. Finally, users did find the service to be trustable, although it is clear that, with the problems in understanding the interface, the maximum possible trust was not created.
PISA, Project Information
Contribution
PISA contributes to key action lines of the IST-programme of the EC:
II4.1: “ To develop and validate novel, scalable and interoperable technologies, mechanisms and architectures for trust and security in distributed organisations, services and underlying infrastructures”.
II4.2: To scale-up, integrate, validate and demonstrate trust and confidence technologies and architectures in the context of advanced large-scale scenarios for business and everyday life. This work will largely be carried out through trials, integrated test-beds and combined RTD and demonstrations.
Goal
The objectives of the PISA-project are:
Demonstration of PET as a secure technical solution to protect the privacy of the citizen when he/she is using Intelligent Agents (called shopbots, buybots, pricebots or just "bots", a short for robot[1]) in E-commerce or M-commerce applications, according to EC-Directives on Privacy.
Interaction with industry and government to launch new privacy protected services. The PISA-project will produce a handbook on Privacy and PET for ISAT and a PISA-agent as shareware. Also a plan for the dissemination of the results of PISA will be produced.
Propose a standard for Privacy Protected Agent Transactions to Standardisation Bodies.
Results
PISA contributes at building a model of a software agent within a network environment, to demonstrate that it is possible to perform complicated actions on behalf of a person, without the personal data of that person being compromised. In the design of the agent an effective selection of the presented Privacy Enhancing Technologies (PET) will be implemented. We label this product as a Privacy Incorporated Software Agent (PISA).
The PISA demonstration model is planned to be a novel piece of software that incorporates several advanced technologies in one product:
- Agent technology, for intelligent search and matching;
- Data mining or comparable techniques to construct profiles and make predictions;
- Cryptography for the protection of personal data, as well as the confidentiality of transactions.
Additionally the project involves:
- Legal expertise to implement the European privacy legislation and the needed development of new rules and norms;
- System design knowledge to turn legal boundary condition into technical specifications;
- Advanced software-programming skills to implement the privacy boundary conditions.
In order to prove the capability of the PISA-model, we propose to test it in a model environment in two cases in e-commerce that closely resembles a real-life situation.
PISA Project Consortium
- TNO-FEL Physics and Electronics Laboratory
Oude Waalsdorperweg 63
P.O. Box 96864, 2509 JG The Hague, The Netherlands
Project co-ordination, Privacy Enhanced Technologies
TNO-TPD Institute of Applied Physics
Stieltjesweg 1
P.O.Box 155, 2600 AD Delft, The Netherlands
Intelligent Software Agents Platform and PISA-demonstrator
- Netherlands Data Protection Authority
Prins Clauslaan 20
Postbus 93374, 2509 AJ The Hague, The Netherlands
Privacy Protection and Legal Issues
- Delft University of Technology, Faculty of Informaton Technology and Systems, Information Theory Group
Mekelweg 4
P.O. Box 5031, 2600 GA Delft, The Netherlands
Cryptography
- Sentient Machine Research
Baarsjesweg 224
1058 AA Amsterdam, The Netherlands
Data Mining, Data Matching and Cases
- FINSA Consulting, Italsoft.
52, Corso del Rinascimento, 00186 Rome, Italy
Intelligent Software Agents and Multimedia Development
- National Research Council Canada
Institute for Information Technology
Montreal Road, Building M-50
Ottawa, Ontario Canada K1A 0R6
Network, Scalability and User Interfaces
- GlobalSign
Haachsesteenweg 1426 Chaussee de Haecht
1130 Brusses, Belgium
Public Key Infrastructure
Table of Contents
1.Introduction
1.1The PISA Interface Prototype
1.2Remote Usability Evaluation
2.Preparing the Prototype for Testing
2.1Changes to Appearance
2.2Changes to Behaviour
2.3Limitations of Testing
3.Experimental Method
3.1Participants
3.2Materials
3.3Procedure
4.Results
4.1Usability Test Method Results
4.2Ability to Understand and Use Interface
4.3Ease of Use Ratings
4.4Usability Opinions and Comments
4.5Trustability Questionnaire Results
4.6Summary of Results
5.Recommendations
6.Conclusions
7.References
List of Figures
Figure 1.1: Configuration of the remote usability tests.
Figure 2.1: The two-column layout of the testing interface.
Figure 4.1: Frequency of ease of use ratings
Figure 4.2: Greeting screen of the PISA prototype.
Figure 4.3: The registration interface screen.
Figure 4.4: Interface screen to acknowledge registration.
Figure 4.5: The main interface screen of the prototype, showing the major functions.
Figure 4.6: The interface screen used to create a new task.
Figure 4.7: The interface for setting job search parameters and privacy preferences.
Figure 4.8: Example of rollover help.
Figure 4.9: The interface screen for entering contact information.
Figure 4.10: The interface for collecting resume information.
Figure 4.11: A Just-In-Time Click-Through Agreement (JITCTA).
Figure 4.12: An interface screen to confirm the information that was entered.
Figure 4.13: The interface screen used to launch the search task.
Figure 4.14: The final interface screen, confirming launch of the task (agent).
List of Tables
Table 1.1: Notable HCI Design Feature in the Interface Prototype
Table 4.1: Percentage of Users Who Understand each Function.
Table 4.2: Summary of Notable Results
Table 5.1: Recommendations for Improving the PISA Interface
1.Introduction
This is the second report on the human-computer interface (HCI) aspects of the PISA project. In the first report, foundational research was presented on design features that can assist in building trustable interfaces. In addition, the European Privacy Directive 95/46/EC (European Commission, 1995) was analyzed and design criteria for building systems that feature "usable compliance" with the specific clauses and intent of the Directive were outlined. Finally, a PISA Interface Prototype was developed to illustrate these design features in a concrete and testable fashion. This second report contains the results of a preliminary usability test of portions of that design prototype. The purpose of the testing was to determine if the design concepts in the prototype were successful in constructing an agent technology that (1) users can use, (2) users can understand, and (3) users can trust with secure and private information. This testing involved laboratory sessions where volunteers interacted with the software system on a computer and answered questions about the features and performance that they experienced.
1.1The PISA Interface Prototype
A stand-alone interface demonstration was developed at NRC in collaboration with the PISA partners. The goal was to develop a set of WWW pages and back-end applications that would demonstrate a "look and feel" for the PISA demonstrator before the actual agent platform was available. This prototype was designed to be both stand-alone, so it could be demonstrated and tested, and also modular and well-structured, so it could be integrated into the final PISA Demonstrator. These design goals were met and the interface prototype can be demonstrated over the Internet. (Interested people should contact Andrew Patrick [ for the WWW address.)
The interface prototype was developed using manually-edited HTML code to facilitate fine-level control of the appearance and behaviour. In addition, using native HTML code, rather than a WWW authoring tool, allowed for detailed documentation of the interface components and design concepts. JavaScript was also used extensively to support dynamic and customizable WWW pages that reflected the individual usage of the system (e.g., whether a user had created an agent or not). Cascading Style Sheets (CSS) were also used to ensure that the entire prototype had a consistent look and behaviour, which is an important design consideration when building trust.