To:Documentation Group
Cc: Kevin De Pew, Instructor; English 203 class
From:Usability Testing Group
Date:April 30, 2003
Subject:Results of Usability Testing and Suggestions for Document Revision
We have conducted usability testing on the draft of the Quick Reference Guide that we received, and the purpose of this memo is to report on our observations and make suggestions for improving the guide based on what is most beneficial to the guide’s users. We will be telling you a little about our experience with usability testing, and we will also be making suggestions for improvement based upon the participants’ actions and comments. Additionally, we will be addressing possible biases in the testing procedure. At the end of this report, you will find a summary of each participant’s profile and comments. We hope that you will be able to implement some of our suggestions into the final document in order to better suit the needs of the users.
Usability Test Description
We began the usability tests by introducing ourselves to the participants and explaining the purpose of the testing. We assured them that we were not testing them—that we were testing the guide. We informed them that we would be taking notes on their behaviors and asked them to think aloud so that we could understand their thought processes and better judge how to use that data to improve the guide. We also told them that we could not answer questions that related to the program and that they should rely heavily on the guide as needed.
We then asked them a series of demographic questions intended to get answers to questions about their gender, age, area of study, experience and comfort with computers, and experience with Adobe Premiere and other programs. We also wanted to get an understanding of what kinds of help they usually find helpful and what kind of project they intended to create with Premiere.
Their assignment was fairly vague. We asked them simply to create a project that implemented two video files, one audio file, and one transition. We gave them very little other instruction, as we believed that allowing them to create the project of their choice would allow them to maximize their reliance on the guide for reference.
Following each participant’s test, we asked them questions about their overall impressions of the Adobe Premiere Quick Reference Guide, their feelings about the guide’s current format, their experience with the program, and their degree of satisfaction with the project that they created.
Statement of Possible Test Biases
We conducted a total of five usability tests on the Quick Reference Guide for Adobe Premiere, but, unfortunately, only one of those tests was conducted within the environment of the DLC. We agreed to test the first two participants in their home because it was more convenient for them, and we figured that the guide might possibly be taken out of the DLC and used by students who have the program on their home computers. However, the next two tests were scheduled to be held in the DLC. We had reserved a four-hour block on one of the High-end PC’s for Sunday afternoon. The participants met us there, but because the DLC was not staffed, there was no one there to enforce the reservation policy, and the students using our reserved computer were unwilling to give it up. Because the participants were already waiting in the DLC with us, we decided that we would have them do the test in the home where the first two tests were done; we did not want to waste their time and lose them as participants. As a result of having conducted these tests outside of the DLC, we may have inadvertently recorded some bias about the guide’s effectiveness, as it was designed mainly to be used within the controlled environment of the DLC.
Additionally, for these four tests, we were unable to have one mediator and two observers, because our schedules did not work out that way. For each of these tests, one mediator and one observer was present. We tried to record a sufficient amount of detail about the participants’ interactions with the guide and with the program, but because the entire group was unable to be there, the possibility that we may have missed some minor details exists.
Summary of Research Findings
All of the participants in the usability testing found certain portions of the guide to be more comprehensive than other portions. For example, the importing and exporting sections were very easy for them to follow. At least two participants commented that they found the step-by-step nature of the capturing/importing section to be helpful, and they were able to import videos from the sample files in Adobe Premiere and were able to capture and import video clips from digital camcorders. The two participants who were skilled with computers were able to import images from their digital camcorder without relying too heavily on the guide, but when they were asked about the directions for that section after the test, they remarked that the directions were fairly easy to understand. Also, even participants who had little experience with computers and with technology were able to use the instructions to import clips into the bin from a folder that was either part of the program or was previously saved on the computer network. The same was true for the exporting section. One participant, who was not comfortable with or experienced with computer technology said that the exporting section was “fun” because she was able to do it so easily. She liked that there were short directions followed by a screen shot that showed her exactly what to do.
Many participants commented that the transition section was difficult to understand and follow. This section does utilize screen shots, but for some reason, it was not as clear as the other sections. Both participants with and without computer experience found this section to be more confusing. Part of the confusion stemmed from their difficulty using the tools listed in the editing section. At least two participants had difficulty using the hand and arrow tools to move items from the bin to the timeline window; perhaps one must click specifically on one part of the clip to move it, in which case that fact should be made clear in the description of the tool. Once the video clips were inserted into the timeline window, the participants had difficulty arranging them so that they overlapped enough to use a transition. Additionally, they thought that they should be able to find transitions from the timeline window. They attempted to rely on the guide for help, but because there was no screenshot of the transitions window, they continued to look at the timeline window, which is the largest picture on the page, and were unable to insert a transition as a result.
Finally, the participants seemed to ignore the portions of the guide that were text-heavy, like the descriptions of the shortcut menus, which could potentially be very helpful for them. They also seemed to get frustrated because there was no table of contents, and when they needed to look for something specific, they had to flip through the entire guide. While the participants agreed that the guide was easy to use and contained some valid information it was not completely thorough in explaining all the important tasks. It was strongly suggested that a table of contents and an index be added, as well as more definition of tools and tasks.
Suggestions for Improving and Revising
Implement some kind of organization that incorporates a table of contents and/or a glossary; participants did not like having to flip through the guide to find information that was critical for the completion of their projects.
Incorporate more screen shots in the text-heavy sections of the guide, and make sure screen shots used in other sections are not misleading. Screen shots might be helpful in the “Starting a Project” section and in the “Shortcut Menu” section. The transitions section would benefit from a closer screen shot of the transitions window instead of or in addition to the shot that incorporates all of the windows on the screen but does not highlight the transitions window.
Modify some of the language of the guide that is more technical; use layman’s terms when at all possible. Most of the times that terms were a problem were in the text-heavy sections, so adjusting them so that technical terms coincide with pictures might also be helpful. Participants did not say exactly which terms were a problem for them, but complained that the entire guide was “too technical.”
Try to incorporate definitions of complicated terms into the sections where the steps are actually being performed so that users do not have to flip back to the beginning to try to find answers to their questions. One participant explained—“How can I be expected to memorize all of this stuff (definitions at the beginning) in order to use it during the procedures?” The two—descriptions of technical jargon and procedures—should go hand-in-hand.
If time allows, create a section where users can look to find answers to specific questions. Such a section might be set up in an “if-then” manner; i.e. if I am having this problem, then I should look on this page.
Participants’ Demographics and Specific Comments/Suggestions
In the section that follows, you will find details about each participant’s demographics and preferences. In addition, each person’s likes and dislikes are summarized, and any suggestions that he or she has made are listed. Additionally, if it would be helpful for you, please contact us for copies of our test notes.
Test #1
- Age 27
- Male
- Junior in computer technology
- Highly experienced using computer technologies
- No experience using Adobe Premier.
- Uses trial and error to problem solve, but prefers a tabbed and bound quick reference guide for easy access.
Technology Used:
- Adobe Photoshop
- QuickTime Player
- Real Time Player
- Windows Media Player
- VCR
- VHS Camcorder
- Digital Camcorder
- Windows 2000
- Firewire/USB
- Video Capture Cards
Felt the user guide was sufficient and the test was “fairly easy but could take time to get proficient.”
Liked:
- Screen shots
- Descriptions of the paths to follow (i.e. File Capture etc)
- Explanation of the icons and windows
Disliked:
- Hard time finding some of the options discussed
Suggestions:
Use more screen shots. Create an index of the different options to help answer specific questions. Explain an icon when it is talked about in the “instructions.” In the section explaining the sort cut menu, give an explanation of why the user might want to use that feature.
Test #2
- Age 22
- Male
- Senior in computer engineering
- Fairly comfortable using computer technology, including advanced experience with Adobe Premier
- Uses Adobe Premier to edit video and compile home videos into digital video. Problem solves by reading more on the program.
- Prefers a PDF file because of its constant availability and accuracy
Technology Used:
- Adobe Illustrator
- Adobe PageMaker
- Adobe PhotoShop
- Macromedia Flash
- QuickTime Player
- Real Time Player
- Windows Media Player
- VCR
- VHS Camcorder
- Windows 2000
- Firewire/USB
- Video Capture Cards
Found the user guide to be sufficient.
Liked:
- Explanation of the simple features
- Written well
Disliked:
- Lacked explanation of certain tasks (i.e. transitioning files)
- Transitions section was not thorough
Suggestions:
Include a table of contents and an index for specific terms. It was not very thorough in explaining all the tasks at hand, or eliminated explanation of “simple” tasks.
Test #3
- Age 35
- Female
- Graduate student in English literature
- Little knowledge of and comfort with computer technology
- No knowledge of Adobe Premiere and just wanted to make “something.”
- Most likely to ask someone when having problems or just shut down program.
- Prefers tabbed and bound quick reference guides because “they’re just easier”
Technology used:
- Real Time Player
- VCR
- VHS Camcorder
- Windows 2000
Found the user guide to be inadequate at times, not helpful for someone who is working alone.
Liked:
- Screen shots to illustrate importing and exporting sections
- Description of the monitor window
- Editing features of Adobe Premiere
Disliked:
- Lack of instruction in transitioning section
- Lack of directions for saving and using internet movie clips
- Inability to move clips with hand tool at times
- Poor video quality, and didn’t know how to fix it
- Timeline section
Suggestions:
Make all sections as easy to understand as the importing and exporting sections. Include a table of contents. Use more screen shots. Make the guide “more like someone is walking you through the program.”
Test #4
- Age 27
- Male
- Sophomore in holistic therapy
- Little knowledge of and comfort with computer technology
- No knowledge of Adobe Premiere, did not know what kind of project to make
- Closes programs down when there is a problem
- Prefers PDF files from the company because they are provided with the program
Technology used:
- QuickTime Player
- Real Time Player
- VCR
- VHS Camcorder
- Windows 2000
Found the guide to be semi-helpful but confusing.
Liked:
- Importing section
- Screen shots
- Right click menu descriptions
Disliked:
- Lack of table of contents
- Hand tool operation
- Editing tool descriptions
- Transitionssection
Suggestions:
Place descriptions of tools in sections where they will be used so that the reader doesn’t have to memorize them at the beginning. Use a table of contents.
Test #5
- Age 19
- Female
- Freshman in English
- Little knowledge of and comfort with computer technology
- No knowledge of Adobe Premiere, but would like to use it for video editing
- Prefers help from experts
Technology used:
- Adobe PageMaker
- Yahoo! Player
- VCR
- VHS Camcorder
- Digital Camcorder
Did not feel that the guide was sufficient
Liked:
- “The idea of the program”
- Organization of the guide
Disliked:
- Terminology of guide—too foreign and technical
- Lack of description of where to begin with a project
Suggestions:
Create a step-by-step instruction section to get started and lead through the completion of the project. Use more “layman’s” terms, and less technical jargon.
1