Xolax Corporation Website DesignUsability Test Results

October 26, 2018

Usability Test Results for

Xolax Corporation’s

Corporate Web Site

Data in this report generated by

Ruth Villesis ()

Diana Hendricks ()

Martha Porter ()

INTRODUCTION

Following is the test plan for conducting usability tests of the corporate web site for Xolax Corporation. The plan covers the following sections:

  • Purpose
  • Problem statements
  • User Profiles
  • Methodology
  • Task list
  • Test environment and equipment requirements
  • Evaluation measures
  • Test report contents and presentation

PURPOSE

The XOLAX Company is a software engineering firm located in Austin, Texas. The company’s main product is a suite of Java development tools called XOLAXware. As per the corporate vision, the purpose of the web site is to provide technical support to its current customers, to act as a sales channel for existing and potential customers, and to act as a vehicle for self-promotion for the company and its products.

The XOLAX web site has recently undergone a major facelift: both content and look-and-feel have been recreated from scratch to produce a more modern, professional look. Additionally, vast amounts of new informational content (primarily in the way of online documentation and technical support) have been added to enhance the value of the site to our users.

The purpose of this battery of tests is to determine if the changes that have been made have produced an overall improvement in site usability, as compared to our previously established baselines (1998). We hope to implement changes suggested by this study before the GOOGOLPLEX trade show in Las Vegas in July.

TEST OBJECTIVES

After analyzing the purpose of this web site, we have determined that there are several broad categories of questions that need to be answered about its usability.

  1. Can our current users find the technical support information they need on the XOLAX web site?
  2. Can potential customers locate product information and ordering instructions?
  3. Can existing customers locate upgrade information and place an order for an upgrade?
  4. Can press personnel locate news releases and other related information?
  5. Can potential employees find job information and apply for a job?

USER PROFILES

Through research performed by the XOLAX marketing department, we have been able to identify several target audiences for the XOLAX web site. They are classified as two groups, a main target group and a secondary target group. Both of these groups may be further subclassed into more groups.

Main Target Groups

Purchasing authorities for large, multinational hi-tech companies are the main target group for this website. This main target group can be separated into two distinct subgroups.

  • Potential Customers - People that already make decisions regarding equipment purchases. They are most likely to be highly educated with advanced degrees in engineering and/or business. They may currently do business with one of our competitors via the web or traditional purchasing channels.
  • Current Customers– Also have a high degree of education. Our research shows that this group has a high loyalty rate but that it also expects a high degree of quality and service. One of our focuses is on heightening our level of service to them.

Secondary Target Groups

Members of this category are likely to use the XOLAX web site, but are not considered primary users.

  • Investors
  • Press/ Media
  • Future Employees

METHODOLOGY

The usability test will consist of the main performance test designed to gather extensive usability data via direct observation.

The main performance test is composed of the following four sections:

1. Participant greeting and background questionnaire

Each participant will be greeted by the area receptionist and offered a seat and refreshments. The participants will be given a very short questionnaire that gathers basic background information. Participants will be notified that they will be anonymous throughout results reporting, and that they will be assigned a unique ID number, which will be used in place of their name.

2. Orientation

The participant will listen to a short, verbal script that will explain the agenda for the day. All parts of the test will be explained. The participants will be notified of their right to leave the session should they become uncomfortable.

3. Performance test

During this section of the test, the test moderator will ask the participants to perform a series of usability tasks on the XOALR corporate web site. Each task will be introduced by way of a short, scripted instruction from the test moderator.

For each task, the data logger will record the participants’ actions, including time elapsed, number of clicks, and success/failure. The individual tasks are enumerated below.

4. Participant debriefing

After all tasks are complete or the time expires, the test monitor will debrief each participant using a questionnaire. The debriefing will include the following:

  • Participant’s perceptions about usability and aesthetics of the site
  • Participant’s overall comments about his or her performance

The debriefing session serves several functions. It allows the participants to say whatever they like, which is important if tasks are frustrating. It provides important information about each participant’s rationale for performing specific actions, and it allows the collection of subjective preference data about the site.

After the debriefing session, the participants will be thanked for their effort, and invited to review other sites.

TEST ENVIRONMENT AND EQUIPMENT REQUIREMENTS

Our testing lab will be a simple lab setup, including a computer workstation, and three seats: one for the participant, one for the test monitor, and one for the data logger. A diagram of the setup we used is given below.

Data LoggerUserTest Moderator

EVALUATION MEASURES

The following evaluation measures will be collected and calculated:

1. The average time to complete each task, and average number of clicks, across all participants.

2. The percentage of participants who finished each task successfully versus those who had errors from which they could not recover.

3. Error classification: to the degree possible, each error will be classified and a source of error indicated. Error classes are as follows:

Observation and Comments -- The test monitor notes when participants have difficulty, when an unusual behavior, or when a cause of error becomes obvious.

Non-critical Errors -- An individual makes a mistake but is able to recover in the allotted time.

Critical Errors -- An individual participant make a mistake and is unable to recover and complete the task on time. The participant may or may not realize a mistake has been made.

4. Participants ranking of usability and aesthetics of the site. (Some questions may be essay-type, rather than rankings.)

TASK LIST

Task List for XOLAX Web Site Usability Test

TASK LIST LEGEND:

MTC = Maximum time to complete

SCC = Successful completion criteria

TASK NO. / TASK DESCRIPTION / TASK DETAIL
1. / Locate the recent press release about XOLAX’s involvement with a local University. / SCC: Navigate to the URL shown below.

MTC: 1 Minute
2. / You would like to apply for a software engineer position. What are the job requirements for this position? / SCC: Locate requirements on page listed below.

MTC: 1 Minute
3. / What is the availability of the FOO++ software module? / SCC: Availability is: shipping in mid-may. Locate information in one of the two following pages:

or

MTC: 1 Minute
4. / You have a damaged FOO format file. You need to use the recover utility. Find out how to use it. / SCC: Locate technical article #4982734 via Tech Support Search page
MTC: 1 Minute
5. / After you installed Service Release 8 for the XOLAXware compiler, your computer has begun crashing. Find out if this is a known bug. / SCC: Use Tech Support Search page to locate article # 94875334
MTC: 2 Minutes
6. / What is the cost of the QUUX-12 filter plug-in module? / SCC: Price is $350 – find this via one of the following pages:


MTC: 2 Minutes
7. / Is there a competitive upgrade path from SPORKTECH’s SNOO++ package? / SCC: Yes, for $399 USD. Find at following page:

MTC: 2 Minutes
8. / You want to purchase the XOLAXware Professional package. Please use this (fake) credit card to place the order. / SCC: Navigate to the URL below, fill out form, and complete transaction

MTC: 4 Minutes
9. / Are there any known XOLAX Y2K or other related date problems? / SCC: Yes, certain products are not year 2038 compliant. Locate article on page shown below.

MTC: 1 Minute
10. / What are the system requirements for the XOLAX Vaporware suite? / SCC: Windows 98 or NT 4.0 SP6 +, 128 MB RAM, 333 MHz Pentium II or better. Find on page shown below

MTC: 1 Minute
11. / What is the maximum size for a FOO bytecode file? / SCC: 128 GB. See page listed below.

MTC: 1 Minute
12. / Please sign up for a free XOLAX developer account. / SCC: Locate form at URL below, fill out form and submit.

MTC: 3 Minutes

RESULTS

Performance Data Summary

TASKS / % of participants performing Correctly (within benchmark) / % of participants performing Incorrectly (within benchmark) / Mean Time (minutes) / Mean clicks
1. Locate the recent press release about XOLAX’s involvement with a local University. / 0 / 100 / 1.01 / 3.57
2.You would like to apply for a software engineer position. What are the job requirements for this position? / 43 / 57 / .67 / 4.00
3. What is the availability of the FOO++ software module? / 86 / 14 / .30 / .86
4. You have a damaged FOO format file. You need to use the recover utility. Find out how to use it. / 100 / 0 / .16 / 2.43
5.After you installed Service Release 8 for the XOLAXware compiler, your computer has begun crashing. Find out if this is a known bug. / 100 / 0 / .30 / 3.43
6. What is the cost of the QUUX-12 filter plug-in module? / 57 / 43 / 1.45 / 7.43
7. Is there a competitive upgrade path from SPORKTECH’s SNOO++ package? / 43 / 57 / 1.39 / 5.86
8. You want to purchase the XOLAXware Professional package. Please use this (fake) credit card to place the order. / 57 / 43 / 1.56 / 9.29
9.Are there any know XOLAX Y2K or other related date problems? / 29 / 71 / .80 / 3.86
10.What are the system requirements for the XOLAX Vaporware suite? / 86 / 14 / .24 / 2.57
11.What is the maximum size for a FOO bytecode file? / 100 / 0 / .18 / 2
Please sign up for a free XOLAX developer account. / 29 / 71 / 1.68 / *

Preference Data Summary

Questions

/ %SA / %A / %N / %D / %SD
1. Was the language on the Task List you were given, easily understood? / 71 / 29 / 0 / 0 / 0
2. Was the amount of information on the home page adequate? / 0 / 43 / 29 / 29 / 0
3. Was the use of color appropriate? / 0 / 57 / 43 / 0 / 0
4. Was information grouped consistently? / 0 / 29 / 14 / 57 / 0
5. Was the navigation inherently intuitive? / 0 / 14 / 14 / 57 / 14
6. Were colors and navigation consistent through out the site? / 14 / 43 / 43 / 0 / 0
7. Were you able to move around the site without getting lost? / 0 / 29 / 43 / 29 / 0
8. Was there too much information on individual pages? / 14 / 29 / 29 / 29 / 0
9. Was there too little information on individual pages? / 14 / 43 / 14 / 29 / 0
10. Was there adequate cross-referencing of topics and information? / 0 / 29 / 29 / 29 / 14
11. Was more important information highlighted in some way? / 29 / 14 / 14 / 43 / 0
12. Were topic and page headings self-explanatory? / 14 / 43 / 14 / 29 / 0
13. Was it necessary to scroll often to reach desired information? / 86 / 14 / 0 / 0 / 0
14. Was the “Table of Contents” (commonly called a Sitemap) helpful? / 29 / 29 / 14 / 14 / 14
15. Was the terminology understandable throughout the site? / 0 / 71 / 29 / 0 / 0
16. Was there an adequate use of graphics? / 29 / 14 / 14 / 43 / 0
17. Were the graphics clear and sharp? / 29 / 29 / 29 / 14 / 0
18. Were text and graphics presented in a visually aesthetic manner? / 29 / 43 / 29 / 0 / 0
19. Was there an adequate use of white space? / 29 / 43 / 14 / 0 / 14
20. Overall, were the pages quick to load? / 43 / 56 / 0 / 0 / 0

page 1

Appendix A

Orientation Script

Hi, my name is ______. I’ll be working with you today. Let me give you a little background on what we are doing here.

We’re here to test how easy it is to use the ______web site, and we’d like your help. We’re looking for potential design flaws and usability problems within our web site.

I am going to ask you to perform several tasks on the web site, and I’d like you to carry out those tasks to the best of your ability. We are testing the site’s usability, not your ability. Try to work in the same manner that you would at work or at home; don’t worry about paying special attention to details, and don’t feel pressured to succeed, since much of what we are looking for is the natural ease of use of this site.

I’d like to assure you that none of the information that we gather here today will ever be released outside of our research team, and your name will never be used in conjunction with your results. For our study we have assigned you an ID number that will be used on all documentation.

Although we don’t anticipate any of the activities today to cause you any discomfort, you should know that it is your right to stop any task that makes you feel uncomfortable or overly stressed out. You may quit the session at any time.

Since we are trying to determine the site’s usability, during this session I will be unable to answer any questions that would help you accomplish a task; you will need to rely only on the resources in the web site itself for task completion. However, you may ask any questions you’d like now before we begin.

After the session is over, there will be a brief period when you can ask any questions you would like, even about tasks that you were unable to complete. We would then appreciate if you would complete an questionnaire about your experience using the site.

Do you have any questions before we begin?

Appendix B

Post Test Questionnaire

Please answer the following questionsbased on your experience using the web site.

Legend

SA = Strongly Agree; A = Agree; N = Neither; D = Disagree; SD = Strongly Disagree

Task Questions/Issues:

1. Was the language on the Task List you were given, easily understood?

SA A N D SD

General On-Screen Issues:

2. Was the amount of information on the home page adequate?

SA A N D SD

3. Was the use of color appropriate?

SA A N D SD

4. Was information grouped consistently?

SA A N D SD

5. Was the navigation inherently intuitive?

SA A N D SD

6. Were colors and navigation consistent through out the site?

SA A N D SD

7. Were you able to move around the site without getting lost?

SA A N D SD

8. Was there too much information on individual pages?

SA A N D SD

9. Was there too little information on individual pages?

SA A N D SD

10. Was there adequate cross-referencing of topics and information?

SA A N D SD

11. Was more important information highlighted in some way?

SA A N D SD

12. Were topic and page headings self-explanatory?

SA A N D SD

13. Was it necessary to scroll often to reach desired information?

SA A N D SD

14. Was the “Table of Contents” (commonly called a Sitemap) helpful?

SA A N D SD

15. Was the terminology understandable throughout the site?

SA A N D SD

16. Was there an adequate use of graphics?

SA A N D SD

17. Were the graphics clear and sharp?

SA A N D SD

18. Were test and graphics presented in a visually aesthetic manner?

SA A N D SD

19. Was there an adequate use of white space?

SA A N D SD

20. Overall, were the pages quick to load?

SA A N D SD

Overall impression of the site

On a scale of 1 to 10, 1 being the worst, and 10 being the best, how would you rate this site based on what you have seen today? (Circle 1)

1 2 3 4 5 6 7 8 9 10

Please add any comments or suggestions in the space provided that you feel will help us evaluate the usability of the web site. Positive and negative comments will all be appreciated and addressed accordingly.