We Conducted Two Types of Inspection

Hotmail
Usability Evaluation

Team C
November 1998

Evaluations

Inspections

We conducted two types of inspection:

·  exploration of the site by two interface designers to become familiar with the product and note likely problem areas – we incorporated some of these problem areas in our scenarios

·  walkthroughs of a simple scenario using a variety of platforms and environments to identify usability problems related to hardware, software configurations, and user characteristics

User Tests

We conducted a minimal number of user tests to supplement the results of our inspections. In the user tests, users fitting the given user profile were asked to carry out scenarios while an observer took notes. Users were interviewed after the testing session.

We attempted to observe three current users of Hotmail and three others carry out scenarios. Inability to connect to the Hotmail site prevented one non-Hotmail user from beginning the test. The five completed sessions provided sufficient information that we felt there was no need to reschedule the sixth.

All participants were heavy users of e-mail. All but one browsed the Internet almost daily, and that one browsed the Internet weekly.

Conclusions

Positive results

All test participants were able to carry out the essential tasks of creating a login name, sending a note, and reading a note.

Test participants came away with generally positive feelings about Hotmail. Participants who were new to Hotmail and who were able to complete the evaluation session would recommend Hotmail to others needing free and easily accessible e-mail. They also reported that they would use Hotmail if they did not already have satisfactory e-mail service. Three participants were already Hotmail users, so they presumably find the service valuable. They do recommend Hotmail to others.

The site appears usable through Lynx, a non-graphical browser. It took one tester about 150% longer to register and send a message using Lynx compared to the same task with a more typical configuration, but this seems reasonable.

Users like the “Forgot Your Password” feature although, as noted under “Problems,” they did not all use it as intended.

Problems

Users click BACK button or left-side navigation bar to leave a form instead of the OK button.
Severity: High – users who do this cancel settings they think they made and may not notice.
Frequency: Low
Possible solution: Make OK and Submit buttons more prominent.

Controls nested under radio buttons do not turn on the corresponding radio button when activated. For example, turning on the “Full Color” radio button does not turn on its parent, the “Customize” radio button. Similarly, choosing a day of the week does not set a weekly reminder. (See examples below.) It is easy for users to believe they have changed a setting when in fact they have not.

Severity: Initially high – prevents operation from being successfully completed. Users who persist figure out the behavior.
Frequency: High – no user successfully set a nested control on the first attempt. Probably not a recurring problem.
Possible solution: Allow interaction with a nested control to turn on the parent radio button.

Secondary windows can get lost behind the main browser window and then can only be displayed from the Windows task bar. The only way out of secondary windows is to close them using the title bar Close box. Clicking the main browser window either accidentally or intentionally hides the secondary window, and subsequent requests for the secondary window do not bring the secondary window to the fore. In this state, the link that launches the secondary window appears to no longer be functional.
Severity: High
Frequency: Low
Possible solution: Provide explicit Close buttons within the secondary windows to increase the odds that users close rather than hide these windows. Bring secondary windows to the fore when user clicks a link that launches the window.

The Hotmail site is unusable with graphics turned off. Some users turn off graphics for faster loading, and visually impaired users may rely on ALT tags to use Web pages.

Severity: High for a small number of people
Frequency: High for a small number of people
Possible solutions: Include ALT tags for all graphics. Include a textual “no graphics” link on the login page that goes to the same page that Lynx displays.

Intent of password hint is not clear. Some users think of this feature as a means of reminding themselves of their password, so they choose questions for which their passwords are the answer. Sample hint questions included “the ---- of Oz” and “my nickname,” where the passwords were “wizard” and the nickname. This is a security problem.

Severity: High because the problem prevents the feature from being used as intended. Low because the problem does not prevent the user from using Hotmail and will probably cause security problems for only a few people.
Frequency: High
Possible solution: Do not call the question a password hint, because it is not a hint. Alternative: “Enter a question that only you can answer. We will ask you this question if you forget your password.”

The default button appearance makes it difficult to know where to click. The column headings in the In-Box and other folders do not look like clickable areas. Even when told that clicking them will sort the folder, users cannot tell they need to click exactly on the label. The main navigation buttons in the left-hand frame also have unresponsive areas between the text and the colored squares.

Severity: Low – users figure it out eventually
Frequency: Medium, until users become familiar with interface
Possible solution: Choose a default appearance with better affordances.

Users press Enter to move from field to field in a form. This causes errors, because the incomplete form is submitted.
Severity: Low – users figure it out eventually
Frequency: Medium, but not recurring
Possible solutions: Allow Enter to move to next field or control requiring input. Give instructions to use Tab or mouse clicks to move through a form.

Users do not notice that their passwords are cleared when they return to a page following an input error. Users correct the original problem and resubmit the form, which is now incomplete because of the cleared password. This results in another error.
Severity: Low – easy to recover
Frequency: High – every user who had to correct an error in a form with a password had this problem
Possible solution: Upon return from such an error, pass users through a page that tells them they need to re-enter their password.

Users have a hard time finding Hotmail from the MSN.com home page. Two of five users got to Hotmail by typing www.hotmail.com in the address field. Another user performed a search. One user who did follow the E-mail link on MSN.com remarked that it was hard to find anything because the page was “cluttered.”

Severity: Low – users get there eventually
Frequency: Medium
Possible solutions: Unclutter the MSN.com home page. Make the Hotmail link more prominent.

The Choose File dialog for selecting a file to attach is confusing. One user did not want to Open the file and so did not want to click the Open button. This user attempted to drag files from the dialog onto the Compose/Attachments page. Eventually succeeded by pressing Enter, which unintentionally triggered the Open button.

Severity: Low
Frequency: Low
Possible solutions: Change “Open” label to “Choose” or “Select.”

Sometimes Hotmail is too busy. This problem caused the cancellation of one user observation and occurred roughly 10% of the time during inspections.
Severity: Low – users can try again later
Frequency: Low

Users encounter server errors. Two of 5 user tests encountered server errors. However, we rarely saw this during inspections. One user commented, “Now I would look for another service.”
Severity: Medium – users can retry and usually get through
Frequency: Low

Hotmail occasionally has slow response times. One of five test participants commented about the slow response times. We also observed this occasionally during inspections.
Severity: Low
Frequency: Low

General observations

Some users do not see clear boundaries between Hotmail, the browser, and the operating system. Users looked for Hotmail help in the browser help, looked on the desktop for files to attach, and used the browser’s BACK button to navigate through Hotmail when that was not appropriate. This confusion may be due to unfamiliarity with Web-based applications. It may also be due to the operating system’s moves to integrate the desktop and the Internet.

It is not clear to users how they should navigate through the site. For example, we observed users pressing BACK, In-Box, or Close to leave the note, and all of these methods work fine. On a form, however, pressing BACK or In-Box cancels the user’s settings. One user clicked the Move to Folder button to go to a folder and not to move messages to the folder. The inconsistent link and button styles described below may contribute to this uncertain navigation.

Link and button styles are inconsistent within and between screens. On the screen below, hypertext links are used for commands like Create, Edit, and Delete. The column headings are inactive, but they look just like the bottom four links in the navigation frame on the left. The top six links on the left have a different appearance from the rest.

On the screen below, push buttons rather than links are used for commands like Move To and Delete. The column headings perform an action, but you would not expect it from their appearance. None of our participants were able to sort the In-Box without intervention. Additionally, even when told to click the column headings, users did not know to click on the text in the heading.

2