Hotmail Usability Test Final ReportApril 21, 1999

Addendum

Deviations from customary test procedures

In conducting the Hotmail Usability test, Practical Products, Inc. followed our lab's standard testing procedures for formal usability testing. We did not deviate from our usual approach and completed the test within our normal time allotment for similar tests. According to our metrics, a typical test using discount usability testing normally requires approximately 200 person hours.

Resources used for the test

Because of competing demands for our staff's time, we assigned six part-time evaluators to the test. This staffing ensured that we could perform the tests promptly and produce the quality results we and our clients expect. Each of the assigned personnel was selected for his or her special expertise in the various disciplines tapped for the test. Accordingly, our team consisted of a cross-disciplinary group composed of software application experts, professional interviewers, videotape editing professionals, and technical writers, all of whom have special training in usability theory and practical experience in testing.

The team spent its time as follows:

Activity / Collective Hours Spent / Notes
Conduct planning meeting and review of client's test goals and objectives. / 12
Write test plan. / 8
Design questionnaires and scenarios. / 20
Follow-up meetings (to review test plan, scenarios, etc.) / 12 / 2 follow-up meetings were conducted
Recruit test participants. / 6
Conduct walkthrough test. / 12 / This was conducted with a model "advanced" user.
Conduct pilot test. / 12
Conduct actual tests. / 60 / Formal testing was conducted on two evening sessions and one afternoon session (to accommodate schedules of test participants).
Analyze results / 12 / Analysis was performed immediately post-test; in addition, follow-on analysis was performed prior to compiling the final test report.
Write final report / 20
Assemble/edit "highlights" videotape. / 44 / We completed a videotape as a supplemental deliverable for this new client.
218 total hours

Comments on the Realism of the Evaluation

In general, the Hotmail usability evaluation was fairly realistic. The issues that the client raised were similar to those we have fielded in other tests our lab has performed.

One minor exception was the quality of our interaction with the third-party contact (a CUE-2 team member) who served as our "customer representative." Normally, our actual clients' representatives are heavily invested in ensuring that our evaluators receive prompt and detailed feedback when we request additional information. We did not receive the thorough and prompt response we normally receive from clients when we query them to get more complete information about target audiences, goals for the product, and so on. However, this is certainly normal to expect from a representative who is only "acting the part" and who, in addition to being busy with competing work and professional responsibilities, has been tasked to shield the client from unnecessary contact.

Page 1