Usability Evaluation Report: Facebook Basics
Megan Fissel
California State University, Monterey Bay
IST 526: Multimedia for Instruction
May 15, 2012
TABLE OF CONTENTS
Introduction……………………………………………………………………………………....3
Methodology………………………………………………………………...……………….4 - 5
Results…………………………………………………………………….………………….6 - 7
Summary……………………………………………………………………………………….8
Apppendices………………………………………………………………………………..9 - 12
Introduction
The idea for this e-learning experience came from my observations of how my students (middle schoolers) use and talk about Facebook and the realization that many of them do not know the best practices for responsible use of the site: making certain things private, who to add as Friends, and appropriate posts and photos for the site.
This prototype module is intended to be the introductory lesson in a series of modules that address Facebook safety and appropriate practices. This module’s objectives are:
- to gain familiarity with the layout and organization of the Facebook site
- to develop or introduce basic Facebook terminology: Home, Profile,Public, Everyone, Friends, and Friends of Friends
- to provide a brief lesson on how to set privacy levels for posts (status updates and photos/videos)
The intended end users are middle school students in grades 7 – 8, ages 13 – 14. Most of these students are likely to already have a Facebook account and be active Facebook users, though they will vary in their amount of use, knowledge of the website, and knowledge of how to change their settings. Students are expected to have at least moderate computer skills, and are expected to have a positive attitude towards Facebook as a subject to study, although many are likely to believe they already know how to use the social networking site well enough.
Subsequent modules would further develop knowledge of the privacy and security settings as well as address issues of what is appropriate on the site and avoiding bullying behaviors.
Methodology
The prototype is a Captivate module that begins with a short cartoon video of a potentially embarrassing situation caused by Facebook misuse in order to “hook” the learner, and then describes an overview of the point of the module (Facebook vocabulary and other basic use issues). The module then explains the following Facebook terms: Home, Profile, Public, Everyone, Friends, and Friends of Friends, most of which are followed by a “Your Turn” – a chance for the learner to do something with the information that was just presented. For example, after the overview of the term Home, the learner then must demonstrate that he or she knows how to return to the Home page from another part of the site. Embedded within the module are five quiz slides: multiple choice and true/false questions which test the learner’s level of understanding of what has already been taught. At the end, the learner is given an overview of their score and whether he/she passed or not.
Six students participated in the beta-testing process over a one-week period. The learners recruited for the usability testing are all 6th, 7th, and 8th grade students between the ages of 13 and 14, and are all students at Los Arboles Middle School in Marina, California.All students recruited were either T.A.s during the tester’s prep periods, or enrolled in a tutorial class which meets during the tester’s prep periods. The project was briefly explained to potential testers, and then they were invited to participate if they wished (one student chose to decline). All students who participated are current Facebook users, though their experience with the sight varied widely in terms of how long they have used it (a few months up to more than two years) and how frequently (some log-in daily, one subject had only logged on once or twice to his page).
The testing was conducted in my classroom, on my personal laptop, during my prep periods (6th and 7th periods). The tester was given an overview of the project, my purposes, and Captivate, and then asked to complete the training. Subjects were asked to try their best to figure everything out on their own, but to ask a question if they felt stuck. They sat at a desk near the front of the room, while I sat at my desk off to the side, where I could see both the subject and the computer screen, and took notes as I observed them work through the module. After they were done with the module, I had them complete a post-test survey measuring what they felt they learned, what they liked and disliked about the training, any specific recommendations, and rating scales on how they liked the training, how easy it was to figure out, the timing, and the utility of the quiz questions. I asked each tester to please be honest and not worry about hurting my feelings, as their feedback was valuable to help me improve my project.
Results
FUNCTIONALITY
Overall, the module worked well and students were able to complete the entire module in a timely manner, without asking for much help. One problem was that the “Your Turn” sections did not pause and wait for the subject to click on the correct spot, but instead moved onto the next slide before they were able to perform what they were being asked to do. For the first few testers, they also were confused by the very slow transition after completing a quiz question before the program moved onto the next slide (approximately 3 seconds); this caused some obvious confusion and most then tried clicking around on the page in the meantime. There was also originally a matching quiz question which, though the subjects figured out the drop-down menu function, the background color of the letter options made it almost impossible to see and the testers had to ask for help.
IMPLEMENTATION
I would have preferred to complete the testing online in one of the computer labs (where students would be completing the final training in a real-world large-scale implementation), but this was not possible because the computer labs were full with classes during the only time I was available to carry out the testing. However, the students seemed content to complete the module in my classroom on my computer. We were interrupted once by the custodian wanting to come in to clean, but again the student did not seem to mind. None of the testers were observed to be distracted by outside noises or other possible distractions.
REACTIONS
The testers all appeared engaged for the duration of the module: they were facing the computer and seemed to have their eyes focused on the screen for the entire time or almost the entire time. Three of the six testers laughed or giggled out loud at aspects of the module that were intended to be amusing (at the opening cartoon video or at my “Whoo” and “Eeh” sounds for correct and wrong quiz answers). The average rating of the training overall was a 5.66: in between “I really liked it” and “I liked it a little bit.” The respondents seemed to rate the various aspects of the training as liking it, but not loving it. The testers stated that they liked the cartoon video as well as the cartoon-ized picture of me, as well as the voiceovers and sounds. The most sophisticated responses expressed a fondness for the interactive questions (“I like the interactive clicking on a Facebook page the most.”). Though half of the testers stated that they did not dislike anything, others stated they disliked the quiz itself (“it was too easy ‘cause I already knew it”), the temperature hints on the first “Your Turn” slide, and the “awkward pauses.” As each of these issues was only brought up by one tester, further testing is needed to determine if these would be unlikeable on a broader scale.
EFFECTIVENESS
I believe the prototype accomplished its goals. Although several of the testers stated they were already quite familiar with the website – particularly its layout and terminology – five of the six testers stated on the post-test survey that they learned something about the privacy settings, and half stated they will change how they control their privacy settings as a result of this brief training. For a ten-minute module produced with almost no cost (except for the many hours spent creating it), this is an effective and efficient introductory module.
Summary
Overall, the test learners appeared engaged in the training module and rated it relatively high. As a result of observing the testers and in response to their written and verbal feedback, I changed several aspects of the Captivate module. First, I removed the matching quiz question, as it seemed to cause more trouble than the assessment information it could provide (besides being nearly impossible to see the answer possibilities, it was also an all-or-nothing response, so if they got one of the three wrong, the feedback was only that they got it wrong). I worked to change the button settings on the “Your Turn” slides, but was only successful with some of them. I also changed the quiz slide settings so that the slides move to the next one more quickly once the learners have answered the question and clicked to move on. In several slides, I also added more features and pop-up items so there was less lag time with only one image: on the Friends-of-Friends slide, for example, I added an additional image and then added several text captions that pop up to describe what the voice-over is talking about.
Next steps would be to further develop the module so that there is even more interactivity built in, so the learners are able to do more as they are going through the training, and to perfect the timing and transition of the buttons (particularly in the “Your Turn” slides). I would also like to add an additional review option: perhaps a different character and voice that provides remedial instruction to students who do not pass the quiz.
Appendices
Post-Test Survey(pg. 10 – 11): completed by testers after finishing the training.
Observation Log(pg. 12): completed by the observer/test coordinator while observing the tester using the module (with the back used for general note-taking).
Facebook Basics Post-Test Survey
Please answer the following questions after you complete the training with Ms. Fissel:
Name (first name only, please): ______Age: _____ Grade: _____
How long have you used Facebook? I don’t 0 – 6 months 7 – 12 months 1 – 2 years 2 years or more
Please answer the following questions as honestly as you can. Your responses will help us to make the demo better for future users!
- Did you feel like you learned anything new about…
- what “Home” meansYes / A little / No
- what “Profile” meansYes / A little / No
- what “Friend” meansYes / A little / No
- what “Friends of Friends” meansYes / A little / No
- what “Public” and “Everyone” means Yes / A little / No
- How to control your privacy settings Yes / A little / No
- What was the most useful thing you learned today? ______
- What did you like the best about the training? ______
- What did you like the least about the training?______
- Did anything you learned make you decide to change the way you use Facebook?
Yes / No / I don’t have a Facebook
If Yes, what will you change?______
Do you have any specific advice or anything you would recommend changing? ______
Please give an overall score of how interesting, enjoyable, and engaging you found the training:
7The most awesome thing ever! / 6
I really liked it. / 5
I liked it a little bit. / 4
Neutral: I didn’t like it or dislike it / 3
I disliked it a little bit. / 2
I really didn’t like it. / 1
That was the worst 10 minutes of my life.
Please rate the following aspects of the training on a scale of 1 – 5:
The training was interesting and fun for me to use:
5Yes! Totally agree! / 4
Kind of / 3
Neutral / 2
Not really / 1
Not at all!
It was easy for me to figure out how to get through the training (where to go, what to do next, etc.):
5Yes! Totally agree! / 4
Kind of / 3
Neutral / 2
Not really / 1
Not at all!
The timing of the training was just about right (not too long, short, quick, or slow):
5Yes! Totally agree! / 4
Kind of / 3
Neutral / 2
Not really / 1
Not at all!
The quiz questions let me test my knowledge about these Facebook terms:
5Yes! Totally agree! / 4
Kind of / 3
Neutral / 2
Not really / 1
Not at all!
What I learned today will make me change my Facebook habits:
5Yes! Totally agree! / 4
Kind of / 3
Maybe / 2
Probably not / 1
Not at all!
Facebook Basics Observation Log
Name (first name only, please): ______Age: _____ Grade: _____
Date of administration: ______Period: _____ Location: ______
Time subject began the training: ______Time subject completed the training: ______
Subject could easily navigate through the topics presented in the module:Yes / No / Yes, but asked for some help: (describe)______
Subject quickly learned how to use the buttons to choose a destination/answer a quiz question:
Yes / No / Yes, but asked for some help: (describe)______
Subject completed all the topics within the module:
Yes / No / If no, which was/were skipped? ______
Subject seemed engaged in the module for the majority of the session:
Yes / No
Notes:______
Subject completed the quiz with a passing score: SCORE:______Yes / No / *Subject did not complete the quiz
For quiz questions missed, the subject went back and thoroughly reviewed the content:
Yes / No / n/a
Notes:______