User-Centered Design, Inc.
Is an Eye Tracker Really Useful in Usability Testing?
Name of report1
User-Centered Design, Inc.
Introduction
Eye trackers have come a long way in their ability to track eye movement.Many units are passive devices that no longer require the wearing of awkward headpieces.They are getting almost affordable for many smaller labs.People can be trained on setup and use in a short period of time, and technical issues in actual tracking are improving all the time. Yet, some users—particularly those with bifocal and trifocal eyeglasses—are still hard to track through their glasses. Some eye trackers cannot pick up quick glances, and the targeted position on screen is only an approximation of the actual position being viewed. Even with the best eye trackers, the most reliable tracking occurs when the participant refrains from head or body movement, even though it is not uncommon for a participant to lean into the monitor when they are particularly engaged with an interactive product.[1]
Despite these limitation issues, eye trackers are increasingly finding their way into the usability lab. Vendors of eye trackers claim they provide “clearer results” and that they “unveil information normally missed” [1]. They claim that the eye tracker will tell you “in what order information is absorbed and processed” [2]. Users of eye trackers often create “heat maps” (a type of visual summary of the places where the participants looked) that are shown to eager clients, despite working with sample sizes too small to be generalized. But do eye trackers really provide the insight claimed? Unfortunately, the answer is mostly “no.”
Humans, like machines, can be conceptualized as having three elements: input, processing, and output [3]. Assuming that an eye tracker is fully capable of monitoring human eye movement, and sensitive enough to detect even quick glances at an object, what the eye tracker is actually showing us is the input to the participant at the sensor level. What the eye tracker fails to take into account is the processing that occurs to this input. This processing determines what happens to this input, i.e., whether it results in any change to us cognitively or behaviorally. Also, this processing is rather sophisticated in the case of humans. This lack of accounting for how visual information is processed results in both false positive and false negative assumptions.
Issue 1: We See What We Look At
Eye tracking data assumes that the data that is placed in front of us is something that we actually look at. This is, in fact, false to a large extent.Our perceptual system is not like a camera; it does not dutifully record what is in front of us. The translation from what is in front of us to what we perceive is influenced at every step in the process.From initial signal detection through initial perception, the input is filtered by limits in attention, and then modified by being evaluated and processed through our emotional and cognitive systems.
Consider some of the facts about basic detection that influence what we “see.” The human visual system is a complex combination of signal detection sensors, pre-processing information that occurs within the eye, and processing information that occurs at several locations within the human brain.The human visual system attempts to make logic and order out of what is put in front of us.The pre-processing of information is reasonably easy to demonstrate. It occurs outside of our awareness and outside of our control.
Consider the two pairs of objects in Figure 1 below.The first image is a combination of a plus sign and a dot.Close your left eye and look directly at the plus sign.Then slowly move closer to the image (or print the page out and move it closer to your face).At a point a few inches away from your face, the dot on the right-hand side will disappear. Note that this will only work if you keep your main focus on the plus sign.
Figure 1: The Demonstration of Visual Data “Fill In”
This phenomenon occurs because this procedure places the dot within the optic nerve of the eye—an area without any photoreceptors. But our visual processing system does not allow us to “see” a hole in our vision.We “see” the visual information that our processing system assumes is there—in this case a white field and not a lone dot floating in space [4].In other words, our visual processing system fills in with white space, and what we “see” is not really what is in front
of us.
Now repeat the same experiment with the image on the right. In this case, at the same distance away as in the first experiment, the two lines will suddenly become joined. Our visual processing system will insert a line segment between the two line segments. “Seeing” at this level is unconscious processing of data that enters (or fails to enter) our eye.
In addition, the relationship of other objects in the area in which we look has a profound effect on how we consciously perceive these objects.This phenomenawas studied as far back as 1889 by Müller-Lyer [5]. Consider Figure 2 below [6]. The figure shows a pair of blocks either side-by-side or on top of each other. One block is light in color and the other block is dark in color—or so it would appear. You may be surprised to learn that the two blocks on top of each other in the image below are actually the same shade of grey. To prove this to yourself, place a finger across the edge where the two blocks meet.
Figure 2: The Demonstration of Relative Perception
The gradient that ends the bottom of the top block and the reverse gradient that ends the top of the bottom block are incorporated into our perceptual system to determine what we believe is in front of us, even though it is not an accurate perception of what is actually there.
Another example of our inability to accurately perceive the information right in front of us is shown in Figure 3 below, originally developed by Roger Shepherd
[8, 9].This figure shows two apparently different coffee tables.In actuality, the two tabletops are exactly the same size, just rotated by ninety degrees. (You’ll probably need a ruler to prove this to yourself since just knowing this information will not change your visual perception.)
Figure 3: Roger Shepherd’s “Turning the Tables”
Another effect that needs to be considered in how we process visual information is attentional filtering. At any given moment, our visual field is being bombarded by a vast amount of information. Except in very limited cases, the amount of information presented to us is more than our visual processing system could possibly handle [10]. As a result, the human mind makes rapid decisions about what information within our visual field should be attended to and what should be ignored.
This effect has been researched for many years by Chabris and Simons [11]. Multiple experiments have been conducted to demonstrate that we cannot process all of the visual information presented to us, particularly when we are trying to think (a condition inherent in a usability study). Chabris and Simons coined the term “inattentional blindness” to describe this phenomenon. One version of the original experiments showing this effect is available on YouTube ( though many other examples can be easily found on the web.
Inattentional blindness and its close relative, change blindness, are not random events. Though they are unconscious processes, they can be shown to be purposeful in nature. We process the information we see in front of us and make a decision about whether it fits or not into what we are thinking. If it doesn’t fit, we often perceive it to be different from what it actually is in order to make it fit our decision, or we ignore it.
For example, this author was involved with research into the use of subliminal messages in commercials. Subliminal messages are generally one or two frames within a film intended to send a message to the viewer without them noticing it. However, in one case the subliminal message was a full second of video—enough visual information that it should have been perceived by all observers. However, the video, which happened to be of a naked woman, was placed within a commercial for a bank. Most observers failed to notice the presence of the woman at all. It had no logical place in a bank commercial, so it was consciously ignored (though probably dismissed unconsciously as a memory and not a visual experience).Those observers that did notice the woman explained the woman’s behavior by stating that she was taking a shower—which was, in fact, not the case. For these observers, the presence of the image was distorted to make it more acceptable.Virtually no observers observed the visual event accurately.
The Chabris and Simons experiment discussed above (which used a gorilla instead of a bear) has been studied with an eye tracker. According to Chabris and Simons on their research into this phenomenon:
“Sports scientist Daniel Memmert of Heidelberg University ran our gorilla experiment using his eye tracker and found that subjects who fail to notice the gorilla had spent, on average, a full second looking right at it—the same amount of time as those who did see it!” [12]
In other words, many of the people who fail to “see” the moon walking bear are very likely to have fixated on it within their visual system but discarded it as not fitting within the context.
Therefore, knowing that a person’s eyes were directed at a spot on a screen for a specific period of time (a “fixation”) does not tell us: (1) whether or not they processed what their eyes fixated on, (2) how accurately they processed it (assuming they did), or (3) how that processed information was integrated into memory (if it was).
Issue 2: We Need to Look at Something to See It
This other issue with eye tracking data is the assumption that we only see what we fixate on. This is also untrue.
The human visual system is made up of two types of vision: foveal vision and peripheral vision. Within the approximate center of each eye is a small (about 5mm), yellowish region called the macula. Contained within the macula is the fovea—an area of highly packed photoreceptor cells. This area of packed cells (generally known as our central vision) provides us with high acuity color information.The central vision provides us with a visual field of approximately 13°.
To “see” an object with our fovea does require us to look directly at the object. And this means that the central vision usually corresponds to our attentional focus, since we usually purposefully move our eyes to cause the object of our attention to fall within this region of our optical system. This is what an eye tracker is tracking.
However, outside the macula is a much larger area of cells that comprise our peripheral vision. This area of the eye, having far fewer photoreceptors, produces information of significantly lower acuity but is much better at detecting black and white information (which means it works well even in low lighting conditions).It also contains a number of specialized cells for detecting things such as horizontal and vertical structures, edges, and movement. The peripheral vision extends the field of view from approximately 13° to approximately 180°. Information presented to our peripheral system is processed, even at the lower visual acuity, for a different purpose than information presented to our central vision.
These two types of vision work together. Researcher Hans-Werner Hunziker [14] identified three primary functions of our peripheral vision: (1) the identification of similar forms and movements, (2) recognition of well-known structures, and (3) the delivery of sensations that form the background of our detailed vision. From a design perspective, the most significant element of our peripheral vision is how we use the information obtained from it to determine how much attention (if any) is needed to better classify an object that is detected in our periphery. In other words, objects detected within our visual field are processed (1) to determine if they are already of a readily identifiable form without requiring direct focus, or (2) to direct our focus on them to get more information.If we believe that we know what the object in our peripheral vision is, we continue to focus on the objects currently within our central vision and do not redirect our eye’s central focus to that peripheral object.[2]
Many people are trained to consciously process the objects within their peripheral system.Astronomers are taught to use this area of their vision to detect faint objects more clearly. A dim star, for instance, is best seen when your eyes are not aimed directly at it.[3]Anyone who plays basketball, soccer, hockey, or any other sport involving passing between its team members learns to make a “blind pass” by passing to a player they are not looking at directly. (Otherwise, the pass is known as being “telegraphed.”) But the bulk of peripheral vision processing is done at an unconscious level.
An example of this phenomenon is common to most people who have conducted a usability evaluation and is often described as “banner blindness”—the tendency for people to ignore items in their peripheral vision that look like ads or banners [15]. However, this is often mistakenly described as users failing to “look at” objects in their peripheral vision. When asked why they did not look at the object, respondents often state that they “recognized” them as an advertisement or some other information that was not pertinent to the task at hand. (This is often a false assumption about the object, since designers often put objects in the user’s peripheral vision that are not advertising and are supposed to be used for the task.) This demonstrates that these objects are seen, processed, and affect the behavior of people without ever having to look directly at them.
Since an eye tracker is associated only with central vision, eye tracking data ignores the information that is processed in our peripheral vision (which, based on visual field size, is approximately 72% of our total visual field).
Issue 3: As Observers, Seeing is Believing
Discussions about the limitation or value of eye trackers are not new. Professional forum discussions occur regularly with strong opinions on both sides. Some researchers warn about the limitations and about making false inferences, but people who have attended usability sessions where eye trackers are used are quite certain of their value and of the validity of the data they are seeing.They state they have seen eye trackers in use and know they work. This may be the most concerning issue of the use of eye trackers.
Presented with the information that an eye tracker can produce, limited though it may be, observers are able to provide rational explanations for what they are seeing.But humans are inherently wired to try to determine cause and effect relationships between observations.This tendency creates a set of reasoning biases,including the post hoc or false cause bias (a tendency to assume cause and effect relationships between two elements even if that relationship does not exist), the confirmation bias (a tendency for people to identify information that confirms preconceptions regardless of whether the information is true), and other biases [16, 17].These tendencies form a powerful basis for rationalization—beliefs that can be explained in a rational and logical way but may often be untrue.
Since rationalization is internal (and therefore internally validated), these beliefs are powerful and difficult to challenge, even if incorrect. Given the limitation of the actual data present, coupled with the certainty with which observers believe what they see can be explained, observers can arrive at false but powerful conclusions.
Even if people could avoid these biases (which is highly improbable), watching the real-time display of eye tracking data during the test (a common way in which eye trackers are being integrated into usability testing) tends to distract the observer and perhaps even the facilitator.It distracts them from “being in the moment” with the participant and therefore being able to detect and understand the participant’s reactions and behaviors. Such events in testing are vastly better indicators of how “usable” a product is compared to participants’ eye movements.
Observers, also human beings, have limited attentional focus, and movement is a strong distractor.Observers tend to follow the eye tracking data and are unable to process other information, such as tone, intonation, body movement, facial expressions, and other elements that provide valuable data about the user experience—data arguably more valuable than what is obtained from an eye tracker.
Conclusion
Eye trackers are not bad things. There are specific cases where an eye tracker can provide useful information. These tend to be in controlled search settings where eye tracking behaviors are studied directly.But even in tracking research, eye tracking data is often compared to other data collected, and there are specific cases where an eye tracker can provide some useful information in a usability study. For example, when testing alternate visual layouts (where the participant does not have to engage with it beyond looking at it), viewing eye tracking data can give some insight into which elements attract the user’s eye (though additional investigation is required to determine whether and how any fixations were processed).But even in this case, it is only possible to investigate fixations that are processed at a conscious level.If a fixation was processed at a subconscious level (where most psychologists believe that approximately 98% of our decision-making occurs), it is, by definition, impossible to discusswith the participant how it was processed and how they perceived it.[4]