The Research & Strategy team gets asked quite a bit about eye tracking. We’ve observed that our hesitancy to engage with it as of late can come across as confusing or maybe even unhelpful. After all, we have a Tobii X2-60 hooked up in our lab, and it wasn’t too long ago in the history of our department when we were running eye-tracking studies for new business pitches, existing clients, and that our eye tracking technology was a big showcase of our office tour.
Through this article, we’d like to articulate the Tekzenit Research & Strategy point of view on eye tracking. Our point of view is fueled by a literature review of the most recent thinking from subject matter experts on of eye tracking – we’re standing on the shoulders of giants. As such, from time-to-time, it will be our responsibility to refresh our source material through subsequent literature review, that may nuance our beliefs, but at this time here is where we’re aligned.
Demonstrative vs. Diagnostic Value
Eye trackers have demonstrative value, meaning they can convince the client in a really visually compelling way that their users don’t view the content the same way they do. But, we believe the jury is still out on whether they have diagnostic value, or put another way, can we actually learn what to change in our designs from them?
Eye-tracking Tells Us “Where,” Not “What”
An eye tracker can tell us where the subject is looking but does not directly tell us what the subject is looking at. For example, in the case of predictive gaze in squash the player appears to be looking at “the wall” or “the floor”, but the real stimulus parameter that is controlling gaze landing is the predicted future bounce point of the ball. Consider these other interesting examples that came up in our literature review:
Participants often would acquire the scroll bar without looking at it. They’d move their mouse over to the right edge of the screen and start scrolling, but their gaze wouldn’t leave the center of display. It seemed they were using their peripheral vision to acquire and use the scroll bar.
Participants would orally tell us they couldn’t see something their gaze was focused on. (Women in my life have referred to this as “Male Refrigerator Blindness” — the inability to see something right in front of you.)
Participants often would click on objects they barely gazed at. They’d focus their vision on some part of the screen, then move their mouse to some place else to actually click.
Most eye-tracking studies are done wrong
The main output of the eye-tracking report is a heatmap. Heatmaps are fast to create, sexy, interesting and even used to make decisions about the design in question.
The trouble with heatmaps is they appear to be qualitative representations of a group of users’ fixations when in reality they are quantitative because they are based on statistics. If you are using heatmaps to actually draw conclusions based on an aggregate of users’ experiences, or if heatmaps are the main deliverable, then eye-tracking requires many more test users than traditional usability studies.
What’s at stake, other than conducting poor research, though? Widely different conclusions can be drawn. Consider the differences between the following images based on six different groups of 10 users :
The Nielsen Norman Group (NN/g) who conducts regular eye-tracking studies has uncovered that a heat map requires no less than 30 participants to render stable results. To put that into perspective, throughout Tekzenit’s history of eye-tracking (less than ten studies since 2014) all of our deliverables have featured heat maps with our participant count at around 7 users.
NN/g recommends that if eye-tracking is still needed and the pool of participants to draw from is tight, qualitative eye-tracking can be done, so long as the main deliverable is a gaze plot, not a heatmap – and what’s more, a think-aloud report from each user explaining the “why” behind their behavior (although this is being debated still within the academic community).
While heatmap usage and participant recruitment are the biggest offenders in studies going wrong it’s also important to note briefly a few other factors. Eye-tracking requires calibration which means that not every participant requires – it tends to be exclusive rather than inclusive. Eye-tracking studies reduce the time you actually collect data from your users.
Another consideration is that most of our everyday behavior does not occur with the person stationary in one location, seated in front of a static 2D visual display. Instead, humans move about, to observe a 3D scene from one point of vantage to another. Running, driving, dancing, bicycling and sports are of course “locomotor tasks” – but also cooking, infant care, tool use, and many forms of social interaction are inherently non-sedentary. If we do not understand how active movements of the body and the eye are used to update our representation of visual space then we will not understand vision.
All the big things can be found without eye-tracking
We believe and agree with the Nielsen Norman Group (NN/g) that eye tracking is an option for companies with a highly evolved usability culture, once a company has run hundreds of regular non-eye tracking studies. [[Since 2014, Tekzenit has run less than ten eye-tracking studies. It’s entirely possible that as a company we have not even run 100 regular non-eye tracking studies – we just don’t have the same volume as a Baymard Institute or Nielsen Norman Group]] Therefore, eye-tracking will not be our first impulse for drawing out the meaningful voice-centered insights that we want to be known for.
We believe the usability catastrophes that are costly in lost business are obvious to an observer who sits next to the user, watches where they click and listens to what they say as they think out loud.
Why? Avoid overkill. You can get a huge amount of usability improvements from the simplest of studies that you can run in a few days. No reason to go to the extra expense of eyetracking.
“The smallest usability project will suffice to produce a long list of urgent redesign recommendations.”
Why? Seen vs. unseen. “I expect to see shipping costs before proceeding through checkout. There’s no way I will type in my credit card number without knowing the total I will be charged.”
Eyetracking is fun to watch and produces cool output. It can serve as a good demonstration that users approach designs differently than we imagine. But, should you use eye tracking in your usability studies? Move along, this is (probably) not the tool you’re looking for. Ask instead if a User Acceptance Test been done prior – it should be the gateway to evaluate design problems.
How to Conduct Eyetracking Studies, Nielsen Norman Group, Pernice and Nielsen (2009)
Prospects and Pitfalls in Combining Eye-Tracking Data and Verbal Reports, Helle (2017)
Eye Tracking in the Wild: The Good, the Bad and the Ugly, Lappi (2015)
Eye tracking for visualization evaluation, Goldberg and Helfman (2011)
Eye Tracking in Advanced Interface Design, Jacob (1995)
Eye Tracking: Worth the Expense?, Spool (2006)