Is Lab Usability Dead?

Posted on | October 17, 2005 | 18 Comments

I would love it if we could simply put a stake into the practice of lab usability. It’s run its course, and it’s simply not well suited to truly measuring the effectiveness of designs in the modern era.

Usability engineering was born of a simpler time. Before everyone was networked. When people used computers for Calculation (basic math, spreadsheets, etc.) and Creation (word processors, graphics programs), not Consumption and Communication. When someone could be expected to focus on a single task at hand for many minutes, if not hours on end. When people had one “program” running at a time.

And now we’re in a world people are interrupted an average of every 11 minutes. Where they have multiple applications, and multiple windows within those applications, open. Where people are storing passwords and bookmarks on their machines. Where people are managing multiple devices, from their computer to their cell phone to their iPod.

And what have we done? We still practice “lab usability,” where we invite someone into a sterile neutral environment (either an observation facility, or a conference room), and ask them to use a strange computer to engage in a series of tasks (whether scripted or self-motivated, it doesn’t matter). And so while we might understand how well someone can use our tool in this exceedingly artificial world, it falls short of helping us appreciate how this tool will fit into their ever crazier contexts.

We need to practice methods that factor the complexity and messiness of reality. We need to get out of our labs, and into our users’ homes and offices. Where they can use their machines, with their myriad applications and devices, and learn what works for them.

Case in point: I recently finished a project where we were going to develop a product to help people remember the important dates and events in their lives. A fairly standard process would have involved prototyping of this product, and then bringing people in to “test” the prototype. What we did, however, was field research. We went into 12 homes, and saw how people currently managed their stuff. And, believe me, it’s messy and complex. One participant used: a church address book, a week-at-a-glance, a Palm-style PDA, a simple address-storing-PDA, and an Access database to manage this task. Had we brought her in to test our prototype, we could have found out all kinds of stuff about how she used this prototype in isolation and away from her tools. But we would have learned nothing about how this tool could possibly have integrated itself into his complex web.

We can’t have people come to us. We need to go to them. I can’t ask someone to come into my lab and use my neutral computer. They have bookmarks, files, passwords stored, and all manner of things that get them through their lives. This is becoming increasingly true in our ever-more Web-2.0 environment, as monolithic solutions give way to smaller point solutions. As people store valuable information in a variety of places (local machine, website/hosted application, paper print-outs filed away, etc.).

At Adaptive Path, we’ve become a lot more inclined to use remote usability. We’ve been playing around with a tool called Ethnio that serves as a Webex-like screen sharing environment, optimized for user research. We’re in our offices; the users are in their own environments. And we virtually “look over their shoulder” as they attempt to get things done. We see the instant messages coming in, the interruptions of email; we hear the phone calls or the people walking by their cube. And we do so for less cost (no renting a facility, no video cameras or video tapes, lower incentives for recruitment since it’s less a burden on the user).

Or, we’re going into people’s homes and environments and observing them _in situ_. And we’re realizing all kinds of insights that we wouldn’t otherwise get.

For our DC workshop, I worked up a table for comparing costs of conducting research with 12 participants, for 1 hour each):

	Lab Usability	Remote Usability	Field Research
Timing
Preparation (Plan, Protocol)	1-2 weeks	1-2 weeks	1-2 weeks
Recruiting	1-2 weeks	1-2 weeks	1-2 weeks
Conducting Interviews	3 days	3 days	4 days (5-6 with travel)
Analysis	1-2 weeks	1-2 weeks	1-2 weeks
Totals	4-7 weeks	4-7 weeks	4-7 weeks
Expenses
Incentives	$600-1200	$600-$1200	$1200-1800
Equipment	(Camera, tapes) $400-600	$50 (audio tapes)	$400-600
Travel	$0	$0	$0 – $3000 (Air, ground, lodging, meals)
Facilities	$0 – $1,000/day	$0	$0
Phone charges	$0	$200	$0
Totals	$1000-4800	$850 – 1450	$1600 – 5400

So, let’s get out of our labs and conference rooms and into our users’ environments. It’s becoming clear that it’s the only way to truly understand, and meet, their needs.

Technorati Tags: fieldresearch, usability, userresearch

Comments

18 Responses to “Is Lab Usability Dead?”

Garrick Van Buren
October 17th, 2005 @ 5:43 pm

Thank you Peter. In addition to – the from a simplier time problem – existing usability practices were developed during a time when people most likely _didn’t_ have access to a personal computer, let alone an webmail account or an Amazon history. 63% of customers now have experience with all these tools or at least a solution in place to solve them (as you describe). This reminds me of the ‘usability not usable’ rant I wrote a while back. Thanks for the refresher.
mikel.org | Michael Boyle's weblog
October 18th, 2005 @ 7:37 am

Peter Merholz asked

the rhetorical question, “Is Lab Usability Dead?” in peterme.com yesterday. I think he makes an important point – computer usage is anything but sterile and disconnected these days (and this has been true for a while). Studying usage habits and pattern…
Julie
October 18th, 2005 @ 7:59 am

Hi, I wanted to reply to this comment.

Developing methodology is an art. We truly need to understand the limits of each of the methods and tactics.

In my opinion the lab will always have its place and use.

Using a controlled environment in order to execute usability testing depends on the goal of the test. It is obvious that the controlled environment should be chosen when and only when we need to evaluate if the artificial grouping (information architecture) and if the user interface elements correspond to the mental mapping and needs of the intended users. That means that the exterior elements to the task aren’t being and don’t need to be considered. We aren’t evaluating the multi-tasking factor here.

If data related to the exterior elements (such as nose, multi-tasking factor, the physical potion of the users,…) needs to be captured and processed, we often will opt for contextual analysis (in order to capture those data) and contextual testing (testing in real environment).

Is Lab Usability Dead? My answer is absolutely not.
Peter Boersma
October 18th, 2005 @ 10:03 am

Peter, I believe you are comparing a research activity (field research) with an evaluation activity (lab testing). The two happen in very distinct phases of a design cycle. Sure, an evaluation will help you understand your users better so your next design meets their requirements more closely, but conceptually they happen in different phases of a design iteration.

By the way: If you would have changed the title to “Lab Usability is Dead” (which you do seem to imply in the article), you would have sounded very Jakob-like. Just a warning, you know…
Samantha Bailey
October 18th, 2005 @ 11:54 am

Another value of lab-based testing is that it allows observers and it allows for the creation of video clips (not impossible in the field, but harder to gather) which can then be used in driving home points about the product. When you work in an environment where you have a lot of control over the design decisions and relatively few masters to convince/cajole having observers and making highlight videos is less important but there are circumstances where there is persuasive power in observing and/or creating video–persuasion that can mean the difference between an iterative design phase in a project or an iterative design methodology for a department being *funded*. Your point is well taken that field research is going to drive better results from a pure design perspective (esp early in the design cycle) but lab usability still has a place–this is a case where the baby could be thrown out with the bath water.
peterme
October 18th, 2005 @ 11:59 am

Peter, you know that I know that. I have two responses to it:

1) People *do* use lab usability as a research activity. Which is foolish, for the reasons above.

but, more interestingly, I think
2) lab usability is failing as an evaluation activity, because of the forces I discuss. It made a perfectly fine evaluation activity in a simpler world. Now it provides a false sense of performance, because no one uses tools in such isolated fashions.
peterme
October 18th, 2005 @ 12:01 pm

Samantha, you can get everything you ask for from remote usability testing.
Abe Crystal
October 18th, 2005 @ 12:23 pm

I agree that field research and remote usability testing are under-used and undervalued compared to lab usability testing. This should change.

A few caveats, though:
1) A lot of usability managers (Dennis Wixon being the most prominent) advocate highly iterative design processes with many closely-coupled test-revise-test cycles. The more testing you’re doing, the greater the value of an efficient recruiting and lab setup, and the harder it is to do field tests. Remote usability testing has promise here–but it seems it would be harder to scale than lab testing. In my very limited experience with remote usability testing, it was a lot of hassle to coordinate, set up, and make sure everything ran smoothly.

2) Sometimes you want to test the task, not the interruption. There has to be a balance between naturalistic exploration of users’ lives, information use, etc. and answering the question ‘Can people really use this when they sit down to do it?’ Particularly for applications that people use regularly for significant periods of time, optimizing the task flow is really important.
david louie
October 19th, 2005 @ 11:06 pm

For the most part i agree, but i think your opinion is biased toward the domain which you design for which is ICT’s. There’s still a lot of design being done for high-risk work environments with time pressure (aviation, shipping, medicine) for which lab usability / research can give you specific feedback that you can’t get in an ethnographic study. An example would be recreating a flight simulator and bringing in pilots to test a new cockpit interface to see if it matches their mental models, directs attention to critical information at specific times… in short is usable. Although design within these more industrial domains has definitely warmed up to the idea of ethnograpic research, lab usability research is still considered critical.
Donna
October 20th, 2005 @ 2:56 am

So Peter, thinking about your reply to PeterB, maybe you should focus on the real issue:

Usability testing is not user research.

You have mixed them up as much as the people you are having a go at. I know you know the difference, but you haven’t made it clear here

Lab testing as user research is ridiculous. Lab testing as evaluation can be useful as long as it is considered as a confirmation of a design process, not a research or design activity. Sometimes it is just far too hard to get your new site/service into someone’s normal environment, but is easier to get them into yours.
Zephyr
October 20th, 2005 @ 5:15 am

Anyone else getting tired of reading articles titled “… is dead”?
peterme
October 20th, 2005 @ 7:12 am

A few thoughts…
David, you’re right that lab usability is appropriate for high risk environments with time pressure (aviation, shipping, medicine)… because those are the environments the method was developed for! But those aren’t the environments in which 99% of lab usability is being done.

Donna, your missing my point. I am coming out and saying that Lab Testing As Evaluation should be done away with. At least, lab testing for evaluating web sites. With remote user testing, you get *all* the methodological benefit of lab testing, AND you let that person be in their normal environment.
Steve Mulder
October 20th, 2005 @ 10:23 am

We’re using remote testing more and more for all the reasons stated above. For a tool, take a look at Macromedia Breeze Live, a screen-sharing tool based on Flash, which means minimal setup for participants. We’re quite happy with it.
John
October 20th, 2005 @ 1:57 pm

If it’s dead then why does your company still offer Usability Testing as a research service?

I have experience with remote usability, enthographic studies, and formal lab studies…I think it just boils down to applying the right method for the project?
scale|free
October 21st, 2005 @ 12:52 am

peterme.com: Is Lab Usability Dead?

Peter Merholz: Is Lab Usability Dead?. And now we’re in a world people are interrupted an average of every 11 minutes. Where they have multiple applications, and multiple windows within those applications, open. Where people are storing passwords and b…
nate bolt
October 24th, 2005 @ 11:34 am

Abe and Steve, the difficulty scaling and having participants join sessions will not be a problem in the very near future. Breeze is very cool but still requires a 2mb install as flash object for the participant. You can find more information and tools at the remote usability Wiki here: http://remoteusability.com
Elizabeth
October 25th, 2005 @ 8:25 am

Hi Peter — I appreciate your points about lab usability, but I cannot agree with what I perceive to be a comparison + substitute methodology discussion. I don’t agree that remote UT can get the same results as lab UT because the methodologies and environments differ. Echoing some other comments, UT is not user research. Having someone come to a lab to evaluate some thing is not the same methodology as user research of the ethnographic type/methodologies you’ve discussed. Each has their place and, in my opinion (which is based on my experience as a practicioner and observer), one is not “better” or “worse” than the other – it hinges on what is being evaluated.
Louise
November 21st, 2005 @ 5:06 am

Hi
Curious as how you would perform a test on an early HTML prototype that cannot be loaded externally? Since it’s best to test early in the design cycle surely this example alone is enough to keep lab testing alive?

peterme.com

Links, thoughts, and essays from Peter Merholz

Is Lab Usability Dead?

Comments

Recent Comments

Hello

Archives