The latest issue of interactions magazine, contains Dennis Wixon’s “Evaluating Usability Methods: Why the Current Literature Fails the Practitioner,”. (You have to pay to read it online, unfortunately.) Dennis is annoyed with the current debate within the usability field about “how many users?” This debate recurs every professional generation, and, as he points out, generates a lot of heat with little light.

(The debate has lately been stirred up by Jared Spool’s group, with “Eight Is Not Enough”.)

Dennis, in a remarkably exacting fashion, demonstrates how this debate is pointless, because it assumes (among other things) that the single most important criterion of usability testing is finding the greatest number of problems. He reminds us that our practices derive from engineering, not scientific method. And as such, making smart trade-offs is essential.

From the essay:

[The criterion] is short sighted in that it ignores that problems should be fixed and not just found. If we considered instead a more relevant criterion—namely, how much can we improve the product in the shortest time with the least effort?—we probably would not have asked this question, or we would have asked it in a different form, such as, “What is the best way of deploying the usability resources we have available for this development cycle in order to maximize our beneficial impact on the product?”

Dennis works with Microsoft’s Game Testing User Research, which has published a number of articles that are freely available (if, annoying, in Word .doc format).

One of the best things I’ve ever (co-)written is now up at Boxes and Arrows, a review of the Nielsen/Norman Group report Usability Return on Investment

We aimed for a New Yorker-style review — to use the subject as a jumping off point for discussing the underlying issues. So, while about half the review is an evisceration of the reports remarkably flawed methodology and lack of usefulness, the other half suggests steps that user experience professionals can take to begin to appropriately value their contribution.

Believe me when I say that when we set out to write the review, it was not as an excuse to engage in Nielsen-bashing. That emerged only with a close reading (and re-reading) of the report, where it became clear that their approach was so broken that you couldn’t take a single aggregate finding seriously.

The reports findings are predicated on case studies. The sampling for those studies was wholly self-selected: people who submitted cases to Jakob’s site. People are unlikely to submit a failing case, which obviously skews the findings. Anyone with a financial sense would see through this, and the report’s thesis will thusly be discredited.

What I found quite revealing was a small detail that spoke volumes. Each case study features a “return on investment metric” which states the percentage improvement of a key metric (sales, traffic, downloads, etc.) The problem is, we never learn the cost necessary for achieving this improvement. And you can’t calculate a “return” if you don’t figure the initial investment. All we know is how much it improved, not what it cost to get there.

And we’re supposed to take to heart this report’s findings on “Usability Return on Investment”?

I’m frustrated with this report mostly because the user experience profession needs research and analysis that demonstrates its value, and the Nielsen/Norman Group’s prominence means many people outside our profession will look to them, and when they read this misleading report, they might dismiss outright the contribution of user experience professionals.

Enough griping. I hope you enjoy the review for the contribution it makes, particularly in the last third or so, with suggestions for steps we can take to better understand our value. It was a great experience writing this with Scott, a newly minted MBA from Haas, who has the ability to frame business arcana in such a way as to make it digestible to mere mortals such as myself. This was one of those experiences where either of us could have written a decent article, but only together could we have written one this good.

(also posted to the Beast Blog, but too good to just leave there…)

This weekend and next week, the Pacific Film Archive has a special treat for film lovers.

On Saturday, July 26th, there’s CHARLES LAUGHTON DIRECTS THE NIGHT OF THE HUNTER: A PRESENTATION OF OUTTAKES FROM THE FILM. And on July 30th, they’re showing the film in pristine condition.

I had the delight of seeing this presentation last year in Los Angeles, and wrote it about it (with lots of pictures). If that doesn’t encourage you to go, nothing will.

[Ignore this if you’re not a web geek.]

So, back in my Windows days, I was a fan of HyperSnap, a screen-capture tool that could “auto-scroll” Web pages, which saves one from having to capture one portion, scroll down, capture another portion, then bring those into an image editor, and stitch them.

On Mac OS X, no tool offers this functionality. But I just figured out a way to do it. Maybe somebody else did, but I couldn’t find it.

1. In your web browser (I use Safari), go to Page Setup.

2. In the Settings menu, select “Custom Paper Size”

3. Click “New”

4. Enter the dimensions for your paper — make sure it’s bigger than a typical web page you’d want (I’ve got 11″ x 33″)

5. Give those dimensions a name (e.g., “Screen Capture”)

6. In the Settings Menu, return to “Page Attributes”

7. In Paper Size, select your custom name.

8. On the web page you want to capture, select Print…

9. Choose “Preview…”

10. This will open the page in Preview. You can Command-C to Copy, and then Paste it into wherever you want it.

This might seem onerous, but you only have to do the basic set-up (1-7) once. From then on, it’s just “Print” to Preview, cut, and paste!

On Thursday the 17th, Turner Classic Movies will show Love Me or Leave Me, a “biopic” of Ruth Etting, who made her way from small Chicago clubs to the Ziegfeld Follies and beyond, in a career managed by a Chicago gangster. My dad clued me into the prior airing of the film with this email:

Just in case you haven’t seen this flick, it is as good a movie musical-bio-drama as I have seen. If that seems a limited category, then let me add that one of the very best studio films I have ever seen.

The movie just misses greatness, but Cagney’s performance is either the first or second best of his entire career. I cried at the end of the flick in 1955 because I knew I would never see another performace that good again. And I haven’t.

Doris Day is a revelation. Her acting and singing and body are everything you could ask of a woman. Nobody knew it at the time, but DD was living through her own tormented marriage at the time she was playing this part.

Cameron Mitchell must have been under studio contract, and his performance shows it.

I just watched the whole thing again a couple of nights ago. And TCM is screening it again at 4:30 this coming AM. In the original wide screen format.

It’s yours for the taping.

After I saw the film, I wrote back:

An impressive, impressive flick. Thanks for pointing it out to me — I would have never thought of seeing it on my own.

> The movie just misses greatness, but Cagney’s performance is either the first
> or second best of his entire career.

After WHITE HEAT?

> I cried at the end of the flick in 1955
> because I knew I would never see another performace that good again. And I
> haven’t.

That might be true. He is amazing in it. He portrays a Man perhaps better than I’ve ever seed a Man portrayed on film…

> Doris Day is a revelation. Her acting and singing and body are everything you
> could ask of a woman. Nobody knew it at the time, but DD was living through
> her own tormented marriage at the time she was playing this part.

Which, from what I read, continued another 10 or so years, until her husband died, and DD realized she was broke.

For me, it was a curious film, because I’m not really cognizant of much of the movie’s context–the life of Ruth Etting (I hadn’t realized this was a biopic until after I saw the film); what people expected of Doris Day at this point in her career; or how it stacks up against other contemporary musicals. All that said, the film stands on its own, marvelously, thanks to Cagney’s performance and Day’s character’s shrewdness. The human emotion seems improbably complex for a studio picture of the mid-50s, making you wonder if everyone understood what they were making. I still haven’t figured out if Etting (in the picture) is conniving, innocent, lucky, shrewd, selfish, misunderstood, or all of these things. You don’t expect a Doris Day vehicle (which I’m guessing is how this was initially considered) to offer so many shades of humanity that you leave the picture scratching your head about what you saw.

To which he responded:

All of your comments are right on the button. It can surprise us–the level
of sophistication that sometimes slipped into studio product. But when you peer deeply into the credits of the director and writers – all of them – you can see that these were interesting and serious filmmakers.

HE WALKED BY NIGHT is another excellent movie on this week. I loved it for what it was when it first came out. But when you see Jack Webb in it, you now know where his concept for DRAGNET got its start.

And He Walked By Night is a good film, but that’s a post for another day. . .

Yesterday caught Pirates of the Caribbean, and was pleasantly surprised at how good it was. A rollicking adventure, it has what you’d expect: lots of swords, ships with damned men, parrots on shoulders, cannons, a damsel in distress (who can hold her own in a fight). It also has good stuff you wouldn’t expect — heaps of good humor, Johnny Depp doing an amazing Keith Richards impersonation, delightfully gruesome skeleton special effects (reminiscent of Harryhausen), clever allusions to the theme park ride, and just a very well sustained sense of fun.

Seeing Keira Knightly in Bend It Like Beckham, you knew an ingenue that pretty would be cropping up again, and IMDB shows her to be quite active. I’m curious about Love, Actually — writer Richard Curtis has proved himself repeatedly in the past, from the Blackadder TV series to Four Weddings and a Funeral to the underappreciated Bernard and the Genie. This will be his directorial debut.

At the DUX2003 conference, among the most commonly asked questions dealt with how user experience can better communicate its value to the world of business.

The questions often were of two types: 1. How can I measure my effectiveness? 2. What methods are there to influence others in my organization?

Those two questions are addressed in Adaptive Path workshops offered in Washington, D.C., August 18-21. Specifcally, on the 21st, I will talk about “Tying User Experience to Business Success,” and Janice and Lane will talk about “Managing Design Politics.”

Here’s the outline of my talk:

• Introduction
  • How are you currently valued at work?
• Review the literature on user experience and ROI
  • Nielsen/Norman Group's "Usability Return on Investment"
  • Aaron Marcus' "Return on Investment for Usable User Interface Design"
  • Cost Justifying Usability
• Overview of Basic Product Finance
  • Cash Flows
  • Net Present Value
  • Spreadsheet fun
• Developing a Metrics Plan
  • User Experience metrics
  • Web metrics
  • Business metrics
  • Tie them all together, and incorporate them into a project process
• Caveats for Tying User Experience to Business Success
  • Organizational Issues
  • More Organizational Issues
  • Even More Organizational Issues
• New Frameworks for Considering UX Value

And while I think my talk is pretty good, Janice and Lane’s talk is great. I wasn’t expecting much going in — 4 hours of talking about organizational politics? But they’ve developed a solid workbook that frames the problem in digestible chunks, and incorporates a lot of activities which you can bring back to your company.

And this doesn’t even include August 20th, where Jesse breaks down the Elements of User Experience, and unveils a new way of thinking about web teams, and Jeff tells you what you need to consider before you begin the purchase process for a new CMS.

And if you sign up with the Promotional Code LLY25, you’ll get a 25% discount. That knocks off over $250 a two-day tutorial, taking the cost from $1,095 to $821.

Head here for more info.

Back in the day, I used to take many a thoughtwander. As I’m getting old and boring, they come fewer and farther between.

Recently I’ve found a couple threads that might tie together. The first deals with how people research product information online. The second concerns a hypertext pattern known as “The Cycle”.

I have been particularly keen on product research since I worked at Epinions.com, and watching numerous people try to find the right product to buy.

In our user tests, the archetypal task was to have people research digital cameras. Digital cameras are great because a) they’re common, b) not cheap, and c) somewhat complex.

Back in the day, fewer people knew much about them. We assumed that, given the task of finding an appropriate digital camera, people would whittle down the attributes such as price, megapixel count, and brand, and arrive at the few options best suited to them. If they had questions along the way, they could read helpful guides that would define terms, suggest comparison strategies, etc.

Again and again in our observations, that didn’t happen. People who knew little about digital cameras made no attempt to bone up. Instead they’d barrel through the taxonomy, usually beginning with a familiar brand, and get to a product page as quick as possible. It was only then, when looking at a specific item, and seeing what it’s basic specifications were, did they pause, sit back, and think, “Hmmm. This has 2 megapixels. I wonder how many I want?” Some would look for glossaries or guides, others would read reviews, and some just guessed by comparing the various products.

They would go through this cycle — looking at a product, reflect on their needs, understand concepts, look at another product, reflect again, etc. — a few times. Todd and I came up with a conceptual model, where the user is something like a bouncing ball, falling straight onto a product, then bouncing up, getting a lay of the land, falling onto another product, bouncing up again, but not as high since they’re starting to figure it out, falling onto another product, and repeating until they’ve found the right one.

Three years later, and I’m researching how people acquire financial products, and I’m seeing the exact same things. People looking for a loan will head straight for the rates, even though they have no real understanding of what the rates mean, because they need something concrete to get their head around. Only then do they try to take this information and develop context for it.

The second thread in this thoughtwander comes from the world of hypertext theory, specifically Mark Bernstein’s Patterns of Hypertext. And specifically within that, The Cycle, where a reader returns to a previously visited node and departs on a new path. In a hypertext, this is how readers build their experience of context. By definition, you don’t read a hypertext from beginning to end, nor in some broad-to-narrow hierarchical fashion. You piece together an experience through exposure to its elements, and their relationships. Understanding relationships requires cycling through the material, returning to the same point more than once, and seeing how it’s all connected.

Well, that resonates with the behavior I’ve seen in product research. Websites tend to be designed rather rigidly and hierarchically, assuming visitors will be good little shoppers, and get a sense of all the basic concepts first (learn about megapixels, memory cards, battery life), then figure out their specific needs (I need a camera with 3 megapixels, using CompactFlash, that can take 50 pictures on a single charge), then find the products that meet those needs, and then choose one and be done. My observations suggest that the process is in fact much messier, and requires constant re-orientation on the part of the shopper to remember which variables are important and which qualities they want. (Something Amazon.com takes advantage of, whether intentionally or not.)

Tying these two together led me to wonder (wander?) whether something more fundamental is at play. At the most essential level, we’re seeing the behavior of decision making. How do I figure out what to do next? And decision making seems to rely on the piecemeal gathering of information and the contextualization of that information through their relationships. So, I did some searching on the web for “decision making” and quickly got overwhelmed. It turns out to be a vast and rich field, combining social psychology, applied psychology, economics, business, and lord knows what else. I had trouble understanding just where to begin.

Maybe you can help?

As if things weren’t bad enough in California, along comes an eye-opening and informative story on how the state’s “tough on crime” policies are rending the black communities in Oakland. Considering the delayed effects we’re just now witnessing, it’s likely we’ll see ramifications for years to come. Great.

Also of note is the haunting photograph by Bay Area photographer Todd Hido. Good on the L.A. Times for hiring such a talented craftsman. His work is always worth a look.

(This article was also posted to the Beast Blog, a weblog about the San Francisco East Bay. And thanks to dad for the original pointer.)

Historically, usability engineering is done toward the end of a product development cycle, before something goes to code (and, sadly, often not until after), and usually by that point, the only revisions that can be made are rather superficial. If usability testing demonstrates that there are fundamental flaws with the product, well, too late, we’re too far along, so just make this thing as “usable” as possible and don’t worry about it.

For aeons, usability practitioners have tried to extract themselves from these testing backwaters. They’ve rightly pointed out that you are more likely to make something usable if you test iterations of the design, catching fundamental flaws before they’ve become too deeply ingrained.

In attempting to expand their mandate, though, usability practitioners were dissatisfied with simply testing what others have created. And so they’ve attempted to re-define usability in order that it be, well, more interesting.

A case in point: Whitney Quesenbery’s 5E’s Of Usability, Effective, Efficient, Engaging, Error Tolerant, and Easy to Learn. At first blush, it all sounds fine and good. At second blush, one word pops out as not belonging: “engaging.”

Her definition: “How well the interface draws the user into the interaction and how pleasant and satisfying it is to use.” I would argue that this has little to nothing to do with usability. It has everything to do with “desirability”, a valuable quality, but outside the purview of usability.

These redefinitions stick in my craw. It contributes to the territoriality of folks who ought to be simply striving toward common goals. It’s one thing for an individual usability practitioner to desire being part of the team that makes a product more engaging. It’s another thing to translate that desire into a professional pillar. All that this does is further confuse the issue. Ought I go to a “usability engineer” in order to make my product more engaging?

I’ll leave you to answer that question.

This relates to my earlier post on the word “design.” Usability, whether it’s practitioners like it or not, is perceived as a commodity service. We’ve seen again and again “usability” projects simply go to the cheapest bidder. (To some degree, I’m fine with this… usability, on its own, is a commodity. Usability on its own is not worth a lot. If usability isn’t being performed in the context of a larger user experience process, and that within a larger product development process, I can pretty much guarantee that whatever findings emerge from usability will be ignored.)

This is not to say usability engineering isn’t important — it’s critical. But it’s also critical that the practice’s inputs and outputs stay focused on making things *usable*, that is, making it so that people are able to use the product. Able as in physically able, cognitively able.