The Slings of Boxes and Arrows — Persecuting Jakob

Posted on | July 29, 2003 | 13 Comments

One of the best things I’ve ever (co-)written is now up at Boxes and Arrows, a review of the Nielsen/Norman Group report Usability Return on Investment

We aimed for a New Yorker-style review — to use the subject as a jumping off point for discussing the underlying issues. So, while about half the review is an evisceration of the reports remarkably flawed methodology and lack of usefulness, the other half suggests steps that user experience professionals can take to begin to appropriately value their contribution.

Believe me when I say that when we set out to write the review, it was not as an excuse to engage in Nielsen-bashing. That emerged only with a close reading (and re-reading) of the report, where it became clear that their approach was so broken that you couldn’t take a single aggregate finding seriously.

The reports findings are predicated on case studies. The sampling for those studies was wholly self-selected: people who submitted cases to Jakob’s site. People are unlikely to submit a failing case, which obviously skews the findings. Anyone with a financial sense would see through this, and the report’s thesis will thusly be discredited.

What I found quite revealing was a small detail that spoke volumes. Each case study features a “return on investment metric” which states the percentage improvement of a key metric (sales, traffic, downloads, etc.) The problem is, we never learn the cost necessary for achieving this improvement. And you can’t calculate a “return” if you don’t figure the initial investment. All we know is how much it improved, not what it cost to get there.

And we’re supposed to take to heart this report’s findings on “Usability Return on Investment”?

I’m frustrated with this report mostly because the user experience profession needs research and analysis that demonstrates its value, and the Nielsen/Norman Group’s prominence means many people outside our profession will look to them, and when they read this misleading report, they might dismiss outright the contribution of user experience professionals.

Enough griping. I hope you enjoy the review for the contribution it makes, particularly in the last third or so, with suggestions for steps we can take to better understand our value. It was a great experience writing this with Scott, a newly minted MBA from Haas, who has the ability to frame business arcana in such a way as to make it digestible to mere mortals such as myself. This was one of those experiences where either of us could have written a decent article, but only together could we have written one this good.

Comments

13 Responses to “The Slings of Boxes and Arrows — Persecuting Jakob”

Ron Zeno
July 29th, 2003 @ 11:52 am

Why did you choose to ignore all the information I gave you earlier on the topic: Business Issues and User Experience?
peterme
July 29th, 2003 @ 12:14 pm

Why do you think that neglecting to mention your input suggests ignoring it? I simply didn’t find it valuable.
Ron Zeno
July 29th, 2003 @ 1:06 pm

My mistake. Actually, you only ignored Mauro’s ROI article, which is by far the best written. You incorporated my observation on Aaron’s article (no consideration of costs), but applied it to NNG’s:

“but without understanding the costs needed to realize that improvement, you cannot actually state a �return.�”

Far better than I recall seeing it said before. Until the costs can be approximated to the same detail as the return, how can anyone find the roi argument at all credible?
peterme
July 29th, 2003 @ 1:23 pm

I downloaded and looked at Mauro’s article, and didn’t like it. Too stuffy, frankly.
Didier Hilhorst
July 29th, 2003 @ 4:21 pm

Excellent article at Boxes and Arrows. Not because bashing NN/g is an enjoyable activity (although it helps), but because this is a report that should not have the merit to be read (or bought for that matter).

I’m currently doing a MSc in Economics & Informatics and even a freshman would throw this report in the bin. Its financial backing is totally flawed. The part on ROI is downright shocking. I think it’s time Mr. Nielsen gets his wallet out and contracts some financial savvy people to do his homework before making statements that would make a CFO laugh so hard it would make him cry.
Jakob Nielsen
August 1st, 2003 @ 6:37 pm

I’ll make you pay, you meddling kids!

No one fucks with Nielsen Norman Group!!!

YOU HAVE UNLEASHED THE FUCKING FURY!!!!!!!
Louise Ferguson
August 6th, 2003 @ 6:38 am

Well, after the last post, I’m not sure this is a good idea , but…

We – at UK UPA – ran an expert panel earlier this year in London. One of the specific questions raised by a member of the audience was whether panel members had encountered the range of ROI quoted in the NN/g report. Nobody on the panel – a mixture of UK corporate usability champions/managers and usability agencies – ever had.
scott hirsch
August 7th, 2003 @ 4:04 pm

Hey Peter,

I wrote a response to the article and posted it to boxesandarrows, but I guess it might make sense here too. But first, Eugene Chen (from AM+A, I assume) commented after me about what he calls the “marketing” impact of UX (or what might better be described as the business strategy impact) as being ultimately more important than the “engineering” impact (the cost reduction). I could not agree more and am glad to hear that other folks are seeing this as an achievable reality rather than an impossibe to touch holy grail. I think BJ Fogg’s, Persuasive Technology is going to prove a valuable resource for folks who want to analyze the behavioral psychology effects of good UX design.

That said, here’s the gist of my comments:

- I think some readers may have gotten caught up in a perception that ROI is used simply to choose the projects with the biggest potential for short-term profitability. While that is most often the case, ROI is more generally a project valuation tool (to assign value to individual projects within a larger portfolio of possible projects). Its purpose is to weigh investment costs against a future cash flow of benefits. Peter and I have both read Cost-Justifying Usability and found it lacking in this investment perspective � the book provides a tremendous amount of detail on using cost/benefit analysis to justify an expenditure, but not to value longer-term investments in gaining customer insight, building brand, and strategic positioning. It focuses more on the short-term and internal goal of reducing costs.

- In order to make better investment decisions, ROI calculations of the type we envision tie �soft metrics� like web traffic, page views, and click paths to �hard metrics� that align with strategic business goals like profitability, market position, and differentiation. Awareness, testing, and tweaking of such calculations gives firms a tremendous opportunity to gain insight into their customers� needs, behavior, motivation, attitudes, and perceptions (read BJ Fogg�s Persuasive Technology for more information). This is where business people and UX/IA people need to meet to develop and share a common vision and common goals. Unlike the problem of figuring out why marketing works, web technology allows us to be in much more control of the customer experience and to collect a wealth of data that traditional marketers could only dream about. Metrics and ROI-type calculations help us to use that data to make better business decisions that manage and meet customers� needs and expectations.

- Peter and I fully agree with comments that ROI is very hard to measure. However, it is not impossible, and if the right metrics are tracked, user experience improvements can be teased out from other mitigating factors. Of course the hard part is choosing the right metrics � my experience is that a good ROI calculation is always a “work in progress” and is never perfect on the first attempt. The process requires lots of scrutiny and fine-tuning by experts *in the company* and not just by external consultants/researchers. However, external people are often very helpful in getting the insiders pointed in the right direction, particularly in facilitating a new type of collaboration between managers and designers � the type Mr. Norman infers is necessary in his famous quote: �usability advocates don�t understand business.� I would add that the opposite is also true: business people do not understand the unrealized value of user experience. The NN/g report does not facilitate such a meeting-of-minds because it is not persuasive from a business standpoint (which was the theme of our review).

- A good point I wish we’d anticipated is that ROI does have a tendency to be abused by unscrupulous or arrogant accountants and managers who are looking for reasons to kill or forward a particular project — but this is true of just about every business analysis tool you can think of and does not invalidate its intended purpose. Corporate scandals aside, business leaders have strong incentives to use the analysis tools they have to make the right decisions for customers, employees, and shareholders — and most do just that. As such, financial analysis tools need to be more than just anecdotal, but rather quantifiable in real terms that business leaders can use to make decisions � or even just to analyze options.

yours in service,

–scott
Don Norman
January 3rd, 2004 @ 12:55 pm

May I respectfully ask you to remove the so-called comment that is signed “Jakob Nielsen”: this is a hoax — it is not from Jakob.

Intelligent criticism of our reports is completely reasonable, but this response is very obviously a hoax. Niether Jakob nor I talk that way. And if you look at the URL provided, it is to http://www.nng.com, which is a company that has no relationship to the Nielsen Norman group.

You owe your readers an apology — you need to do some moderating of contributions and fact checking.

Thanks

Don Norman
peterme
January 3rd, 2004 @ 2:58 pm

Don–
It’s because it is, as you put it, “very obviously a hoax,” that I left that comment. And it’s silly, which I kind of appreciate. And I think that Jakob, and you, have thick enough skins to deal with such idiocy.

Frankly, it’s the original report that is in need of editing. While I suppose it doesn’t need “fact checking” – I don’t doubt the numbers in it, it only goes to show that facts are meaningless if presented meaninglessly.
sapin Mathilde matoo
June 30th, 2004 @ 6:12 am

Nature is not anthropomorphic.
mature wife
March 13th, 2005 @ 1:07 am

Mature thumbs mature sex, mature wives mature women nude. Mature women sex granny gallery, sexy grannies mature ladies. Mature videos mature fuck, mature video grannies. Free granny sex free granny sex, mature thumbnails mature videos. Mature nudes mature hardcore, mature amateur mature amateur. Mature hardcore mature pussy, mature movies mature sex. Granny galleries mature woman, mature galleries sexy grannies. mature girls
brandy
March 30th, 2006 @ 8:24 pm

Holla and Happy Thanksgiving. black fun jack