Unfinished thoughts on user research

I’ve started a bit of a discussion within Adaptive Path on the subject of design research, and my concern that designers are flocking to research as a way to not have their work so coldly scrutinized. I just wrote up an email to an internal list trying to capture my thoughts, and have realized that it’s to muddled even for the AP blog, so I’m posting it here, because, well, I look to my readership to help me figure out when I’m making sense and when I’ve lost it.

I think it goes without saying that I support user research. I love the insights that we’re able to develop, and I do believe it often leads to superior designs. I have no desire to cast it aside.BUT, what I am reacting to is a trend I’ve been witnessing.

I talk to a lot of designers, both in-house and in the community, and a common thread of late is how much they love research. With a very definite implication that they’re less interested in design, and instead want to focus on how research frames the problem of what to design. I even get a sense that many feel they have grown beyond design, that they’re kind of pooh-poohing the craft of design, that they want to focus on more strategic concerns now.

My concern is that I think this desire to shift toward research is often motivated by research’s lack of accountability. Design is easy to criticize, easy to test, easy to measure. It is relatively straightforward to determine a design’s success (even if that success is determined subjectively by the key stakeholder). Research on the other hand, is hard to criticize, hard to test, hard to measure. The results of research, typically some understanding of the audience and a plan that takes advantage of those insights, aren’t held up to such scrutiny.

I fear this emphasis and idolatry of research, for two reasons. First, it loses sight that research is simply a means to an end — that end being to deliver results. Second, if research doesn’t demonstrate explicit value, it becomes a target for line-item- removal, particularly when things get rough.

Perhaps a tangent, but I think a revealing one, comes from a discussion I had with Jared Spool, when I interviewed him for our website a bit back
=====
PM: What are typical mistakes, or misguided notions that you see when observing others engaged in user research?

JS: I think probably the biggest thing is not understanding the difference between observation, inference, opinion, and recommendation. Those four things are quite distinct and independent of each other. And if you don’t realize there’s a difference, you tend to muddle them up, and then things get very confusing.
=====

In the full transcript, Jared defines them essentially as:

  • observation: what you saw (the user clicked a link)
  • inference: why you think something happened, usually because of causality (the clicked a link because they thought it would take them where they wanted to go)
  • opinion: a statement about the situation based on inferences (the link has a confusing label)
  • recommendation: how to change the situation to achieve a goal (the link should be renamed as ______)

You pretty much have to twist Jared’s arm, or give him lots and lots of money, to make recommendations. Because he doesn’t want to give recommendations unless he’s pretty certain they will lead to the desired change. Which means he needs to collect *tons* of data to give him that confidence. This might seem awfully reductive, but I think it is key, because it makes his research findings *accountable*.

I’m going to leave it at this for now. I’ve probably already blathered too much.

8 thoughts on “Unfinished thoughts on user research

  1. Something, not identified here, is sticking in your craw. Spit it out.

  2. Peter-

    ’bout time you allowed comments again. I’ve been often disappointed at my inability to properly heckle.

    No I don’t think you’ve lost it. The trend is clear, and I’ve been thinking a lot about it. I lived with it for 3 years at ID (Institute of Design). What I hadn’t considered until now is your comment: “I think this desire to shift toward research is often motivated by research’s lack of accountability.” That’s interesting.

    Until now, I’ve thought it might be part of the evolution that designers (and the field in general) go through. Remember when there were lots of usability practioneers? Then we all got bitter that we didn’t have enough influence and some of us decided to get into fancier fields: Interaction Design, Information Architecture, etc. Then some people realize that problems actually go deeper into process and organization. So they take it back another step and focus on design strategy, organization, and “innovation”. It’s why Cooper Interaction changed to Cooper. It’s why ID is teaming up with the business school and creating degrees and programs that are less and less focused on the craft of design. Folks like Chris Conley still bring it into their courses, but the school as a whole continues to march further into “strategic innovation.”

    And in a way it’s easy to argue for. Usability testing can tell you that your product sucks before you ship it. Good IA and IxD can make it better to begin with. But User Research and Strategy can tell you _what_ to make, and how to make a billion dollars while you’re at it. Why would you want to be doing anything else? It’s the most important role of all! Or so I hear :)

  3. “…but what I really want to do is direct!”

    I’ve been hearing the design/research version of this for years and years. Perhaps it’s self-selecting because I’m strongly identified with “design research” so those people approach me to make that statement.

    There’s a flavor of getting OUT of design and getting INTO research that may possibly reflect the weirdness of design as a career path – it’s not widely known or understood and some people find themselves in it educationally->professionally without really knowing the work and the culture until they are in it.

    There’s also a strong sense – as you describe – of not having power or influence and wanting to reinvent in order to get that.

    Grass is greener, I guess. I envy the marketability and camaraderie of people who “make” stuff; who can point to their outcomes. And we still get asked to make sure products don’t suck. It’s not all about coming up with the wild ideas to begin with. It’s a piece of a much larger puzzle.

    Toss in that for the last 10 years any firm that owns a video camera has been putting “ethnographic research” on their website. The companies that we talk to about helping with a capability they admittedly don’t have still put it up on their website. I think the “anyone can claim to do it” nature of research (since after all, you’re just talkin’-to-people) plays some role in the attractiveness of the research alternative.

    But I’m not seeing anything new in these desires. It could just be me, though.

  4. [...] Peterme blathers about designers excited embrace of user research. [...]

  5. Perhaps the opportunity for user research is to help a business inform product folks if the product has value proposition in the first place. Question is – how many companies would be willing to invest in research only to be told that the product ain’t going to fly. Most want to leap into design & build because its fun.

  6. We don’t tell our clients not to do X. We tell them things like “the most interest is in Y” or “aspect Q is the most exciting part of what you are doing” or “In order for people to understand/adopt/embrace X, you’ll have to be sure and provide the following as part of your offering…”

    That is, there’s always a positive, exciting, challenging, rewarding path to follow out of research. [Saying always means that someone can always propose a counter example, but allow me a little hyperbole].

  7. The “research dogma is bad” angle is definitely making the rounds. I guess its about time for a backlash…

    My perspective is that recent design students are acutely aware that the opposite pole from “design strategist” is “production artist” territory and it’s far too easy to get carried downstream when you know how to make things. From that perspective, design research can seem pretty appealing. Design education also contributes by avoiding an over-emphasis on technical skills with the hope that students’ marketability will extend beyond the current version of Photoshop.

    Even though it’s not a direct parallel, I picked up on a counter-perspective from the anthrodesign community:
    http://dori3.typepad.com/my_weblog/2006/12/ad418_final_ant.html
    Elizabeth Tunstall makes the argument that design education actually focuses too much on “practice” and not enough on “research.”

    There’s certainly a balance to be had. The world has plenty of Flash gurus but thinking without making can be just as dangerous.

  8. Research can act as a framework that makes the process of design look more like the process of transactional business, e.g., design decisions look more like business transactions, informed by data.

    A designer who has an inspired idea that looks risky to stakeholders can, through research, translate the decision into a kind-of cost–if all goes well, the stakeholders see the decision as cost-effective rather than “risky.”

    But, to the degree that research happens as part of a risk aversion process, I think it (the research process) can impose limitations from the experience of the past on the designs for the future.