Why designers fail: the report

Last week I announced a study on why designers fail – exploring the reasons why designers, and people who work with designers, believe designers don’t achieve the results they desire. I presented the results as UIE 13 last week, and as promised here is a summary. Prize winners will be announced soon.

Top line summary

The survey consisted of 41 issues, divided into three categories: Psychological, Skill and Organizational. Each participant ranked each issue on a scale from 1 to 5, with 5 meaning the issue was highly significant in explaining why designers fail, and 1 meaning least significant (3 was identified as a neutral value).

The 389 survey respondents self identified as:

Designer		33.7%
Project manager		16.5%
Programmer / Tester		11.8%
Usability engineer		9.5%
Group manager		6.9%
Business / Marketing		3.9%
Documentation		1.3%
Other		16.5%

The top 15 issues, ranked by average scores were:

People in non-design roles making design decisions	4.18
Managers making design decisions w/o design training	4.14
Designers don’t seek enough data before designing	3.92
No time is provided for long term thinking	3.81
Not receptive to critical feedback	3.69
Lack of awareness of the business fundamentals	3.66
Only lip-service is paid to “User centered design”	3.64
It’s never made safe to fail or experiment	3.62
Designer’s power diluted by too many cooks	3.60
Over-reliance on one kind of design style	3.54
Poor collaboration skills	3.51
Poor persuasion / idea pitching skills	3.49
Poor communication skills	3.49
Poor understanding of domain	3.48
Pressure to use first solution, not a good solution	3.45
Big Ego / Expects others to cater to their whims	3.41

Average scores per grouping

The average scores for groupings showed little different in weightings: there was no single grouping of issues that proved to be significantly more important in explaining why designers fail.

Organizational issues:		3.37
Skill issues:		3.15
Psychological issues:		3.11

Managers vs. Individual contributors

(Note: these charts are quick and dirty. If you have a pretty design stick and know how to use it, happy to share the data so you can make better charts).

One research question of the study was to see how individual contributors and managers varied in their thinking on failure. 49% of those surveyed identified as playing a lead or management role.

Designers vs. non-designers

Another research question was how designers and non-designers results would compare. The results showed only minor variance in how designers and non-designers view causes of failure.

Conclusions

Many top reasons for failure are not typically considered design issues, such as collaboration skills, persuasion skills, and receiving critical feedback.
General consensus on top issues: managers, non-managers, designers and non-designers all had highly similar scores.
Nearly half of all respondents took time to write in additional issues and thoughts. There was a great deal of interest in discussing this topic further.

Background and Disclosures

This survey was designed primarily for qualitative use and as a basis for further discussion and research. I’m sure there are flaws and bias in the study design but I believe this study is valuable anyway.
The issue list was based on 3 things: my own experiences managing UX design training for Microsoft from 1999-2002, many years of debating this question at drunken design conference receptions, and this discussion on a previous blog post.
The issue descriptions listed above were modified to fit in the post – see the actual survey if you’re suspicious of leading questions or other survey bias issues.
The survey was distributed via this blog, the iXda mailing list and the pmclinic list.
A PDF version (95k) of the actual survey can be downloaded.

Here is a PDF version of the full UIE 13 talk on Why designers fail (6MB PDF).

I’m looking for folks to help continue this research. If you’re interested leave a comment.

If you liked this there's more: please subscribe to this feed or follow me on twitter

Posted Monday, October 20th, 2008 at 1:11 pm

in Design, Management, creative thinking | 43 Comments »

Steve Baty - October 20, 2008 at 3:37 pm #

Scott,

Could you clarify the two comparison charts for me (Manager v Individual; designer v non-designer) and confirm that the Individual and Non-designer roles were exclusive sub-sets of the full data set? That is, there was no overlap between the designer/non-designer data; or the manager/individual data sets?

The correlation between responses seems remarkably strong.

Steve

Scott - October 20, 2008 at 4:03 pm #

Steve:

Good question. Each individual was forced to provide two pieces of data:

1) are you a designer
2) are you a manager

The two charts (Designer vs non, Manager vs. non) shown include ALL of the survey responses, simply divided into two piles based on the above pieces of data.

So yes, there was overlap. You could identify yourself as a designer and a manager and have your data show up on both of those charts.

Just for sanity I’ll go back and make sure I didn’t do anything really stupid, but the data is different, just not significantly so.

Lee - October 21, 2008 at 2:43 am #

Interesting results Scott.

I didn’t have time to complete the survey unfortunately, but some of the key findings certainly correlate with my own experience.

The first two findings though seem extremely similar to me – non-designer making design decisions, and managers making design decisions. Aren’t these the same thing in many cases?

Anyway, apologies if this has been discussed before, but it seems like these two issues are common causes of problems.

Congratulations on the good work anyway, and it’ll be interesting to see where you can take this!

David Pashley - October 21, 2008 at 4:39 am #

Interesting post, but your graphs let you down. They shouldn’t be line graphs, as the X axis is not a continuous value, but categories. A bar chart is a better graph to use.

Scott - October 21, 2008 at 8:51 am #

Hi David: The main goal of those two charts was to compare results across all 41 issues, and after experimenting with different charts the range of differences was easiest to see with a line graph, especially at the 400 or so pixels that fit in a blog post. Technically you are correct, a line chart implies non-discrete data, but I chose ease of use for understanding the chart, over technical accuracy.

Frankly, I found all the charting tools I could get my hand on totally sucked. Excel in particular. If anyone pointed me at a good charting tool, I’d update the charts with something that looked less like it was made by an 8th grade science teacher.

Dorian Taylor - October 21, 2008 at 9:49 am #

Scott, going over your list, I see a few patterns emerge of larger undercurrents to these problems. I’ve taken the liberty of grouping them together and attaching remarks.

People in non-design roles making design decisions 4.18
Managers making design decisions w/o design training 4.14
Not receptive to critical feedback 3.69
Designer’s power diluted by too many cooks 3.60
Big Ego / Expects others to cater to their whims 3.41

Design is often associated with power because the team ends up doing what the designer prescribes. This may end up in the hands of the manager, a senior programmer, or in contemporary Agile circles some kind of consensus. In the former cases there is a conflict of interest, and in the latter there are several. A way to defuse this effect is to officially recognize design as orthogonal to, rather than above, other production-oriented work, and assign the job of enforcing conceptual integrity to one brain only.

Designers don’t seek enough data before designing 3.92
It’s never made safe to fail or experiment 3.62
Over-reliance on one kind of design style 3.54
Pressure to use first solution, not a good solution 3.45

I’m not sure what it is that compels us to expect that we can concoct a design in a vacuum and then unveil the completed artifact to an audience and have it work perfectly the first time. I blame TV.

Lack of awareness of the business fundamentals 3.66
Only lip-service is paid to “User centered design” 3.64
Poor understanding of domain 3.48

I think the designer is easily confused as to what their role is. Design really is a muddy term. I’ve met far too many artists (or programmers, or managers) that consider themselves designers, and I’ve met far too many designers that understand themselves to be engineers (or marketers, or CEOs, or whatever). To me, design essentially boils down to conceptual integrity, and therefore a designer is the one in charge of that.

Poor collaboration skills 3.51
Poor persuasion / idea pitching skills 3.49
Poor communication skills 3.49

I see these elements as being endemic to professional life.

No time is provided for long term thinking 3.81

And this one as being endemic to business.

I could slice these points a few other ways, but I’d be here all day! Cheers.

Jenn_lee_ca - October 21, 2008 at 2:34 pm #

Just reviewed the results. And from experience, I think it boils down to:

Poor leadership with a poor vision.

All strategic work should stem from having a clear vision. And all actions (or tactics) should support the strategy. In plain language, are the things that you do, the services that you provide, the products that you build consistent with your vision?

Does leadership reward the values that they want in a public manner?

Do the leaders make it crystal clear on the vision of the organization? Do employees know how they help contribute to the vision?

Do the leaders seek to understand the customer? Do they empower their employees (and reward their employees) to talk in various formats to customers?

Does the company walk the talk?

I think design fails because everyone is racing towards some place but no one understands why they are racing and what the track looks like.

Pedro Ornelas - October 21, 2008 at 2:50 pm #

As with many things in life, it’s dificult to isolate an independent variable for a cause/effect relationship.. probably the combination of some of those at the same time produce failure (non-linear relationship).

I know this is an obvious observation, but the truth it that we still rely on linearity to draw conclusions on complex systems! Just look at what happened to the financial system!

my 2 cents

Lynn Cherny - October 21, 2008 at 4:11 pm #

Office 2007′s version of Excel would have helped you make prettier charts. If you want me to, I’ll do them for you. I also think the numeric averages need bar charts.

Yes, I’m volunteering to help with further work as I said before-!

greg - October 21, 2008 at 11:36 pm #

Ditto on the bad presentation of the discontinuous data in the charts. If the point is to compare results across two groups, I would be inclined to aggregate the data, and highlight the specific results with larger intergroup differences.* As to charting software, I used to use Sigmaplot when I was in working in research, but don’t know if it’s still around.

I missed the original survey, but I’m interested in the question itself. Who says designers fail? Why do people assume that they do? How do we mesaure design success or failure? Do designers “fail” because the expectations put on them are unrealistic? Using “meaningful criteria”(TM), how frequently *do* they fail?

*PS. I don’t know Edward Tufte and haven’t (yet) read his book…

Gagan Diesh - October 22, 2008 at 9:08 am #

Hi Scott:

I am wondering if the skewed nature of the study responses, is a problem? Would the graphs for example not be more meaningful/different, if we looked at equal representation between designers and business stakeholders?

I think it would be interesting to also use design to do some data visualizations around this study if it produced more complex data ponts that have not been shown above!

Thanks, I support studies that help bridge the gap in design communication, management and buy-in.

Scott (admin) - October 22, 2008 at 9:30 am #

Gagan:

I dont know what skewed nature you’re referring to. You have to be more specific. Do you think the survey respondents were skewed? The data?

Ian Swinson - October 22, 2008 at 3:01 pm #

Hi Scott,
Thanks for kicking off this research and discussion. I’ve been working in design for 15 years and have encountered so many situations that set up designers for failure, and so many designers who, regardless of the situation, set themselves up for failure.

Right now my biggest concern is creating a highly successful multidisciplinary design team in a fast-paced, agile, rapid iteration software environment. We’re constantly hiring so we have the added challenge of training seasonal professionals and complete newbies.

Consider me interested.

ian

uxdesign.com - October 25, 2008 at 12:33 pm #

Design means making decisions. This can put designers and managers, if their relationship isn’t already grounded in trust and mutual professional respect, in immediate, if not explicit, conflict. Inexperienced and immature managers (and above) get a charge out of playing de facto art director. This is like telling your doctor how to operate on you. No good can come of it.

I wholeheartedly agree with Dorian T. Oct. 21. Great comments. I would add that confusion (lack of distinction) between art and “design” is partly to blame, as well as TV.

The best arbiter of conflict between a designer and manager is a user. Choose them wisely, and use their feedback more wisely. Distinguish between what they need versus want. Same for the client.

Designers are idea peddlers. We have a responsibility to persuade. Language shapes our thoughts. As with colors and color combo’s: “The difference between the right word and almost the right word is the difference between a lightening bug and lightening.” -Mark Twain.

I’ve come to see the “soft skills” of facilitating collaboration, communication, empathy, patience, campaigning on design’s behalf, creating a good environment for design, etc., as even more necessary than the techniques and methods and tool using “hard skills” we must also bring to bare. Turns out the soft skills are harder. Maybe just from lack of good training? Oh, we’re back to TV again.

“Managers Who Love Design and the Designers Who Don’t Love Them:
http://uxdesign.com/ux-management/article/managing-design/17

Mark - October 28, 2008 at 9:12 am #

Reason #8 on the top 15 issues for “Why designers fail:”

It’s never made safe to FAIL or experiment.

So if it is made safe for us to fail, then suddenly we stop failing. I find this really funny for some reason. It is really cracking me up. Don’t get me wrong, I get the concept, it’s just the wording that sounds so funny. Make it safe for me to fail and I won’t fail so much. Oh I’ll still come up with designs that fail but now we’ll call that a “learning exercise” instead of a failed design.

No wonder people don’t get designers…

steve kennedy - October 30, 2008 at 4:46 am #

Scott:
Great job!
I am a designer with my own practice and teach design and typography at the college level. I am very much interested in continuing this discussion especially as it relates to students. Stressing the fundamentals of research, brainstorming, critique, business classes for design students, collaboration, presentation, and communication skills would help eliminate or mitigate most of these numbers to skew much lower.
As for the following problems:
People in non-design roles making design decisions
Managers making design decisions w/o design training
No time is provided for long term thinking
Designer’s power diluted by too many cooks

These are historical problems designers have never had control over. Having the savvy and experience when NOT to take on a project or when to fire your client usually are skills learned OJT.

Tired of DesignSnobs - December 26, 2008 at 6:40 am #

Gee, how predictable . . . your replies are mostly from designers and SURPRISE, they think that it’s others that mess up their work.

Wow, what next . . . story on dog biting man?

From a non-designer’s perspective, designers taking too much damned time on a ‘pencil’ when they were given specific instructions to ‘give me something I can just show the client’, is what messes things up most.

LISTEN, snobs, you’ll be better off.

Arun - February 9, 2009 at 9:57 am #

I ‘m not sure if this has been mentioned but my observations has been that designs (including mine) fail most often due to insufficient time developing it.

Most of the time up-front design time is all that is available and this is too short especially if the designers are less skilled.

But even after the designs are made, if there is no flexibility in changing it (as the designer themselves if tasked at implementing it often do) or no time available to do so, the designs would stand a change of making a bigger-bad impact than otherwise.

Edge cases and known subtleties are often hidden to the maintainers or 3rd parties who often take it up, as it so often happens in big corporate places. Continous improvement is un-avoidable even with world class designs, which are i feel sure to show as history continous improvements behind them. No one really makes systems that are world class at 1.0 these days.

IowaBiz - February 4, 2009 at 6:59 am

All That Sparkles……

I hope you’ve enjoyed the recent interviews with Chad and Kristin … I’m lining up a new set of interesting people/topics for early spring, so be on the lookout. Right now, though, there’s a more pressing issue to discuss: laundry…….

Ãâ€”ÃÂ°Ã‘â‚¬ÃÂ¸Ã‘ÂÃÂ¾ÃÂ²ÃÂºÃÂ¸ ÃÂ¿ÃÂ¾ Ã‘â€šÃÂµÃÂ¼ÃÂµ | Fresh: ÃÂ½ÃÂ¾ÃÂ²ÃÂ¾Ã‘ÂÃ‘â€šÃÂ¸ ÃÂ¼ÃÂ¸Ã‘â‚¬ÃÂ° Ã‘Å½ÃÂ·ÃÂ°ÃÂ±ÃÂ¸ÃÂ»ÃÂ¸Ã‘â€šÃÂ¸ - March 17, 2009 at 10:03 pm

[...] ?? ??????????? MX’09, ??????? ???????????? ???????? Adaptive Path, ?? ?????? ?????? ??? ???????? ????????, ??????? ?????? ??? ?? ???? ??????????? ??????? ???????? ? ???? ?????????? ???????. ????????, ??? ??? ?????? ?????? ???????? ?????? ?????? ??????? «Why designers fail & what to do about it». [...]

Ada Lovelace could kick your ass « Scott Berkun - March 25, 2010 at 12:55 pm

[...] Few people know that Ada Lovelace was likely the first computer programmer in history. She worked with Charles Babbage, a man who is most famous for making a machine that didn’t quite work. A testament to the role of failure in the making of every success. [...]

Seven Must-See Web Design Videos and Presentations - Smashing Magazine - April 24, 2010 at 8:48 am

[...] the perfect cases doesn’t inform as much as the fail cases do.”Related Link:Why Designers Fail: The ReportGreat Designers StealSpeaker: Jeff VeenA very inspiring and informative talk based on a classic [...]