Why research labs fail at innovation

Research labs are old - really old. Edison’s Menlo park lab in the 1870s had most of the elements hi-tech R&D groups try to emulate. But that doesn’t stop every wave of innovators from starting from scratch, attacking the challenges of innovation as if they were the first to try.

In researching the myths of innovation I studied how research labs in established companies work and many miss the point.

There are many reasons for failure - foremost that creating new things is hard - but I posit that labs succeed or fail most often because of their relationships with product teams and not their technological brilliance or war chest of breakthrough ideas.

Here’s why in simple terms:

• Ideas are easy. It’s not that hard to find interesting ideas. They’re everywhere. There are many R&D groups and universities with lots of good ideas. Hell, there are many programmers or marketers with good ideas and given time and some funds they can develop them into impressive prototypes. It’s certainly hard work but it’s the easier part - it’s getting new ideas into products and out to customers that’s crazy hard.
• R&D creates culture conflict. This is the killer - the limits of sociology kills innovation more often than the limits of technology. The jargon is technology transfer, which means “how do we get ideas from R&D into products?” It’s the least sexy role in R&D and it’s never staffed with great talents for either research or product development: so guess what happens? Nothing. R&D sets goals unapproachable for product team’s 3/6/12 month horizons and wonder why they’re ignored 90% of the time, while product teams look at R&D and think they’re insane (10 year horizons? WTF). Managers love to couch this problem with jargon like “innovation pipelines” and “invention waves” but its BS - they don’t see the problem as social: R&D groups and product teams have conflicts & resentment on philosophical, goal and ego grounds and until those issues are confronted little else matters: great ideas will go nowhere.
• R&D never has a service arm. One obvious way to connect R&D to products, or to build good relationships, is to dedicate 10% of R&D as a service to Product team engineers. What problems are too big for release schedules, but too tactical for standard R&D budget? That’s the sweet spot where technology transfer starts. Any R&D manager who invests in closing the culture gap buys chips to spend when R&D has a breakthrough idea that ordinarily would scare the pants off product teams. The problem is that R&D ego would see this role as a concession to product team “superiority”, which is why to my knowledge its never been done.
• The reward models are broken. Often the conflict is created at the VP level: R&D is directed at satisfying VPs - R&D managers bet that when they deliver a great innovation the VP will ditch the old product team and start a new one - but this is radical move for any VP to make (see below) and rarely happens. In the meantime this VP-centric model encourage researchers to treat line level product managers and programmers as dead men walking, instead of as collaborators, prototype testers, or (gasp) peers in developing new ideas.
• Everyone wants to feel creative. Would you want to be told to only work on other people’s ideas? Any R&D group faces the challenge of being in the ivory tower: they’re resented as soon as they walk in the door, especially if they throw their PhD’s and research pedigrees around at engineers with different values. Innovation is a social process that smart motivated people want to participate in - if you propagate the belief that only special people in special roles can do it, something is broken.
• At the end of the day someone has to take the bet . Good R&D minimizes risk but never eliminates it. It produces explorations and potentials, but someone has to say “I’m betting my successful division on new thing X”. That’s a hard thing to say: I’ve never said it, have you? Most R&D managers despite their confidence have never said it either. In the end every failed R&D effort has a Product VP who was unwilling to take the risk, either for good reasons (the ideas were not worthy) or bad (they didn’t believe or didn’t have the the courage).

I admit this is a simplified and biased view: there are more complexities than I suggest (R&D groups have a wide range of possible goals) and I’ve worked with, but never in, R&D groups. So there, I confess, I’m a product side innovation zealot (Not really, but if you hated this post feel free to call me this especially if it’s less offensive than whatever you had in mind).

References:

• This Booz-Allen report evaluates R&D spending and supports the notion that pure R&D matters less, in a business context, than transfer.
• Mark Stefik’s book Breakthrough: Stories and Strategies of Radical Innovation was the best R&D management book I found. It’s interview centric to a fault and not the typical list of do’s and don’ts found in most management books - it attempts to provide a framework for how leaders of R&D should think about innovation with many supporting stories.
• There is a shortage of good books on managing R&D - trade secrets I guess. The closest thing to a recommended reference I found was The human side of managing technological research edited by Ralph Katz, a collection of papers that covers performance, process, invention and human nature.
• The primary reason research labs fail is that successful innovation depends on variables beyond our control - you can do everything right and still not deliver the breakthrough everyone expects - a theme explored in depth in the book.

Posted in Innovation, Management |