Curing the Rampant Waste in Product Innovation

Curing the Rampant Waste in Product Innovation

Why So Many Product Innovation Efforts Fail to Achieve Their Objectives

“We lack a focused, cohesive vision for our company. We want to do everything and be everything — to everyone. We’ve known this for years, talk about it incessantly, but do nothing to fundamentally address it. We are scared to be left out. We are reactive instead of charting an unwavering course. We are separated into silos that far too frequently don’t talk to each other. And when we do talk, it isn’t to collaborate on a clearly focused strategy, but rather to argue and fight about ownership, strategies and tactics. … I’ve heard our strategy described as spreading peanut butter across the myriad opportunities that continue to evolve in the online world. The result: a thin layer of investment spread across everything we do and thus we focus on nothing in particular.”

– Brad Garlinghouse, SVP at Yahoo! in a November 2006 internal memo leaked to the Wall Street Journal and later dubbed “The Peanut Butter Manifesto” (*)

“50% of all advertising is useless. We just don’t know which 50%.”

– David Ogilvy, legendary advertising executive and co-founder of Ogilvy and Mather

Product innovation is both a mantra and a mystical term. It is supposed to occur in one main organization – R&D, or Engineering – but also occurs in subsidiary areas such as product management, product marketing, professional services, outsourced/offshore developers, external partners, and sometimes even in internal end-user departments such as finance, legal, marketing, sales, or support. Unfortunately, the overall track record for innovation efforts in most tech companies is a spotty one, with horror stories everywhere of failed or failing efforts among the many development projects that do end in helping each company succeed.

Product Innovation – A Question of Strategy

As so often is the case when talented and dedicated people fail to achieve their objectives despite all their creativity and hard work, I think you have to start by assuming that everything starts with strategy – or lack thereof – because developers generally don’t fail through lack of interest or effort. What I most often see is unclear or undefined market and product strategies that cause management teams to flail in their attempts to cover all bases, participate in every much-hyped or fast growing product category within their grasp (or their mission), and/or try to fulfill every imaginable type of customer request. Very often engineering take on the challenge “because we can”. Or, sales, marketing or other functions suffer acute anxiety not to lose out to their closest competitors and bully product management and engineering into chasing after them at all costs; or even, in the case of young startups, because their CEO, venture partners or board members egg them on to jump in to new spaces wherever possible to attract new customers and increase the company’s valuation.

This phenomenon is happening every day among companies that are casting their fortunes in hot areas such as cloud, mobile, big data, cyber security, work management, collaboration, and IoT. What’s needed in these situations is much more stringent prioritization. And prioritization is virtually impossible without strategy – which as much as anything is about deciding what not to do as well as what to do.

Peanut Butter & Waste – Root Causes

The reason why the peanut butter image caught on so quickly in the industry and still resonates with management teams today is that it describes exactly the problem that afflicts many companies. Fast-growing young companies nurture big ambitions and even bigger appetites, and they fear being left behind by leaner meaner competitors if they don’t pursue every opportunity in front of them. The faster their growth, the harder they struggle to resource every opportunity; and if their growth stalls, as has been the case with former high flier Yahoo! for so many years now, their cumulative and collective anxiety drives them even harder to do a bit of everything in the attempt to win somewhere. Even under Marissa Mayer, CEO since 2012, Yahoo! appears to have been thrashing around, first recasting itself as a web content publisher and now emphasizing its role as a budding leader in mobility, through fifty or so acquisitions in less than 36 months.

Especially if companies have stalled and thus been lean on resources for a few years, they never have sufficient developers to pursue every product innovation opportunity successfully. This leads to “bubble-up innovation”: myriad attempts in various parts of the company to catch up, keep up, get ahead, fix bugs, get some custom code or scripting into the product to keep customers from mutinying or going elsewhere. The problem is that the more frenetic the action in different functions and divisions, the less effective the outcomes. In other words, what they get into is an extremely vicious cycle from which it is difficult to escape.

As for David Ogilvy’s comment about the waste in advertising, if he had been around to watch the evolution of the tech industry he could easily have made an identical claim about the waste in product development. With advertising, I guess we can all understand some of the vagaries that advertisers and agencies have to deal with; after all, even armed with today’s diverse targeted advertising technologies, how can you know for sure who among a mass of consumers will notice your ad and then take action to purchase your product as a result? Surely engineering, a discipline founded on using mathematics to produce precise answers to thorny problems, when armed with at least some amount of market research and knowledge of customer requirements, should be more predictable and more effective in its use of resources? Well, it turns out that software engineering can often be virtually as much of a black art as advertising, marketing, and sales seem to be to those who work in other fields.

In a nutshell, peanut buttering results from a lack of strategy (= prioritization) and leads directly to waste, in fact sometimes astronomical amounts of wasted resources. This problem is not made easier by the relative lack of transparency in product development organizations. Often it’s almost impossible to identify the waste and reduce or eliminate it without carrying out a forensic examination of every so-called innovation project. Companies routinely plunge into ill-defined development projects in the attempt to make up lost ground against their competitors, or throw an immature new offering over the wall to their marketing and sales teams. Sometimes they try to address the demanding custom requirements of a few major customers without every managing to develop a repeatable solution to solve mission-jeopardizing problems of any one group of target customers. Another frequent sin is sticking with products that almost everyone in the company acknowledges should be end-of-lifed except that there’s a small but vocal minority of customers or salespeople who can’t let go of them just yet.

So, how can we put some order in this chaos? Why is this problem not just about tactics (i.e., which development tools or techniques to adopt, how to obtain the right skillsets to apply to each project, etc.), or execution (i.e, managing projects tightly to get teams to complete development projects on time, on spec, and on budget)?

My vote goes to strategy, first and foremost. Once we get the strategy right, the talents and hard work of engineering teams can take over. Despite the rapid growth in today’s tech markets there is no global lack of dedicated, gifted, and experienced engineers, product managers, and executives to drive for results once they have a clear direction and guideposts to follow.

The Three Types of Innovation Investment

First, we need to agree on what we mean by innovation. For starters, when left to their own devices engineers and engineering-centric organizations tend to intuitively equate innovation to differentiation. “If we produce this innovative product or feature on time, our customers will buy our product/service, and our revenue growth objectives will be fulfilled”. However, our empirically developed models tell us that, perhaps not surprisingly, innovation is a multi-faceted activity. It turns out that, in business terms, there are actually three types of innovation investment that can result in positive outcomes, all of them requiring a distinct, different strategy.

These three innovation types are Differentiation, Neutralization, and Optimization (AKA Productivity). They each have their unique characteristics and resourcing requirements:

  1. Only one of these three, Differentiation, creates competitive separation in the minds of customers (and partners), and thus leads to premium pricing, preference (i.e., winning more deals than your competitors), and higher growth and/or margins than your closest competitors.
  2. Of the other two, Neutralization is the type of innovation that enables you to overcome competitive disadvantage, allowing you to stay in – or get back in – in game, however fleetingly. It’s important to keep in mind that as soon as you (think you) have neutralized a reference competitor’s advantage, they may already have launched, or be about to launch, another version or new offer that puts you behind again. So you need to always be ready to launch new neutralization efforts, as the need crops up.
  3. Optimization, the third innovation investment, provides for cost reduction which allows you to either lower your prices or raise your margins, or a combination of both. An example of an optimization innovation may be the use of new open source technology to complete a new version of an existing offer more quickly and inexpensively than if you had continued to use your existing tools and approach to the project.

Taken together, these three innovation goals account for no more than 50%-65% of most tech company R&D expenditures. That said, if waste can be brought under control, the situation can be significantly remedied. One important note is that deciding which innovation investments are actually differentiating as opposed to neutralizing or optimizing (or none of the above) requires an important strategy exercise unto itself, not to be neglected. But this topic is for another day.

The Damage Wreaked by Waste

There is a green-headed monster called Waste which often accounts the remaining 35%-50% of total development expenditures throughout and outside the company. Discounting a reasonable percentage of legitimate Failed Experiments, which are the only benign form of waste, you might find that all the malignant forms of waste account for “only” 20%-35% of total development expenditures. How come such a significant slice of the pie?

When you consider the following four criteria as major contributors, you will quickly see how waste can become such a significant drag on a company’s innovation resources and effectiveness:

  • Differentiation projects that don’t go deep or far enough, and either just achieve neutralization or miss that outcome too. This can be extremely costly to the company in terms of opportunity cost because the customer preference and premium margins will never be realized. Since differentiation projects typically require a concentration of quality resources, failing to achieve the objectives also costs heavily in terms of opportunity cost.
  • Neutralization projects that go beyond good enough, but take too long to do so. This occurs with great frequency because some or all of the engineers on the project – and maybe their product management colleagues as well – are misguidedly thinking that they will succeed in differentiating their offer against those of their closes competitors, when no such possibility exists.
  • Optimization projects that fail to cost-reduce development. It is painfully common in tech that optimization goes neglected or even completely ignored, especially in younger companies where everyone wants to believe that every innovation effort is aimed at achieving differentiation.
  • Bubble-up innovation projects that end up creating (sometimes massive) redundancy, and/or never coming to fruition. These typically exist in multiple functional areas of every tech company, and the waste is often enormous though very hard to calculate accurately.

Organizing Resources for Multiple Innovation Investments

So far so good, in terms of description; but what does this tell us about why companies struggle so mightily to manage their development projects to successful outcomes, as measured in customer adoption or preference, premium pricing and margins, customer retention, and so on? As implied earlier in this post, one of the main misconceptions among engineering teams is to treat all innovation efforts as differentiating. After all, the desire of every engineer is to work on “cool” projects, by which one might infer that the feature or product they are working on is bound to differentiate the company’s offering(s). But one secret of the R&D business is that not all engineers are ‘inventors’ focused on new creating forms of differentiation from scratch; many more are ‘deployers’ whose best work is performed when adding new features that result in improving their products dramatically or incrementally; still others are more talented at ‘optimizing’ spaghetti code, adopting new tools and techniques, or simplifying a code base that has become top-heavy and unwieldy for maintenance purposes.

Because so many management teams don’t have a model that distinguishes in this way between innovation investment types, the next mistake is to ask each development team to pursue multiple innovation objectives in the same project. In other words, ask the team to differentiate, neutralize, AND optimize all in the same body of work. Bad Idea!

The organizational strategy that we have found to work best to avoid waste, and instead succeed in enabling differentiation where it is feasible to do so, while also neutralizing and optimizing effectively, is actually to separate teams by their different innovation objectives. Most often this requires that you deconstruct large projects into their component sub-projects based on which parts are differentiating versus neutralizing or optimizing. In other words:

  1. Teams working on a Differentiation (DIFF) sub-project will be asked to go as deep and far (and sometimes as long) as needed to achieve the differentiation they are seeking;
  2. Teams working on Neutralization (NEUT) sub-projects will be tasked with getting to ‘good enough’ quickly and economically, and at all costs not to go beyond that;
  3. Teams working on Optimization (OPT) sub-projects will focus on employing continuous improvement tools and techniques – such as open source software components – to continually cost-reduce the company’s code base and offerings.

Call to Action

Getting the DIFF / NEUT / OPT mix right and thus significantly reducing Waste can lead to astounding returns on innovation for companies. The reality is that, if you deconstruct all of the development/engineering projects in motion in the typical tech company, you are likely to find that more resources are aimed at achieving Neutralization than Differentiation, with the smallest percentage of resources allocated to Optimization work. The good news is that no single company generally needs to differentiate in more than one or two vectors – whether in specific key capabilities or functionality – where they genuinely possess crown jewels in potential or in actual form.

But in order to attack this major problem, light needs to penetrate the dark caverns of engineering, and lessons need to be acted on decisively. This requires that the R&D executives and product managers expose their black box secrets to other functional groups in the company. Otherwise the problems that Brad Garlinghouse described at Yahoo! in 2006 and that persists in so many companies today – multiple siloes and very little collaboration, proliferation of different types of development organization (new product engineering, sustaining engineering, professional services, outsourced but uncoordinated offshore efforts, etc.), lack of ownership and accountability for product innovation outcomes between different functions such as engineering, product management, product marketing, sales, pre-sales, and PS, all resulting in unfocused efforts, unfinished projects, and exploding development costs – will continue for many more years. A true, widespread peanut butter nightmare.

Among present-day Saas, Paas, and Iaas companies I know many where product innovation efforts are spread throughout the organization and outside its four walls, among myriad offshore organizations as well as other development partners. In most of these situations, no distinction is made between differentiating, neutralizing, and optimizing innovations. As a result, massive waste is occurring on a daily basis. The main reason that it escapes closer scrutiny is that engineering is so often a powerful and relatively unquestioned constituency and executives in other functions feel unsure about how to challenge it, and where to start.

One thing for sure: much remains to be examined and improved in the often opaque world of R&D.

* Today Brad Garlinghouse is president & COO at Ripple Labs, original developer of the Ripple Protocol for secure funds transfer.


Michael Cannon

about 3 years ago

Brilliant. I would add that differentiation has 2 versions 1) diff. from status quo / better answer to why change and 2) diff. from competition / better answer to why you.


Philip Lay

about 3 years ago

Michael, Thanks for your feedback. Regarding the distinction you are drawing here, as long as the differentiation is valued by customers, we're in agreement. To be clear, we don't count as differentiation something which is different or better than the vendor's prior offer.



about 3 years ago

I'm not defending Marissa, but the pressure in big companies to place big enough bets to grow revenue meaningfully, often precludes experiments that can lead in future years to significant revenue growth. Those types of bets do require a vision for what will be the next big thing. Also, many companies pick a route for differentiation that is not valued by their intended prospects - an extra degree of security, say, when the existing solution is satisfactory. In many of these cases, companies are doubling down on an attribute that they have that is "different" instead of figuring out what prospects would value that they could develop.


Philip Lay

about 3 years ago

Kathryn, Thanks very much for your comment. Indeed, real differentiation is about something that prospects and customers value, resulting in them giving you their preference. As for the struggle that larger companies have in placing their bets, this is generally because they don't have a clear filter for assessing different opportunities. Besides the model described in this article, we use a Three Horizons planning model to distinguish between options that will provide a return within different time frames. Each type requires a different resourcing/org strategy, metrics, and governance.



about 3 years ago

As a product strategy and management professional for many years, I have to agree with your core thesis - that issues with product innovation are usually rooted in the lack of clear (or clearly articulated) strategy and that simple frameworks for separating out the type of innovation, such as those derived from the Porter 5-forces model, may add clarity in achieving desired outcomes, especially in determining how much and how long to invest.However, I would stop short of advocating that the solution lies in splitting teams by innovation type for any of several reasons: a) to your earlier points, mere separation of teams will not prevent, for instance, "Differentiation projects that don’t go deep or far enough" or "Neutralization projects that...take too long", etc. b) in the software business, it is often impossible to tease apart these objectives for practical reasons - they touch the same code base or it is inefficient or impossible to duplicate specialized resources (e.g. "authentication code expert" or "senior developer who knows the billing modules in and out"). This is ofcourse why development teams are most often organized by expertise (shared services model) or software module structure (domain expertise model). SCRUM works in a manner similar to what you advocate, albeit at the time-boxed 'feature' level. c) people dynamics - team members are beholden to the vested interests of their formal managers whether they are in traditional organization structures or virtual teams, which may conflict with the objectives or the implementation approach chosen by the team, and contribute to wasteThe classification of investments get even more complicated with 'platform'-like products e.g. operating systems, database servers, middleware, etc. where the relationship between strategic objectives and the investment choices is a many-to-many graph.In my opinion, what would really help, instead of splitting up teams by innovation category, is creating some notion of a predictive model that describes the expected trajectory of the sub-project in terms of outcomes coupled with whatever data can be collected to track and validate the core assumptions and re-assess the project as needed - the rigor of the model should be calibrated to the needs and capabilities of the team. The other thing that would help, if at all possible, is instituting a management culture that is courageous enough to take bold prioritization decisions in the absence of clear data instead of the painless route ("peanut butter"), course-correct firmly when the data suggests it, and be open to admitting mistakes because everyone makes them from time to time.


Philip Lay

about 3 years ago

Raj, You make some excellent points from the perspective of one who manages product development resourcing realities. I can't fault anything that you say. My main response is to use the model as a forcing function to help PM and engineering management make choices on how to best resource projects by breaking down every product or feature set into as many sub-projects as makes sense to do so, then resourcing according to what you see, and the engineers you have on staff or sourced externally. The key is to go through the exercise and as far as possible not have engineers thinking that they are differentiating when what they are doing is neutralizing or optimizing. This rapidly becomes a resource sink hole, often one of epic proportions. I also take what you say about management discipline to make tough choices, absolutely true. Well put.


Leave a Comment

Please be polite. We appreciate that.
Your email address will not be published and required fields are marked