In Learning

Not all variation is evil. Did I really say that? The six sigma police will be knocking on my door pretty soon. This post examines an idea I find fascinating that sometimes in a product development context (as opposed to a manufacturing or even a service context) increasing variation can actually create value.


Benefits variability

Let’s look at variability in the benefits enabled by a development pipeline. The fundamental discovery nature of product development (customer doesn’t know what they want, developer doesn’t know how to build it, things change) means that we might know, at best, some kind of probability for the likely benefits.

Consider the following project types:

  • Project A (low reward, low risk) has a 90% chance of making a profit of $0.1m
  • Project B (higher reward, higher risk) has a 20% chance of making a profit of $10m

(These two projects have the same cost.)

If I was interested in minimising variability of benefits, then I would only invest in projects of type A (low reward, low risk). If I invest in 10 of these projects, 9 of them will likely give me a profit of $0.1m and one will give me a profit of $0. My total benefits would likely be 9 * $0.1m = $0.9m

On the other hand, if I invest in 10 projects of type B (higher reward, higher risk) Then it is likely that 2 of them will generate $10m and the other 8 will generate $0m. Total benefits would then likely be 2 x $10m = $20m but with much greater variability.

The variability of our benefits has increased but so to has the value to the company! So in this situation increasing variability maximises value.

Investing in product development is like poker in this respect – champions don’t win every hand but they do maximise their overall winnings!


Change the economic payoff function is easier than reducing the underlying variability

The example above highlights that we are not actually interested in variability per se – this is just a proxy variable for what we are really interested in – the economic costs of variability. This economic cost depends on how variability is transformed by an economic payoff function. In the example above the economic payoff of Project A ($0.1m) was tiny compared to Project B ($10m) which prompted us to choose the greater variability of Project B and still maximise value.

It’s often easier to change the underlying payoff function than it is to reduce the underlying variability.  To take a different example; reducing the amount of errors generated quickly gets tricky and expensive. An alternative strategy is to improve the speed at which errors are fixed which would reduce the economic impact of an error occurring. Improving the speed at which errors are fixed is normally (relatively) cheap and easy.

As a general rule, fast feedback is the key to changing payoff functions. For example, within a development pipeline, the best lever is usually to kill bad ideas early (and not try to eliminate them from entering the pipeline altogether) – so make sure the process is quickly able to find out whether ideas are bad!


Development pipeline have a lot of inherent variation

Now I come to think of it, variation is inherent in a development pipeline. No variation = No value creation. Consider:

  • We are dealing with ideas. They are unique in in their size, their scope and when they occur. Can’t do much about this except to hold them upstream until there is capacity to deal with them.
  • We are effectively creating recipe’s for solution, not actual solutions themselves. The recipes we produce are by their nature are also unique.
  • The value created by a solution is the most unpredictable and unmanageable aspect of a development pipeline. As has been said before, understanding the market risk associated with a solution is an inherently tricky problem.

No wonder that IT best practice and the whole lean-agile movement is a lot about practical ways of applying the philosophy of: “try a bit and see!”


Oversimplifications do not help

I hope I’ve given you a hint that it doesn’t make sense to resolve all uncertainty associated with IT development – only that which it is in our economic interest to resolve. Oversimplifications like “eliminate variation” don’t help – there is actually an optimum uncertainty rate which we can’t reach if we focus only on minimising variability and some forms of variation are inherent in the generation of value in a product development context.

Read more on this topic in The Principles of Product Development Flow (tough read but worth it!)

Sign up to receive our newsletter and new posts

Recent Posts
Showing 3 comments
  • Rene Tenazas

    Interesting article. Just a couple of thoughts: 
    — Shouldn’t we frame the decision on whether to choose a low-risk, low-reward strategy vs. a high-risk high-reward strategy within the context of the decision-maker’s goals. In the example above, selecting the low-risk strategy that has a maximum profit of $1MM makes no sense if the decision-maker needs at least $1.5MM in profit to keep his job or to earn his bonus. Even in the best-case scenario, that strategy will fail. His only option is to try the high-risk strategy that might yield $2MM in profit — there’s a low probability, but better to have a 20% chance of success than none at all. 
    Conversely, if he only needs to show $0.5MM in profits, and there is little or nothing to gain by exceeding the target, then he should not try the high-risk strategy. 
    — When doing the analysis, we should consider using prospect theory to adjust the expected value to reflect human psychology.

  • Matthias Marschall

    Your point about “dealing with ideas” is worth a closer look: In production or in services we try to optimize flow of “more of the same”, but when optimizing development we need to optimize flow of totally different ideas. We need to keep that in mind when applying lean thinking to our software development process.

  • Raazi

    At a tactical level, variation is also inherently connected to your iteration length.

    The cheaper the iteration, the more you can experiment and the faster you can take a decision on proceeding or not.On another level, limiting any one user story to maximum 4 days out of a 2 week sprint (i.e. 40% assuming 1 developer team) ensures variation is managed at a structural level.

Leave a Comment

VFQ Blog - Systems Thining