All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.
– Arthur Schopenhauer
In 1714, the British government offered a prize for a method of determining longitude at sea, with an award of £20,000 (three million pounds in today’s terms). John Harrison, a Yorkshire carpenter, worked on the project for several decades and eventually in 1761, came up with a design that proved accurate on a long voyage to Jamaica. The scientific establishment refused to believe that a Yorkshire carpenter could possibly have solved the problem that had stumped the best scientific minds. In 1773, when he was 80 years old, Harrison received a monetary award in the amount of £8,750 from Parliament for his achievements, but he never received the actual prize. A Yorkshire carpenter was the wrong person to have solved the problem.
In 1865, Gregor Mendel, an unknown professor read a paper at two meetings of the Natural History Society of Brünn in Moravia, giving the results of studies in which he had cultivated and tested some 29,000 pea plants. His study presented a solution to a problem that had stumped the finest scientific minds. The paper was ignored by the international scientific community for the succeeding 35 years until it was eventually realized that Mendel had indeed come up with the solution. His work later became known as Mendel’s Laws of Inheritance and he was hailed as the father of modern genetics, but his work was ignored by the scientific community. A researcher on peas in Moravia was the wrong person to have solved the problem.
In 1981, Barry Marshall, a pathologist in Perth, Australia, came up with an odd idea: stomach ulcers are caused by the presence of spiral bacteria. His idea was at odds with the thinking of the international scientific community. It was ridiculed by the establishment scientists and doctors. How could bacteria possibly live in the acidic environment of the stomach? In 1984, frustrated by the widespread disrespect for his ideas, Marshall actually drank a Petri dish of liquid containing the spiral bacteria, expecting to develop, perhaps years later, an ulcer. He was surprised when, only three days later, he developed ulcer symptoms. It took until 2005 before he was awarded the Nobel Prize for Medicine. An obscure pathologist from Perth had been the wrong person to have solved the problem.
Time and again, we see that when a bold new idea coming from an unexpected source challenges an entire way of thinking of an international community, it can languish in obscurity for decades, even though the solution to a problem that needs to be solved is staring the experts in the face.
The best-kept secret in management today: Agile
Something similar seems to be happening in management which for decades has struggled to solve a fundamental conundrum: how do you get disciplined execution along with continuous innovation and fully engaged staff? Promising efforts to improve one dimension always seem to cause losses on the other one. Disciplined execution crushes innovation and staff engagement, and innovation is by its nature undisciplined. In the 20th Century, the problem seemed insoluble.
Yet, just over a decade ago, a set of major management breakthroughs occurred. These breakthroughs enabled software development teams to systematically achieve disciplined execution, continuous innovation and full engagement, something that had been impossible to accomplish with traditional management methods. Over the last decade, these management practices, under various labels such as Agile, Scrum, Kanban and Lean, have been field-tested and proven in thousands of organizations around the world. My own recent work distills, builds on and extends these principles, practices and values so that any organization can now achieve to apply the elusive combination of disciplined execution and continuous innovation.
Unfortunately, these management discoveries were not made by “the right people”: academics in business schools or high-paid managers in big corporations. The discoveries were made by the people that you would think, in prospect, are the least likely people to have solved a management problem: geeks. Software developers were known to be antipathetic to both managers and management. Badly dressed, unkempt, even sometimes unwashed, speaking about issues that managers could hardly grasp, these employees were the most problematic of a big organization’s employees. How could they possibly have solved a problem that had stumped the finest management minds on the planet for decades?
In retrospect, it’s not hard to see why they solved the problem. In the 1990s, huge sums of money were being lost because the work of software development was always late, over budget, and plagued by quality problems. Clients were upset, and firms lost money. The developers were seen as culprits and were punished. They worked harder and harder. They labored evenings and weekends. They got divorced. It made no difference. The software was still late, over budget, and full of bugs. They were fired, but their replacements did no better. The standard prescriptions of management didn’t work with software development. Something different had to be found.
Management remains in denial
Yet the management world remains generally in denial about the discoveries of Agile.
You can scan the pages of Harvard Business Review and find scarcely even an oblique reference to the solution that Agile offers to one of the fundamental management problems of our times.
Or take a look at any standard management textbook, say MGMT4 by Chuck Williams (South-Western College, 2011). There is not a single reference to Agile, Scrum or Kanban and only one obscure reference at the very end of the book to Lean.
The only general management book that I have discovered to mention Agile in any significant way is Mark Addleson’s Beyond Management: Taking Charge at Work. Addleson writes,
By looking for practices that are good for knowledge-work, which break the stranglehold of management on how to organize work, we can learn a lot from from a mini-revolution in software development known as ‘agile methods.’… Agile methods offer a way out of this impasse, with a far reaching, even radical solution to building good software.
Beyond Management is one of the very few general management books to recognizes the obvious truth that the solution to an intractable management challenge has come from an unexpected source.
An earlier version of this article was published on Steve Denning’s Forbes column on Radical Management at http://blogs.forbes.com/stevedenning/
Steve Denning is the author of six business books and consultant to organizations around the world on leadership, innovation, management and business narrative. His most recent book is The Leader’s Guide to Radical Management: Reinventing the Workplace for the 21st Century (Jossey-Bass, 2010). Other books include The Leader’s Guide to Storytelling (2nd ed, 2011) and The Secret Language of Leadership (2007). He worked for many years at the World Bank: as the director of knowledge management (1996-2000) he spearheaded the introduction of knowledge management as an organizational strategy. You can follow him on Twitter at @stevedenning. His website is at www.stevedenning.com.