The Chaos Reports are one the most citated reports used in Agile presentations and courses. The most famous piece of data is the chart that shows that, in the projects studied, very few features developed were actually used. Minimizing the amount of unused features is core to Agile projects. But there is something odd about the Chaos Reports. The data they contain might be valid (or not), but the targets that they measure have one serious flaw, they measure the wrong objectives. Even more troublesome is the fact that most customers I deal with measure the same objectives: conformance to budget and deadlines. I call for suppliers to urge their customers to start measuring up and collaborate on achieving what really matters to their customers.
The problem lies not only in measuring the wrong objectives, keeping an eye on the budget is not wrong in itself, it’s using quantified objectives to measure success that has some big issues. Reading Demings arguments against management by objectives (MBO) makes me think twice before using cost or deadline to measure a projects’ success. Deming: “It is human nature to assume that there is something better about random events that meet arbitrary objectives, and assign their superiority to a non-existent special cause”. This is the heart of the matter: when a customer sets a deadline and his supplier meets this deadline, it’s human nature to assume the supplier did something special to meet the deadline. But there is no cause-and-effect here, meeting a deadline does not make you a good or bad supplier, it doesn’t mean anything.
Most proponents of Agile still use these same old arguments, arguing Agile will bring your project in on time and on budget. But this is only a superficial answer to the customers (real) objectives. Because variation of scope is used to attain these goals, you are actually doing exactly what Deming warned us about: cheating the system to meet the objectives. Why is this cheating? Because you’re still making the numeric objectives the goal of the project. What about the consequences of this behaviour? Are projects that are on time and within budget better than projects that are not. Perhaps the projects overshooting their deadline actually have a higher ROI. More likely there is no relation between ROI and staying within budget.
One way to get out of this objective trap, is to measure up. Don’t measure progress towards the deadline or the budget but focus on things like customer satisfaction, increase in sales or actual reduction in cost or process cycle time. It will free up your project managers to lead teams to do the right thing and include the complexity of attaining actual ROI into the project. Scrum originated from this holistic approach (pdf), but project management hijacked the burn down chart and inserted the classic objectives again.
One note of warning, I’ve never seen this happen myself. I do know of companies with good track records in this regard, but they are not stricly IT companies. Do you have the courage to add business value to your burn down chart and measuring up?