Defining and measuring the success of an Agile methodology

<< It is not possible to estimate what? | Home | Monitoring and supporting the emergence of team dynamics toward positive directions >>

Defining and measuring the success of an Agile methodology

Alistar Cockburn describes in this post the success of a methodology with the formula:

Methodology Success = Project delivered + Staff would do it again

In software projects people is a fundamental variable, so here with methodology is not intended say RUP or FDD or say XP or Scrum or Kanban. Instead is intended that

A methodology is the conventions the team agrees to and actually follow

When the team achieves a methodology success as defined above, it is possible to infer that the team and also the method they used, RUP or FDD or one of the others, helped the project success.
In case of failure, it is hard to say if it was because the method was not helpful. Indeed there are many factors that affect the project. These factors include the team together with the tools and technology used and the available resources. Also the context matter, the environment within the team operates (i.e. the company culture, structure, rules, ...). The project itself could be impossible to achieve.

So from a failure is not possible to infer information on the method itself. Instead

when there is a statistically relevant number of teams successes with a specific method

we can infer that the method, say RUP or FDD or say XP or Scrum or Kanban, helps project success.

Let me expand on the meaning of Project delivered + Staff would do it again based on my personal subjective experience and on reflections I made reading this book [1]:

It means that team accomplished the project goal and fulfilled team members needs

Is useful to note that the project goal includes fulfilling the needs of the users, the stakeholders and of the core group of the company [2].
Is also useful to take into account that a project has multiple outcomes that should be considered too:

the project could have

increased the team-work effectiveness
evolved the tools and technology used for accomplishing projects
elaborated useful information and taught new things and generated new knowledge
resolved conflicts influenced or attained consensus between teams and departments within the company

or the opposite, the project could have

created more technical debt
increased the distrust of the customers
increased conflict and misunderstanding between company departments

Is useful to note when considering 'Staff would do it again' that this evaluation to be meaningful requires that the Staff does not show evident dysfunctional dynamics because that would affect his judgment. Extreme examples from real life are people addicted that don't want to change an harmful habit or people that intentionally ends functioning beneficial relationship.

A final and interesting note, the comment from Alistar in in the final comment:

Really astonishing how few projects pass those two tests

About the first test, project delivered, this data is available [3][4]. And about the state of Agile development this data is available [5].

[1] Small groups as complex systems; H. Arrow, J.E. McGrath J.L Berdahl; Sage Publications; 2000
[2] Core group theory, Art Kleiner
[3] http://www.ambysoft.com/surveys/success2011.html
[4] http://blog.mountaingoatsoftware.com/agile-succeeds-three-times-more-often-than-waterfall
[5] http://www.versionone.com/state_of_agile_development_survey/11/