 |
2007 IT Project Success Rates Survey
Results
|
 |
| |
|
|
 |
This survey was performed in early August 2007 and there was
586 respondents.
The survey was announced on the Dr. Dobb's Journal
(DDJ) mailing list.
|
|
The results of the survey is summarized in the article
Defining Success
published in the December 2007 print issue of
Dr. Dobb's Journal.
I also analyzed the data a bit deeper in my November 2007 Agile newsletter
entitled
Is Agile
Really that Successful?
Some findings include:
- Respondents reported that Agile software development projects have a
71.5% success rate, traditional projects a 62.8% success rate, and offshored
software development projects a 42.7% success rate. These figures are
summarized in Figure 1.
- Respondents with only Agile experience, 15 out of 586, indicated a
success rate of 84.3%, respondents with traditional-only experience (164 of
586) reported a success rate of 66.5%. Respondents with both Agile and
traditional experience (336 of 586) indicated success rates of 70.9% for
Agile and 61.1% for traditional.
- Time:
61.3% of respondents believe that delivering when
the system is ready to be shipped is more important than delivering on
schedule.
- Money: 79.6% of respondents believe that providing the best ROI is more important
than delivering under budget.
- Scope: 87.3% of respondents believe that meeting actual needs of stakeholders is more
important than building the
system to specification.
- Quality: 87.3% of respondents believe that delivering high quality is more important than
delivering on time and on budget.
- Staff: 75.8% of respondents believe that having a healthy workplace is more important than
delivering on time and on budget.
- When asked to prioritize the five criteria listed above, quality is the
most important factor followed by scope, time, staff, and money
respectively.
- 40.9% of respondents considered cancelling a troubled project to be a
success.
- 68.6% of respondents had been on a project which they knew was going to
fail right from the very start
Figure 1.
Project
success
rates.
You may use this data
as you see fit, but may not sell it in whole or in part.
You may publish summaries of the findings, but if you do
so you must reference the survey accordingly (include
the name and the URL to this page). Feel free to
contact me
with questions. Better yet, if you publish,
please let me know so I can link to your work.
- Government success rates were little different from commercial success
rates (i.e. for Agile projects, 68.8% for govt. vs. 71.9% for commercial)
- This survey measures success as defined by the respondent, it does
not force a definition of success on them. The success rate was
calculated by summarizing the weighted average of each range (i.e.
90-100% averages to 95%) times the number of respondents.
- These figures vary significantly from those of the
Standish Group’s Chaos Report
which reports a 34% success rate and a 51% “challenged” rate. They
define success as “on time, on budget, meeting the spec”, but that
definition doesn’t seem to hold when we ask people what they actually
value. I’m not convinced that it’s appropriate to force a definition of
success on people, regardless of how easy it would the processing of the
resulting data.
- It is difficult to compare the numbers from the Chaos Report and
this survey. This survey is open, you have complete access to the
original questions and data, yet the Chaos Report is closed to
outsiders. I invite the Standish Group to open source their material.
- Agile projects may be more successful than traditional projects
because the agile teams are comprised of more highly disciplined
developers and because the agile teams are smaller (often due to better
organization to begin with, and sometimes because organizations aren’t
applying agile at scale yet).
- There may be some bias towards traditional development. There
was 164 respondents with traditional-only experience compared to 15 with
Agile-only experience.
- The request that went out indicated that the survey was exploring
success rates, so the success rate figures could be a bit higher as a
result due to selection bias (organizations that are really struggling
may not be reporting).
- There was a slight difference in the success rates of
commercial/private and government organizations. The survey shows that
there is a difference between commercial and government when it comes to
defining success, so perhaps we’re comparing apples and oranges.
- Few business stakeholders responded to the survey (only 18 in total)
so we need to treat the stakeholder figures cautiously.
- When it comes to comparing success rates the Asian numbers should be
suspect as there was only 33 responses. Asians rated offshoring as more
successful than Americans or Europeans, but because a lot of Asian
organizations are the service providers for offshoring projects so they
may consider their work more successful than their actual clients.
- Project managers seem to be a bit out of sync with everyone else.
Project managers are more interested in delivering on time and on
budget over delivering when the system is ready and spending the money
wisely than the others. This may be because of the reward structure in
place within your IT organization, but if that was the case then we’d
see similar numbers for IT management and non-mgmt IT. It may also be a
reflection of the values promoted by the Project Management Institute (PMI).
- People’s attitude towards a healthy workplace is a serious problem.
Although 75.8% said that having a healthy workplace is more important
than delivering on time and on budget only 53.3% of business
stakeholders surveyed said so. I’m happy to say that 80% of IT
managers, and 70.3% of project managers, put the needs of their staff
over being on time and budget, but I’m definitely worried about the
business attitudes towards IT staff. This may be indicative of the
general frustration which business stakeholders often have with IT in
general as well as a common belief that IT services are becoming a
commodity. Until these figures get closer to 100% I think we have a
problem.
- Although 75.8% of respondents wanted a healthy workplace for IT
staff, only 53.3% of business stakeholders wanted that. This is a
serious issue, I suspect it’s partly driven by the fact that many
business stakeholders see IT merely as a necessary evil.
- This survey suffers from the
fundamental challenges faced by all surveys.
Links to Other Articles/Surveys
- My other surveys
- Answering the
"Where's the Proof that Agile Methods Work" Question
-
Geri Winters' blog posting about this survey
Why Share This Much Information?
I'm sharing the results, and in particular the source data, of my surveys for
several reasons:
- Other people can do a much better job of analysis than I can. If
they publish online, I am more than happy to include links to their
articles/papers.
- Once I've published my column summarizing the data in DDJ, I really
don't have any reason not to share the information.
- Too many traditionalists out there like to use the "where's
the proof" question as an excuse not to adopt agile techniques. By
providing some evidence that a wide range of organizations seem to be
adopting these techniques maybe we can get them to rethink things a bit.
- I think that it's a good thing to do and I invite others to do the same.
Suggested Reading
 |
This is an eye-opening book for anyone who is trying
to understand how to measure concepts, such as developer productivity
levels for example, which are often perceived as difficult to measure.
If you choose to think outside of the metrics box for a bit you'll
quickly realize that you can easily measure information which is
critical to your decision making process. |
|
|
This is the latest edition of Capers Jones' classic
book summarizing the wealth of data which he has collected over the
years pertaining to software development projects and IT in general.
Although you need to read between the lines a bit, most of his data is
from traditional waterfall projects so it can be difficult to apply it
in the context of iterative or
agile projects, I still find this book to be a
valuable resource. If you're interested in improving the way that
you work as an IT professional, this book is an important resource for
making fact-based decisions. |
