Surveying the Agile
Are Agile Surveys Valuable?
Assuming that the survey is well designed, the results of an agile survey can
be very valuable for people who are trying to make informed decisions around
whether to adopt agile approaches, or to further expand their adoption efforts.
Many people want to see industry data, and surveys are one way to get that.
Of course surveys aren't the only source of information, actual experience with
agile techniques also provides important insight, as does academic research and
forms of anecdotal evidence such as case studies, experience reports, and
All of these sources of information have their place and each has their
advantages and disadvantages. One size does not fit all.
A common reason that people give for not filling out surveys is that they
don't feel that the information is valuable to them. I have to assume that
they're correct in their belief that they're not getting value out of surveys.
However, that doesn't mean that others don't see value in survey results.
Furthermore, by not filling out a survey, other than saving time of course, all
you will accomplish is that you're making it harder for your voice to be heard by
senior decision makers within IT departments (few decision makers are trolling agile
mailing lists trying to sniff out the occasional word of wisdom). And yes,
despite all the talk about self-organizing teams within the agile community, the
fact still remains that senior management in your organization can and will make
decisions which affect what you do and how you do it. As a community it
behooves us to invest time to provide the best information that we possibly can
to decision makers, and effective surveys are part of that strategy. We
need to remember that there is a wealth of information available to decision
makers showing that traditional strategies are effective in practice, a very
good source for example is Capers Jone's
Applied Software Measurement 3rd Edition, and we need to motivate senior
management to start questioning some of the advice that they're getting.
Designing An Effective Survey
Here are some quick thoughts based on my experiences over the years.
- Know the topic. If you don't understand the topic that
you're exploring, there's very little chance that you'll design an effective
survey which explores the topic. Do some reading on the topic first
and understand what
already been run regarding agile software development. Get involved
with the community, and identify what issues actually need to be explored.
- Let people opt out of questions. I typically make questions
mandatory but will allow people to indicate that they don't know what the
answer is, or that the question isn't applicable to their situation, or
simply give an option of "other". If you don't allow people some way
to opt out of a question then you run the risk that they will give the
closest answer or simply choose any answer simple to move to the next
question, thereby reducing the quality of your data.
- Prefer to ask about observable facts over opinions. Ask questions that focus on observable facts that the
respondent could realistically answer. For example, instead of asking whether a team is large
ask if the number of members was between 1-10, 11-20, and so on. Of course you will often
still want to explore people's opinions in some questions, but always step back
and ask yourself if it would be better to instead explore facts.
- Keep it short. People are very busy and don't have the time
to fill out long surveys. The longer the survey, the lower the chance
that people will fill it out and therefore the lower the applicability of
your findings because you'll have a small sample size. I realize that
this is hard because you likely are interested in a lot of issues, but it's
far better to explore a few targeted issues well in most cases.
- Explore new issues. It doesn't make a lot of sense to cover
the same ground that's already been covered by others, unless your goal is
to confirm their work (this can be important too). Instead, either try
to extend our knowledge by exploring an issue in detail (for example, the
DDJ 2008 Agile Adoption survey found that the majority of agile teams
were doing some up front requirements and architecture envisioning, the
DDJ 2008 Modeling and Documentation survey explored how people were
going about doing so). Or, repeat an existing survey for a targeted
group. For example, when I present results of various surveys at
conferences (I give a presentation called
Agile by the
Numbers which I've given at conferences and to customers around the
world) I'm often asked for the detailed numbers for a specific geographic
region, such as Scandinavia or South Africa, or for a specific domain, such
as banking or manufacturing. If you have access to a mailing list for
a targeted group of people then it would be interesting to discover whether
they exhibit different trends than the large community does.
- Get help. Get some help designing a survey from people with
agile experience and with survey experience (get feedback from the
Agile Survey Reviewers).
- Beta test it. Send it out to a small group of people that
you know, hopefully one which is a reasonable representation of the group
that you're targeting, to determine if they understand the questions that
you're asking. There's nothing worse than finding out that you
miswrote a question. For example, the
DDJ 2006 Agile Adoption Survey asked about whether people were doing
Feature Driven Development (FDD) and many people responded that they
did. But, very few people seem to do FDD in practice, even though it's
a very effective approach, but respondents indicated that they did because
they were capturing their requirements in the form of feature statements
(which is fine, but that doesn't mean you're doing FDD). The problem
was that few people understood what was actually being asked and
misinterpreted it to mean something else. If I had beta tested the
survey first I likely would have noticed this abnormal result and hopefully
addressed the problem appropriately.
- Invest some time to learn about survey design. Read some of
the resources suggested at the bottom of
My advice is to:
- Make the source questions available. People should see what
questions were asked, how they were asked, and in what order they were
asked. This puts the results into context and enables people to
identify any biases that you may have introduced through your wording.
For all of the surveys that I
run I make a PDF of the survey available online.
- Make your analysis available. It should be as easy as
possible for people to learn about the important findings, or at least what
you think is important, of your survey. For all the
surveys that I run I make a
PowerPoint presentation file available that people can reuse in their own
presentations, with proper attribution, and often include graphic images of
some results which I share on my site (usually I'm using the graphics in an
article somewhere online).
- Make the source data available. This enables people to
analyze the data for themselves, they don't have to trust your analysis
(which will also potentially introduce bias). For all the
surveys that I run I make a
CSV file of all the source data, with the exception of identifying
information (due to privacy concerns), available online. Many
surveyors will not make their survey data available because they see it as a
competitive resource that they shouldn't share. My philosophy is that
once I've used the data for my own purposes, usually to write an article
about what's going on within the IT community, then I had might as well
share it with others and hopefully enable them to gain some value too.
Personally, I don't trust any survey results when the surveyor doesn't do all
three of these things.
I would be remiss if I didn't discuss some of the known challenges with
surveys. In addition to design and
publishing-related difficulties, there are also a few
inherent challenges with surveys which can be difficult to overcome:
- You will only get responses from people willing to be surveyed.
The opinions of people not willing to be surveyed are important too.
;-) Bottom line is that this is one aspect of selection bias.
- You risk getting responses from people with strong feelings about the
topic. Even the title of a survey can contribute to this problem
This problem is one of the reasons why I now run "State of the IT Union
Surveys" -- this is a fairly generic title that doesn't reveal what the
specific topic is. These new surveys also address several topics, not
a single theme, so as to reduce the respondent drop out rate.
- Very often questions capture opinions, not facts. This is
perfectly fine as long as the results are presented as opinion (which can be
difficult to do sometimes). For example, the
2009 Agile Practices Survey explored how people are adopting agile
practices. It's fair to indicate that certain practices are believed
to be more effective than others but it wouldn't be fair to state that some
practices are more effective than others (this is something better left to
more specific research. However, recognize that it is possible to ask
factual questions such as the length of time that they've been working in
IT, their age, and so on (yes, they may still choose to misrepresent
- The biases of the communities will be reflected in the results.
People form communities for a reason. For example, people join the
mailing list because they're interested in TDD and probably even trying to
learn TDD. My
Test Driven Development Survey was sent out to that list because I
wanted to explore what they were actually doing in practice. Because
this community is biased towards TDD they wouldn't be a good source of
information about TDD adoption rates but they would be a potentially good
source of information for how people are actually doing TDD in practice.
This is why I indicate who the surveys went out to, so that you can
determine what selection bias may have been introduced due to who the survey
was sent out to.
- It's circumstantial evidence.
There are other alternatives for gathering data, such as ethnographic research,
that can provide a higher quality level in the results. These appoaches are
expensive and time consuming and are usually performed by university researchers
instead of industry practitioners such as myself. Having said that, there is
clearly room for the type of research that myself and others are performing
via surveys. Yes, it would be wonderful to have incredible amounts of
empirical evidence about the efficacy of various software development strategies.
Yes, it would be wonderful to have better information that what I'm able to provide.
However, the information that is provided by this work proves to be sufficient,
or dare I say
just barely good enough (JBGE) to answer many important questions that people have.
Perfection is the enemy of good enough.
It's really easy to run a survey using online tools such as
Survey Monkey so a
lot of people do so. This wouldn't be such a bad thing if the surveys
provided value, were designed well,
and the results were properly published.
However, this often isn't the case and as a result fewer people choose to fill
out online surveys because they feel that their time is being wasted (and sadly
it often is).
Common anti-patterns with agile surveys:
- Misguided students. A common
problem with "agile surveys" occurs when university or college students are
given an assignment to do some research pertaining to agile development because they
often put together a survey which covers topics which others before them have previously
surveyed or they explore issues which reflect traditional (not agile) strategies to development.
The students have the best of intentions but due to lack of experience, and
often lack of support from their already overworked professors, they execute the
survey poorly. These surveys have almost no hope of finding out pertinent information and are
inadvertently making it harder for everyone else because they're annoying the
people they're hoping to survey and reducing the likelihood that they're respond
to future surveys. My advice to the students is to see some help
designing your survey, both from your professors
and teaching assistants as well as from the agile community.
- Thinly disguised marketing. Every so often a survey is sent
out which is nothing more than a marketing gimmick for a consultant or
product vendor. My advice is to recognize that you're not fooling
anyone and worse yet are running the risk that all you're going to
accomplish is that you'll turn people off to whatever it is that you're
trying to sell.
Better Approaches Than Surveys
Yes, surveys clearly aren't ideal. For example, ethnographic research
where the researcher(s) spend months and sometimes years directly observing people is clearly more effective.
And more expensive and time consuming. Many researches will start with a survey to help them to identify
potential candidates to talk to and then interview them to obtain more details as to what they're actually
doing in practice. Also more expensive and time consuming (and fraught with opportunity for the bias
of the researcher to creep in unbeknownst).
Every so often I run into someone who has a negative opinion about surveys. OK, everyone is
entitled to their opinions and as I indicated earlier surveys clearly aren't perfect. Worse yet are
poorly designed surveys, and there are a lot of those out there. Fair enough. But, I find that there
is a significant difference in the quality of the conversation when the person is a researcher who
has real experience trying to actually do research in the IT space versus someone who has never
done so. The experienced researchers definitely hope for better but at the same time are often impressed
how I was able to get the data that I did get (and yes, they often leverage that data to guide their
own work). The inexperienced people tend to have unrealistic expectations, wanting better quality results
(fair enough) without realizing what it actually takes to get those results. And naturally they
rarely have done any research themselves, nor are they willing to do so (can't say I blame them),
are not able or willing to allow researchers to come into their teams to explore what's happening,
and rarely have any viable suggestions for doing better (and sometimes even complain that
I didn't look into topic X, sigh). Interestingly, when asked where they can find better quality
information around whatever topic my survey explored they often admit that my stuff is amongst
the best they found. It's about this point in the conversation I ask what type of cheese they
would prefer to have with their wine.
Survey-based research isn't perfect. That's why I'm as careful as I can be, I reach out for help
when designing a survey, and am 100% open about the design and results. As I described earlier, I
always share the questions exactly as asked, described how I got my sample set, and share the data
exactly as answered. This is 100% open because I have nothing to hide. Personally, I am skeptical
about any research where they don't work in a 100% open manner. Since I started this in 2006 I'm
glad to say that I'm seeing a growing trend in the IT community towards this sort of approach.
Of course there are much worse sources of information than surveys results. Anecdotal evidence, for example "My
experience is that 65% of developers eat junk food every day for lunch" is an interesting observation
at best. Rumours, for example "I heard an expert say that three quarters of
programmers eat junk food every meal", are even worse. The IT industry is replete with oft-repeated
rumours that seem to have little basis in fact, but it's something that "everybody knows."