The Current State of Data
Management Survey Results: July 2006
The results of this survey are summarized in the article
the November 2006 of
Dr. Dobb's Journal. The article
also proposes a new vision for data management which you may find interesting.
Here's a few interesting results:
- Of the 1,176 respondents 618 indicated that they were developers, 188 IT
management, and 98 data professionals
- 66% of respondents indicated that development teams sometimes go around
their data management (DM) groups. Of those, 20% found that their DM
group was too difficult to work with, 36% felt the DM group was too slow to
respond, and 19% felt the DM group offered too little value. I believe
that this is a clear indication that a
cultural impedance mismatch exists between developers and data
professionals. See Figure 1.
- 95.7% thought that data was a corporate asset, yet of them only 40.3%
had a database test suite in place to validate the data. Of those
without a test suite, only 31.6% had even discussed the concept.
- 63.7% of respondents indicated that their organizations implemented
mission-critical functionality in the database. Of those, only 46% had
regression tests in place to validate the logic.
- 61.9% indicated that they have production data problems, yet 18% had no
strategy to fix the problems and 33% hoped not to make things work
(effectively no strategy as well).
Figure 1. Reasons why development teams go around data groups.
You may use this data
as you see fit, but may not sell it in whole or in part.
You may publish summaries of the findings, but if you do
so you must reference the survey accordingly (include
the name and the URL to this page). Feel free to
with questions. Better yet, if you publish,
please let me know so I can link to your work.
- I'm not convinced that respondents understood my questions about
testing. The numbers were a lot higher than I suspected, a little
over 40% indicated that they're doing some form of database testing, but in
practice my experience is that data testing is very spotty. The Data
Quality Techniques survey explores this issue and I will report the results
in early January.
- I'm not convinced that respondents understood what I meant about fixing
existing legacy problems via
refactoring. I'd love to think that the technique has been adopted
to the extent that the survey indicates, but I'm just not seeing that in
practice. The Data Quality Techniques survey also explores this issue.
- There was a fair representation of organization sizes. The results
didn't appear to vary substantially based on organization size, with the
exception that larger organizations were more likely to have data groups
than smaller organizations.
- We got a fair representation of respondents. Roughly half (618)
were developers, but 188 identified themselves as IT management, 133 as
project managers, and 98 as data professionals.
- At the beginning of the survey the respondents thought they were getting
adequate service from their data groups (avg rating of 3.6 out of 5, see
Figure 2) but after thinking about the quality issues
explored during the survey rated their data groups lower (avg rating of 3.31
out of 5, see Figure 3). It leads me to believe
that data groups in many organizations may currently benefit from people
having low expectations about what a data group can actually accomplish.
As more and more organizations adopt agile techniques, and thereby start
adopting quality-oriented techniques which provide concrete results, data
groups may find that the bar is slowly being raised on them. They can
and should start adopting agile data
- The data professionals rated themselves slightly higher on average than
any other group, but it wasn't a big difference. The trend was still
the same, even data professionals felt that their data groups weren't doing
as good of a job by the end of the survey.
- This survey suffers from the
fundamental challenges faced by all surveys.
Figure 2. What people initially thought of their data groups.
Figure 3. What people thought of their data groups at the end of the
Links to Other Articles/Surveys
From what I can tell, nobody has ever done a similar survey to this one.
However, there are some interesting resources available:
Why Share This Much Information?
I'm sharing the results, and in particular the source data, of my surveys for
- Other people can do a much better job of analysis than I can. If
they publish online, I am more than happy to include links to their
- Once I've published my column summarizing the data in DDJ, I really
don't have any reason not to share the information.
- I think that it's a good thing to do and I invite others to do the same.