Current State of Data Management: 2006 Open Research

This open research into the current state of data management was performed during the first three weeks of July 2006 and there was 1,176 respondents. The survey was sent out to the Dr. Dobb’s Journal mailing list.

The Survey Results

The results of this survey are summarized in the article Whence Data Management? in the November 2006 of Dr. Dobb’s Journal. The article also proposes a new vision for data management which you may find interesting.

Here’s a few interesting results:

Of the 1,176 respondents 618 indicated that they were developers, 188 IT management, and 98 data professionals
66% of respondents indicated that development teams sometimes go around their data management (DM) groups. Of those, 20% found that their DM group was too difficult to work with, 36% felt the DM group was too slow to respond, and 19% felt the DM group offered too little value. I believe that this is a clear indication that a cultural impedance mismatch exists between developers and data professionals. See Figure 1.
95.7% thought that data was a corporate asset, yet of them only 40.3% had a database test suite in place to validate the data. Of those without a test suite, only 31.6% had even discussed the concept.
63.7% of respondents indicated that their organizations implemented mission-critical functionality in the database. Of those, only 46% had regression tests in place to validate the logic.
61.9% indicated that they have production data problems, yet 18% had no strategy to fix the problems and 33% hoped not to make things work (effectively no strategy as well).

Figure 1. Reasons why development teams go around data groups.

Downloads

The Survey Questions(162 K)

Raw Data(162K)

Summary Presentation(155K)

What You May Do With This Information

You may use this data as you see fit, but may not sell it in whole or in part. You may publish summaries of the findings, but if you do so you must reference the survey accordingly (include the name and the URL to this page). Feel free to contact me with questions. Better yet, if you publish, please let me know so I can link to your work.

Discussion of the Results

I’m not convinced that respondents understood my questions about database testing. The numbers were a lot higher than I suspected, a little over 40% indicated that they’re doing some form of database testing, but in practice my experience is that data testing is very spotty. The Data Quality Techniques survey explores this issue and I will report the results in early January.
I’m not convinced that respondents understood what I meant about fixing existing legacy problems via database refactoring. I’d love to think that the technique has been adopted to the extent that the survey indicates, but I’m just not seeing that in practice. The Data Quality Techniques survey also explores this issue.
There was a fair representation of organization sizes. The results didn’t appear to vary substantially based on organization size, with the exception that larger organizations were more likely to have data groups than smaller organizations.
We got a fair representation of respondents. Roughly half (618) were developers, but 188 identified themselves as IT management, 133 as project managers, and 98 as data professionals.
At the beginning of the survey the respondents thought they were getting adequate service from their data groups (avg rating of 3.6 out of 5, see Figure 2) but after thinking about the quality issues explored during the survey rated their data groups lower (avg rating of 3.31 out of 5, see Figure 3). It leads me to believe that data groups in many organizations may currently benefit from people having low expectations about what a data group can actually accomplish. As more and more organizations adopt agile techniques, and thereby start adopting quality-oriented techniques which provide concrete results, data groups may find that the bar is slowly being raised on them. They can and should start adopting agile data techniques.
The data professionals rated themselves slightly higher on average than any other group, but it wasn’t a big difference. The trend was still the same, even data professionals felt that their data groups weren’t doing as good of a job by the end of the survey.
This survey suffers from the fundamental challenges faced by all surveys.

Figure 2. What people initially thought of their data groups.

Figure 3. What people thought of their data groups at the end of the survey.

Why Share This Much Information?

I’m sharing the results, and in particular the source data, of my surveys for several reasons:

Other people can do a much better job of analysis than I can. If they publish online, I am more than happy to include links to their articles/papers.
Once I’ve published my column summarizing the data in DDJ, I really don’t have any reason not to share the information.
I think that it’s a good thing to do and I invite others to do the same.