Agile Database Techniques
Effective Strategies for the Agile Software Developer
John Wiley & Sons ISBN#: 0-471-20283-5
Part 1 describes the fundamental skills and knowledge that everyone on an agile project team should have. This includes the basics of object orientation, relational databases, the object-relational impedance mismatch, data modeling, and how to deal with legacy data issues. Without this common base of knowledge it is very difficult for application developers and data professionals to work together effectively. A significant problem in the IT industry is that most data books do not cover object-oriented development issues and most object books seem to ignore data issues. This needs to stop.
Part 2 focuses on how to take an evolutionary approach to data. This section sets the foundation for a model-driven development (MDD) approach, or more accurately an agile model-driven development (AMDD) approach where your application code and database schemas are based on agile models. This isn't the only way to work, you may decide to take a test-driven development (TDD) approach instead or better yet combine it with AMDD. Both methods support evolutionary development but because MDD is very common within the data community and I suspect that they will gravitate more towards an AMDD approach rather than a TDD approach. However, some agile developers, particularly extreme programmers, prefer TDD over AMDD. Luckily the two approaches work very well together so it really doesn’t matter. The implication is that TDD will become more important to data professionals in the coming years. This section also describes database refactoring, an evolutionary technique that enables you to improve your database design in small steps. In many ways database refactoring is normalization after the fact. Chapters describing mapping objects to relational databases, performance tuning, database encapsulation, and supporting tools are included in this part because they enable evolutionary development
Part 3 focuses on implementation techniques and strategies such as concurrency control, security access control, finding objects in relational databases, referential integrity, and the effective use of XML. An important observation is that many of these topics are traditionally thought of as data issues, but as you'll see there is far more to them than this – it isn't a black and white world.
Part 4 describes strategies for adopting agile database techniques. These chapter provides advice for individuals who want to become agile software developers and for organizations that want to adopt agile techniques.
Chapter 1: Data Can Be Agile. Explores how application developers, database administrators (DBAs), enterprise architects, and data administrators can work together effectively in an agile environment.
Chapter 2: From Uses Cases to Databases – Real-World UML. Object technology is the norm for modern projects, therefore it is critical for everyone to understand the basics of object orientation and the Unified Modeling Language (UML) 2.x (including UML data modeling).
Chapter 3: Data Modeling 101. Data modeling is a fundamental skill that all software professionals, including object professionals, require if they wish to store data effectively. UML data modeling is covered in detail.
Chapter 4: Data Normalization. Normalization is a collection of design strategies that ensure data is stored in one place and one place only, promoting the design of highly cohesive and loosely coupled data schemas.
Chapter 5: Class Normalization. The concepts of normalization can be applied to object schemas, a complementary technique to design patterns and programming idioms.
Chapter 6: Relational Database Technology, Like it or Not. Relational databases (RDBs) have been the dominant technology for persisting business objects and will likely remain so, therefore you need to understand technology.
Chapter 7: Overcoming The Object-Relational Impedance Mismatch. Object technology and relational technology are based on different paradigms, presenting us with a technical impedance mismatch that we must overcome. Worse yet is the cultural mismatch between object professionals and data professionals that must also be bridged.
Chapter 8: Legacy Databases – Everything You Need to Know But Are Afraid to Deal With. Virtually every project team finds that it needs to work with legacy data sources, and when doing so quickly discover serious data quality, database design, and data architecture problems that they need to overcome.
Chapter 9: Vive La Evolution. This chapter argues that agile software development is real and here to stay, that data is an important aspect of most systems, and that all agile methodologies take an evolutionary approach to development. Therefore if data professionals wish to remain relevant they must embrace evolutionary development.
Chapter 10: Agile Model Driven Development (AMDD). Agile Modeling (AM) is a chaordic collection of practices for the effective development of models and documents. AM defines a streamlined approach to evolutionary modeling for agile developers.
Chapter 11: Test-Driven Development (TDD). Test-driven development (TDD) is a development approach where developers add a test before they add new functional code. This minimizes the feedback loop while providing developers with the confidence to proceed in small, evolutionary steps.
Chapter 12: Database Refactoring. A database refactoring is a small change to a database schema that improves its design. The process of database refactoring enables you to evolve your data schema in step with the evolution of the systems that access that data.
Chapter 13: Database Encapsulation Strategies. Encapsulating access to data sources enables developers to evolve both the data schema as well as their application schemas independently of one another.
Chapter 14: Mapping Objects to Relational Databases. When working with object and relational technologies you must map your object schema to your database schema, evolving it over time as your two schemas evolve.
Chapter 15: Performance Tuning. The need to ensure sufficient system performance is often a primary motivator of evolutionary changes late in the lifecycle for traditional projects. On agile projects it motivates changes almost from the beginning.
Chapter 16: Tools for Evolutionary Database Development. An overview of testing tools, ...
Chapter 17: Implementing Concurrency Control. Concurrency control deals with the issues involved with allowing multiple people simultaneous access to shared entities, be they objects, data records, or some other representation.
Chapter 18: Finding Objects in Relational Databases. A common programming task is retrieve one or more objects from a database, perhaps to display a list of people that work in a department, to list available inventory items that meet user-defined search criteria, or simple to implement a report.
Chapter 19: Implementing Referential Integrity and Shared Business Logic. When one entity references another then that other entity should exist. Referential integrity is an issue when a row references another row, an object references another object, or an object represents a data within a database (or vice versa).
Chapter 20: Implementing Security Access Control. Your system must ensure that authenticated users access only what they are authorized to. The complexities of modern development require developers to understand a wide range of options to do exactly this.
Chapter 21: Implementing Reports. There are several common strategies to support reporting functionality within your system and/or to integrate with a common reporting application that crosses a variety of systems.
Chapter 22: Advanced XML? No, Just Realistic XML. XML is a robust and growing set of technologies. Unfortunately many people seem to have forgotten the data community’s hard-earned lessons and seem to think that XML is the panacea that will solve all of their problems. This is hardly the case. XML has a very bright future but level setting is required.
Chapter 23: How to Become Agile. As an individual you need to decide to become more agile in the way that you work. First you need to adopt “the agile attitude” and then you need to actively work towards gaining the skills to make you a valued member on an agile project team.
Chapter 24: Bringing Agility Into Your Organization. People, and therefore organizations, resist change. To adopt agile database techniques you need to overcome this resistance to change.
Since the early 1990s I've been working with both object and relational database (RDB) technologies to build business applications, and since the mid-1990s I've done a fair bit of writing on the subject. These writings have appeared in Software Development, in several of my books (in particular Building Object Applications That Work and The Object Primer), and on my personal web site. The two white papers at my site, one on mapping objects to RDBs and the other describing the design of a persistence layer, have proven to be incredibly popular with several hundred thousand downloads over the years. The persistence layer paper has even been used as the basis for several open source products. Although it's been very rewarding for me to share my ideas through these writings I never took the time to collect this work in one place nor have I written everything that I have to say about the topic. This book rectifies this situation.
As a consultant I've worked with object and data professionals, their related technologies, and of course their techniques. In doing so I've worked in traditional environments that take a near-serial approach to development as well as more modern environments that take an agile and evolutionary approach to development. Over time I've worked on many different project teams in various roles. Data oriented issues where important, and sometimes even critical to success, on each project. Although traditional project teams seemed to have a handle on how to deal with data issues the more agile ones often struggled – in part because the data professionals in those organizations preferred to take a serial approach and in part because the object developers didn't appreciate the importance of data-oriented issues. Being an ex-data specialist (oh no, my horrible secret is out!) and being experienced in object technology I often found ways for the two groups to work together. My experience was that data professionals were often overly focused on data to the exclusion of the wide variety of challenges faced by object developers and similarly object developers had little or no data-related experience. So I would help the two groups find ways to work together, to mentor them in each other's techniques, and to help them overcome what is known as the object-relational impedance mismatch. For these two groups to work together effectively they need to understand and appreciate what the other group is focused on, and I would even call into question the wisdom of having separate groups to begin with. This book describes the skills that both data professionals and object professionals require in order to build modern-day software.
As a methodologist I have actively tried to find ways to develop software effectively, and over the years have run the gambit from prescriptive approaches such as my work with process patterns and the Enterprise Unified ProcessTM (EUP) to agile approaches such as Agile Modeling (AM) and now agile database techniques. In part this book is an extension of AM to help describe how data professionals can take an evolutionary (iterative and incremental) approach to development. Although many people within the data community are adamantly opposed to evolutionary approaches, interestingly enough I've often found that those opposed to it have never actually tried it, the reality is that agile software development is real and it's here to stay. For data professionals to remain relevant they must be prepared to work in an agile manner, otherwise project teams will very likely find ways to work around them (I suspect you see this sort of thing happen within your organization all of the time). My experience, on actual projects, is that you can in fact be very successful taking an agile approach to data-oriented activities if you choose to do so. Many people will tell you that it won't work, but all they're really saying is that they either can't make it work or they don't want to. This book describes numerous, proven techniques that support evolutionary data-oriented development.
When I first started writing this book I intended its focus to be on the Agile Data (AD) method. This method describes how data professionals and application developers can work together effectively on agile projects. It also describes how enterprise professionals, such as enterprise architects and data administrators, can support agile development teams effectively. Because I was taking an iterative and incremental approach to the development of the book I quickly realized that the real value lay in detailed development techniques instead of yet another methodology. So I refocused.
You! Anyone who is interested in improving their productivity as a developer, or who wishes to bring effective database development practices into their team or organization.
We actively work with clients around the world to improve their information technology (IT) practices, typically in the role of mentor/coach, team lead, or trainer. A full description of what we do, and how to contact us, can be found at Scott W. Ambler + Associates.