Software Process Improvement (SPI) Best Practices

	Software Process Improvement (SPI) Best Practices

Home \| Articles \| Agility@Scale Blog \| Books \| IT Surveys \| Podcasts \| Contact Me \| Mailing List \| Site Map

Over the years I've helped a wide range of organizations, within a wide range of industries, improve their software processes. I've generalized and shared many of my experiences online at the Agile Modeling, Agile Data, Enterprise Unified Process^TM, and this site. I've also discovered a collection of "best practices", first published in More Process Patterns but more recently in The Enterprise Unified Process, which seem to work in most situations:

Be aware of your organization’s current culture. One of the significant forces that affect the success of your process improvement efforts is the culture of your organization. Organizations with cultures that are positive toward process improvement are likely to want to supply a quality product with reasonable business returns, have middle managers that are willing to set and work toward targets of meeting your organization’s needs and business goals, and have senior management leadership that is willing to launch and sustain a long-term change effort.
Expect to change your organization’s structure and culture. Process, organizational structure, and corporate culture all go hand-in-hand – change one and you will affect the others.
Keep it simple. A common mistake that organizations make is to over specify the processes that they intend to follow. Never forget that your goal is to produce working software that meets the needs of your user community and that your staff likely has a pretty good idea of how to do this although they could help with a little guidance from time to time. Give them just enough guidance for their needs. An example of a well defined, yet simply defined, process is the Agile Unified Process (AUP).
Align your software process with business goals and objectives. You could have the best process in the world, but if it doesn’t meet your organization’s goals then it doesn’t matter. Do you intend to build a portfolio of applications that integrate with, and build upon, one another? If so then portfolio management is important. Do you intend to sell shrink-wrapped software to be used by millions of users? If so then architecture may be less important to you in favor of getting a product to market quickly.
Regularly hold retrospectives. A retrospective is a process improvement meeting where you ask four fundamental questions: What did we do well that if we don't discuss we may forget? What did we learn? What should we do differently next time? What still puzzles us? Retrospectives can be simple 15 minute discussions or formal meetings over a day or two. The goal is to learn from your experiences.
There's a serious difference between "lessons learned" and "lessons indicated". The result of a retrospective is a collection of lessons indicated, they don't become "learned" until you actually improve your process. I've been in organizations where we've held a retrospective, sometimes called a post mortem or "lessons learned" session, which resulted in a list of really good process improvement suggestions. Always one to cause trouble, I then asked if the client had any similar documents from four or five years ago. Comparing the documents, we've often discovered significant overlap between the lists.
Keep the real goal in mind. My experience has been that software processes, when applied intelligently, increase the productivity of developers. My experience has also been that when processes are applied less than intelligently, or when the paper pushers have too much influence within an organization, processes can also decrease your productivity. Organizations that keep the end goal in mind – that of developing, maintaining, and supporting software that fulfills the needs of their user community – will be successful with implementing software processes. Those that follow processes simply for the sake of doing so are likely to fail.
Recognize that the fundamentals remain the same, the details vary. Contrary to popular belief, the fundamentals of software development have been known for many years. You need to perform requirements engineering. You need to model. You need to write code. You need to test. You need to perform change control. You get the picture. Every successful software organization will have a similar set of processes but the way that your organization brings them in and how they implement them will differ. Your requirements process may be slightly different than your competitors, but you will both have one that will generally do the same sort of thing.
You need more than one process. You wouldn't run a project team of thirty people the same way that you would a team of three people. Nor would you run an outsourcing project the same way that you're run a project developed by a co-located team. Nor would you run a data warehouse project the same way that you'd run a .NET application. You need different processes, or at least different flavors of your process, for different situations. Use the right process for the job.
Run a trailblazer project to validate your new processes. Regardless of how well you define a process, no process is perfect. Test your new software process using a trailblazer/pilit project, one that is given the extra resources required to try new techniques and to update them appropriately.
Treat process improvement like a project. Have an experienced project manager, ideally someone with experience in both process-oriented and object-oriented development. Define the requirements for your processes, model them, implement them, test them with a trailblazer project, and then improve the processes.
Improve your processes in priority order. The reality of process improvement is that you cannot make all of the changes that you want to immediately; it is simply too great a change for your organization to absorb at once. This is why we have efforts such as the Software Engineering Institute’s (SEI’s) Capability Maturity Model Integrated (CMMI) efforts and the Software Process Improvement Capability Determination (SPICE) efforts of the International Standards Organization (ISO). Both of these organizations suggest that you prioritize the process improvements that your organization needs to make, expect that it will take several years to make the needed changes, and expect that you will experience difficulties while doing so. There are five maturity levels in the CMMI for a reason: you need to progress in order from level one to level two to level three and so on. The implication is that by knowing which aspects of a software process map to which CMM maturity levels you have a rough idea of the order in which you should introduce those processes to your staff. Experience shows that organizations that try to make immediate, large-scale process changes are likely to fail doing so. The reality is that it takes time, often several years, to permanently improve the productivity of your software development efforts. There is not a simple, quick fix to your problems.
Communicate your plan. It is essential that you let others know what are you doing, why are you doing it, the business case for it, success stories, etc. Keeping them informed on how things are going, even if you are encountering difficulties, will help get them (and keep them) on board. There are various options for this: a newsletter, a web site, periodic emails to key personnel, etc. Trumpet your successes and share your lessons learned with the appropriate people.
Accept that the big picture is overwhelming. Because of the complex nature of software development most developers specialize in one aspect of it and focus solely on that. This becomes a problem for organizations that wish to tailor a software process for their exact needs because when they put the individual pieces together the overall process becomes very large. For example, the Rational Unified Process (RUP) is over 3,000 HTML pages in size, and it is only a development process. Once developers see how large your organization’s software process is they often go into denial, claiming that you can not possibly achieve your goals. Yet when you ask an individual which part of their process they can simplify they’ll often balk at the idea. An alternative approach might be something such as the Agile Unified Process (AUP), which is a bit smaller (it's around 30 HTML pages).
Democracies do not always work, nor do dictatorships. Organizations that wish to reach consensus regarding their software process tend to flounder. You’ll never achieve complete agreement on how things should be done, although organizations that dictate processes from above tend to fail as well. Effective process improvement efforts seek consensus at some points and dictate things at other points.
Identify the consumers and suppliers for each process. Every process has inputs and outputs, and you need to ensure that there is a supplier for each input and a consumer for each output. Fundamentally, if nobody is going to use an artifact that is produced by a given process then why bother producing it? You also need to look at collections of processes to see if the artifacts that they produce add value in combination.
Defining a process is the easy part. Many organizations are very successful at defining a software process, often producing binders (or web pages) of documentation. However, when I've revisited these organizations a few months after introducing a new process it will be all but forgotten. Getting people to accept your new process, and making the changes that go along with it, will take significant time and effort to accomplish. Writing a process is the easy part, following it is the hard part.
Introduce a Software Engineering Process Group (SEPG) to your organization. The sole responsibility of your SEPG is to support the definition and improvement of your organization’s software process. The SEPG should be kept small – as a rule of thumb, we suggest one SEPG member for every one hundred developers in your organization. SEPG efforts are a component of the EUP's Software Process Improvement (SPI) discipline.
Staff your SEPG with actual practitioners. The people who best understand how to develop software are the people who are very good at developing software. Yes, that sounds blindingly obvious, but what isn't obvious is that they are usually the best ones suited to define your software process. Try to staff your SEPG mostly with current practitioners, people who are now managers but had did some COBOL programming 20 years ago are often poor candidates, although include some people with a process background to ensure that they're not reinventing the wheel.
Reuse existing process materials. There is a wealth of process materials out there, you very likely don't need to write your own.
Avoid “fire hazard processes". A common mistake is to produce volumes of documentation describing your processes. Your goal is simply to describe your process materials to such a level that they can be given to a professional skilled in the techniques of that process so they can work the processes appropriately.
Adopt processes because they make sense. If a process makes sense to you, and you believe it will add value to your effort, then adopt it. Otherwise, do not.
Hold everyone responsible for process improvement. Senior management must be willing to actively support and sustain process improvement, project managers must be held responsible for ensuring that their teams follow the defined processes, and developers must be held responsible for learning and then following the processes. This is often a difficult task because senior management often demands immediate results, whereas process improvement often takes years. Project managers resent diverting scarce resources from their projects, and developers often resent being told how to do their jobs.
Bring in an expert to advise you. Process improvement is a complex and difficult endeavor, one for which you are likely to need help to accomplish. You can increase your chance of success by bringing in a consultant who has both a process background and an OO development background – someone who has been actively involved in a process improvement program and who has worked on large-scale, mission-critical software development projects using OO technology.
Do not think that everyone is on board. There is likely to be a small core of people within your organization who do not want to use object technology for large, mission-critical projects, and these people will actively undermine your efforts. You need to identify these dissenters and work together with them to help them see the advantages of working with object technology and of following a set of defined process patterns to help in the development OO software.
A fool with a process is still a fool. For your organization to be successful with a software process your software professionals will need to understand the processes, the concepts, the techniques, and the problem domain. Implementing a new process in your organization involves more than going out and purchasing a couple of new books and development tools.
Develop a user guide for your process. You can make it easy for your staff to learn your chosen processes by providing a well-written overview of your process as it is to be implemented in your organization. In fact, this may be all the process material that you need.
Have patience. Progress will be slow at first, slower than you hoped or expected. Introducing a software process into an organization takes time – the required culture shift often takes years to complete.
Don’t flounder in bureaucratic requirements. Too many process efforts run aground because of preconceptions forced on them by senior management, an overly burdensome documentation and review process, or unrealistic requirements to achieve consensus.
Define your process early. The longer you leave process definition the bigger the mess you will have to clean up. Without direction, developers will typically go and do what they think is right, the only problem being that each person has their own idea of what “right” is.

Let Me Help

I actively work with clients around the world to improve their information technology (IT) practices as both a mentor/coach and trainer. A full description of what I do, and how to contact me, can be found here.

Software Process Improvement (SPI) Best Practices

Suggested Reading

Let Me Help