Use Root Cause Analysis for Defect Prevention in your Software Development Process

Recently I decided to review all the bugs we fixed in our last release to determine if they originated from a few common causes.  In software development, there is not a lot of data that can be mined that may provide insight into actions to take to improve software development, but I thought this might be one.  We fixed about 40 bugs in the last release, and I can tell you right now that my analysis has not lead to any significant changes in the way we develop software.  This is not because we didn’t find areas that could benefit from improvement, it is just because, in our case, attempting to make improvements to reduce the source of the bugs is not currently more valuable than other tasks we can be working on.

Of the bugs we found, I considering them to be “coding flaws”,  “configuration flaws”, “design flaws”, “process flaws”, or “requirements flaws”.  I realize that my technique is crude and I have very little data to draw definite conclusions from, but this is an exercise that anyone on a project team can perform if you are tracking the defects fixed in a release.  You may find patterns that help you identify aspects of software development most in need of improvement.  An excellent paper based on a lot of software development data at NASA is online here: http://www.hpl.hp.com/hpjournal/96aug/aug96a2.pdf.  The goal of the study in the paper, and my goal is well, was to use this “root cause analysis” to determine ways to prevent and reduce defects in the software.  Given that excellent article provides much more detail than I enter into here, I recommend that you read it if you are really interested in improving your development processes through root cause analysis of your bugs.

I will add here, a list of reasons that I came up with for the sources of bugs in our code:

18% – Coding Flaws.  The programmer did not think the issue was a bug, or the programmer did not test the issue thoroughly, or the programmer felt it was insignificant and lacked the time to address the problem.

39% – Coding Flaws.  The programmer should have noticed the problem.  But perhaps senior developers or business analysts should have provided a junior developer more training and input so that the junior developer would have recognized this as a problem.

3% – Coding Flaws.  The bug is a bug in 3rd party software we use (in this case Microsoft), that we need to wait for them to fix, or rewrite the feature.

5% – Coding Flaws.  The programmer should have just provided an error message to the user that is easier to understand.  Perhaps the message provided should always be determined by a business analyst.

5% – Coding Flaws.  A known bug was released.  The bug had minor impact and the cost to fixing it was high, as was the cost of delaying the release.

8% – Configuration Flaws.  We use code generation for much of our code.  The business analysts made mistakes that led to improperly generated code.

5% – Design Flaws.  The design did not allow for a feature to work as it needed to in some situations.

8% – Process Flaws.  We don’t test 100% of the User Interface possibilities in our application because the time required makes it impractical.  However, such testing is unnecessary for most of the code as long as we follow our processes for development.  In a few cases, we failed to follow the correct processes.

3% – Requirements Flaws.  The developer was not aware that the feature was desired.

3% – Requirements Flaws.  The requirements provided to the developer were incorrect.

3% – Requirements Flaws.  We removed a feature from the software that we thought was not being used, but it was and we had to re-add it.

We also improved performance in the last release.  Another way to phrase that is that we fixed performance problems.  It is difficult to decide if improving performance is fixing a bug or not when it doesn’t violate a clear service level agreement.  A code change is not always objectively a bug fix or an enhancement.  If the client believes it was in the original specification, the client will say the missing feature is a bug; but if the developer thinks the feature was not in the specification, then the developer will consider the addition of it in the next release to be an enhancement.  Just part of the joys of software development.