Rob Kraft's Software Development Blog

Software Development Insights

Archive for the ‘Process’ Category

Subversion 1.7 provides a big performance boost!

Posted by robkraft on March 4, 2012

Last week we decided to install the latest version of subversion just to see if it fixed any minor bugs or could speed up performance a little. I’m thrilled to report that it improved performance by a lot, not a little! We can pull data out of the repository three times faster than before, and checkins and all other features of subversion are much faster. Subversion got a major redesign and refactor in release 1.7 that greatly affects performance on Windows. If you are using subversion on Windows, go out and get this great upgrade as soon as possible.

We use the subversion from Collabnet, and they provide a video and other great information free about the changes in subversion that have made it so much better.

Another important action item to take away from this is that all software development shops should examine upgrading all of their tools regularly to the latest release and service pack. Software developers know these upgrades can fix bugs, fix security flaws, offer new features, and, as in this case, provide large performance improvements. Make a list of all of your tools and then make a task to consider upgrading each tool at least once a year.

A big thank you to the team that contributes to subversion at Collabnet!
http://www.open.collab.net/downloads/subversion/

Posted in Coding, Dev Environment, Process | Leave a Comment »

Get started using FxCop against your nightly build without first resolving all the violations

Posted by robkraft on February 13, 2012

FxCop is one of the best known free static code analysis tools available for .Net software. Version 10.0 of FxCop was published in June of 2010 and is available for download here (http://www.microsoft.com/download/en/details.aspx?id=6544). FxCop is easy to install and use to analysze your code for potential problems, but I think the real value of FxCop is when you run it every night against your nightly build and receive a report via email of new violations recently introduced in your code. I have long believed this to be the best use and value of FxCop, but only recently implemented it. And the reason I only recently implemented nightly scans of our code using FxCop is because when I run the tool I get a report with thousands of violations. Ouch! Are all these violations things I really need to fix? The answer for me, and likely the answer for you, is no. However, a lot of the violations should be fixed.

I think of the violations in these groups:

  • Violations like misspelled words that are not misspelled, but just need to be added to a custom dictionary so that they are no longer flagged as violations.
  • Violations that could, and would, cause problems if that bit of code was ever used in ways others than intended, but I know that won’t happen.
  • Violations that would be problems if the software runs in other countries using different languages, different date/time formatting, etc.
  • Violations that really are problems we should fix.
  • Violations that represent waste such as methods that are never called.

So, as I stated we have thousands of violations across all of our code, but I didn’t want to implement the nightly FxCop scan until all of those were resolved. If you look at the categories I used for the problems, many of the violations are not worth our time to resolve because we have more valuable work to do right now. So how could we begin using FxCop regularly, without fixing all the problems? We did it by disabling rule checking for all the rules that were currently failing. Then we set up the automated job to run FxCop every night against our compiled assemblies. Doing it this way gives us a starting point, and it keeps us from violating many of the rules defined in FxCop. Our next step is to begin turning on rules and fixing them one by one. Of course we will start with the important or easy violations and work our way down the list until we have them all re-enabled.

From a high-level perspective, you have to perform just two steps:

  1. Build your FxCop project files that analyze your assemblies and ignore failing rules.
  2. Build your nightly batch job to run against all the assemblies and email results to you.

Some tips before you begin:

  • Tip #1 – Use the same directory path for the DLLs you analyze with FxCop on both your developer PCs and your build machine. FxCop stores the path to the DLLs you select in the .FxCop configuration file. If the path is the same between your developer PC and the build PC, then you can easily configure FxCop on your PC and it will run without modification on the build computer.
  • Tip #2 – Do not plan to use one .FxCop project to scan all your assemblies unless you have just 10 or 20 assemblies. At some point, .FxCop fails to work when too many assemblies are loaded.
  • Tip #3 – FxCop does not work on Silverlight projects.

Here are the steps I recommend:

  • Step #1 – Add all the assemblies for your logical group into the .FxCop project and analyze them. Logically group the DLLs and Executables you analyze.
  • Step #2 – Copy the CustomDictionary.xml file provided by FxCop into the folder you with your FxCop projects, modify the CustomDictionary to recognize some of your acronyms and names as valid – not violations.
  • Step #3 – Sort the results of the analysis by Rule, then from the Rules tab go through and deselect all the rules that you have violations for. Save the .FxCop project file.
  • Step #4-n – Use the .FxCop file you saved as the starting point for the next group of assemblies you desire to analyze (save it with a different name of course). Exclude all the targets from the first analysis and add your new set of targets.
  • Step #5 – We use nant in our nightly processes, and nant includes a task specifically for running FxCop. Here is how we do it. Create a batch file to run nant:
cd\
cd\dev\fxcop
REM This next line appends the FxCop directory to the existing path so we can run FxCop from any folder
PATH %path%;c:\program files\microsoft fxcop 10.0
C:\WINDOWS\system32\nant.bat fxcop -logger:NAnt.Core.MailLogger
  • Step #6 – Configure an nant .build file for executing FxCop. Ours looks something like this:
  • <?xml version="1.0" encoding="utf-8" ?>
    <project name="FXCOP" default="build"  xmlns="http://nant.sourceforge.net/release/0.86-beta1/nant.xsd">
    	<property name="MailLogger.mailhost" value="smtp.ourdomain.com" />
    	<property name="MailLogger.from" value="build@ourdomain.com" />
    	<property name="MailLogger.success.notify" value="true" />
    	<property name="MailLogger.success.to" value="developers@ourdomain.com" />
    	<property name="MailLogger.failure.to" value="developers@ourdomain.com" />
    	<property name="MailLogger.success.subject" value="FXCop Violations" />
    	<property name="MailLogger.failure.subject" value="FXCop Build Failure" />
    	<property name="MailLogger.success.attachments" value="MailLogger.success.files" />
    
    	<fileset id="MailLogger.success.files">
    		<include name="ProjectGroup1.html" />
    		<include name="ProjectGroup2.html" />
    	</fileset>
    
    	<!-- this loads any required dependencies that nAnt needs to do all of our tasks-->
    	<loadtasks assembly="C:\Program Files\nAnt\Contrib\bin\NAnt.Contrib.Tasks.dll" />
    <target name="fxcop">
    	<fxcop analysisReportFilename="ProjectGroup1.html" applyOutXsl="true" includeSummaryReport="true" projectFile="MinimalRulesGroup1.FxCop"/>
    	<fxcop analysisReportFilename="ProjectGroup2.html" applyOutXsl="true" includeSummaryReport="true" projectFile="MinimalRulesGroup2.FxCop"/>
    </target>
    </project>
    
  • Step #7 – Over time, enable rules in FxCop and resolve the violations that arise. Eventually, hopefully, you will be able to get all of your code to pass all FxCop rules.

On a final note, I’d like to make sure you are aware that you can also write your own FxCop rules to ensure the code your team is written follows your shop standards.

Posted in CodeProject, Coding, Dev Environment, Process | 1 Comment »

Use Root Cause Analysis for Defect Prevention in your Software Development Process

Posted by robkraft on February 5, 2012

Recently I decided to review all the bugs we fixed in our last release to determine if they originated from a few common causes.  In software development, there is not a lot of data that can be mined that may provide insight into actions to take to improve software development, but I thought this might be one.  We fixed about 40 bugs in the last release, and I can tell you right now that my analysis has not lead to any significant changes in the way we develop software.  This is not because we didn’t find areas that could benefit from improvement, it is just because, in our case, attempting to make improvements to reduce the source of the bugs is not currently more valuable than other tasks we can be working on.

Of the bugs we found, I considering them to be “coding flaws”,  “configuration flaws”, “design flaws”, “process flaws”, or “requirements flaws”.  I realize that my technique is crude and I have very little data to draw definite conclusions from, but this is an exercise that anyone on a project team can perform if you are tracking the defects fixed in a release.  You may find patterns that help you identify aspects of software development most in need of improvement.  An excellent paper based on a lot of software development data at NASA is online here: http://www.hpl.hp.com/hpjournal/96aug/aug96a2.pdf.  The goal of the study in the paper, and my goal is well, was to use this “root cause analysis” to determine ways to prevent and reduce defects in the software.  Given that excellent article provides much more detail than I enter into here, I recommend that you read it if you are really interested in improving your development processes through root cause analysis of your bugs.

I will add here, a list of reasons that I came up with for the sources of bugs in our code:

18% – Coding Flaws.  The programmer did not think the issue was a bug, or the programmer did not test the issue thoroughly, or the programmer felt it was insignificant and lacked the time to address the problem.

39% – Coding Flaws.  The programmer should have noticed the problem.  But perhaps senior developers or business analysts should have provided a junior developer more training and input so that the junior developer would have recognized this as a problem.

3% – Coding Flaws.  The bug is a bug in 3rd party software we use (in this case Microsoft), that we need to wait for them to fix, or rewrite the feature.

5% – Coding Flaws.  The programmer should have just provided an error message to the user that is easier to understand.  Perhaps the message provided should always be determined by a business analyst.

5% – Coding Flaws.  A known bug was released.  The bug had minor impact and the cost to fixing it was high, as was the cost of delaying the release.

8% – Configuration Flaws.  We use code generation for much of our code.  The business analysts made mistakes that led to improperly generated code.

5% – Design Flaws.  The design did not allow for a feature to work as it needed to in some situations.

8% – Process Flaws.  We don’t test 100% of the User Interface possibilities in our application because the time required makes it impractical.  However, such testing is unnecessary for most of the code as long as we follow our processes for development.  In a few cases, we failed to follow the correct processes.

3% – Requirements Flaws.  The developer was not aware that the feature was desired.

3% – Requirements Flaws.  The requirements provided to the developer were incorrect.

3% – Requirements Flaws.  We removed a feature from the software that we thought was not being used, but it was and we had to re-add it.

We also improved performance in the last release.  Another way to phrase that is that we fixed performance problems.  It is difficult to decide if improving performance is fixing a bug or not when it doesn’t violate a clear service level agreement.  A code change is not always objectively a bug fix or an enhancement.  If the client believes it was in the original specification, the client will say the missing feature is a bug; but if the developer thinks the feature was not in the specification, then the developer will consider the addition of it in the next release to be an enhancement.  Just part of the joys of software development.

Posted in Code Design, CodeProject, Process, Project Management | 10 Comments »

Send SQL Server query results as nicely formatted emails in Outlook

Posted by robkraft on February 4, 2012

I find myself sending the results of a lot of SQL queries I run in SQL Server Management Studio via e-mail to co-workers.  The problem is, that the email content often looks terrible when I paste it into outlook from SQL Server Management Studio.

However, I have found that I can make it look better using the following steps.

  1. Run the query and let the output results go to grid (the default for most people).
  2. Right-click in the results and select ‘Copy with Headers’.
  3. Open Excel and paste the results.
  4. Click in the upper left corner of the Excel grid to select all rows and columns.
  5. Double-click between any two columns (A and B) to resize the widths of all columns.
  6. Select the “Center Text” alignment so that all column values are centered, rather than the default left-justified.
  7. Now copy all the contents of the Excel spreadsheet and paste the results into the body of your email in Outlook.

I find the results usually look pretty decent.  Can you provide me a way to do this that is easier?

Posted in Process, SQL Server | Leave a Comment »

Reduce Waste During Software Development by Increasing Communication

Posted by robkraft on January 16, 2012

Lean software development focuses on reducing waste. One source of waste is time spent implementing a difficult solution when the product owner would have been satisfied with a simpler solution. The likelihood that this source of waste occurs increases as the interactions between developer and product owner decreases. When the environment discourages frequent communication between product owner and developer, and the developer discovers some minor features will be more difficult than expected, the developer is likely to attempt to implement the solution as the product owner requested it. But when the environment encourages frequent communication, the developer is more likely to let the product owner know that a specific feature is more difficult than anticipated, but that the same feature goals could probably be achieved with an alternative implementation. This allows the product owner to decide if the alternative implementation is sufficient, or if he wants to pay the extra time to have the feature as originally envisioned.  A few of the factors that can contribute to an environment that discourages developer to product owner communication include:

  1. Physical distance. For several reasons, physical distance discourages communication. Developers don’t want to interrupt the product owner, and it is difficult to know if the product owner is currently in a meeting, or involved in other activities when the product owner is not visible.
  2. Personal perception. Developers don’t want the product owner to think they are unable to make decisions themselves, and may be concerned the product owner will gain that impression if the developer is frequently asking for product owner input.
  3. Unfriendly product owners. Developers quickly sense when product owners are getting annoyed with questions. Once a developer feels he has used his allowance of goodwill, he is more likely to just attempt to implement difficult solutions per the original spec rather than suggest an alternative.
  4. Introverted developers. Some developers simply prefer to minimize interactions with people they are not extremely comfortable with. When faced with the choice of interacting with a person they aren’t comfortable with, or trying to make the software work as originally requested, they frequently choose the latter.

It does not matter whether you develop software using Lean (Kanban), Agile, or Waterfall. If you can increase communication between the product owners and developers, you can probably reduce or eliminate this source of waste.  Product owners pay the price of this waste, and they are also best positioned to reduce the problems I mentioned above.  Product owners, at least the people that make the detailed decisions about how the software should work, should strive to make themselves constantly accessible to software developers.  When a developer comes with a question, the product owner should treat the developer as a welcome interruption, not as an inconvenience.  Product owners can also improve the relationship with the developer by engaging with the developers on social, non-work, levels.  This fosters a growth in trust between product owners and developers and increases the odds that developers will come to the product owner when the potential exists to eliminate waste.

Posted in Code Design, CodeProject, Coding, Process | 4 Comments »