Rob Kraft's Software Development Blog

Software Development Insights

Archive for the ‘Process’ Category

What Makes A Software Programmer a Professional?

Posted by robkraft on June 16, 2019

Many people write code, but not everyone that codes considers themselves to be a professional programmer.  What does it take to be a professional?  This article lists the practices you undertake when you are a software development pro.  From my experience, many of these practices are not taught in coding schools, but they are essential to delivering quality software to customers.

Before covering the practices, let’s first briefly consider different classes of those who write code:

The Tinkerer
A tinkerer is a person that writes a few lines of code here and there, perhaps a macro in Excel, perhaps connecting services in IFTTT or Microsoft Flow, or perhaps a script to run in a database. A tinkerer may also use a language like basic or javascript in order to create a program or web site for their own personal use and enjoyment.

The Amateur

A programmer becomes an amateur instead of a tinkerer when the programmer starts writing software or web sites for others, especially when the programmer is compensated for their work.  Amateur programmers often create good software and write code well.  But to be considered professional, there are several practices a programmer will follow.

The Professional

The following lists represents what I believe are practices that every professional software developer will follow.  As in all things, there may be some cases where it makes sense that one or two of these practices is not performed by a professional; but I believe most would agree that professionals follow all of these practices most of the time.

  1. Use version Control
  2. Back up everything off site
  3. Track the changes, fixes, and enhancements in each release
  4. Keep the source code related to each deployed version that is in use
  5. Keep a copy of your software in escrow
  6. Use automated builds
  7. Schedule builds
  8. Write regression unit tests
  9. Use a bug tracking system
  10. Use a system to track tasks and features being developed
  11. Keep customers informed about the progress of the software development
  12. Keep third party software used updated regularly
  13. Understand the security risks
  14. Ensure proper compliance with industry standards such as PCI, HIPAA, SOX, and PII
  15. Educate yourself continuously
  16. Invest in your development tools
  17. Properly license development tools and software

1. Use Version Control

A version control system should be used by all professional software developers.  It is difficult to imagine a solution for which version control would not apply.  Today, most developers use GIT, but many also use Subversion (SVN).  The version control system used should have the following features:

  • Allow people to add comments explaining each check in, and track who checked it in
  • Allow people to view a history of changes checked in for each file
  • Allow people to revert to earlier versions of the software
  • Allow people to compare the code changes made across check ins

Zipping up the files that are part of each product release or check in, instead of using a version control system, is a sign of an amateur programmer.

2. Back Up Everything Off Site

A professional programmer will make sure that all of the code they write is backed up regularly, and that back up needs to be an offsite location to prevent loss of all of the source code in the event of a fire, flood, theft, or some other event.  Given the ubiquity of the Internet today this is usually easily achieved.  Simply using an online repository like github or bitbucket for version control almost fully meets this practice in most cases.  Along with the backups of the source code, a professional will make sure that that the scripts and commands for the tools and processes used to build the software is also backed up remotely.  A professional programmer should be able to start from nothing, acquire a new computer, and reconstruct everything needed to continuing developing the software by recovering all of the software from the off site backup.

3. Track the Changes, Fixes, and Enhancements in Each Release

A software professional provides more than software to their clients, they also provide a list of all the new features included in the latest version.  Of course there should also be a comprehensive list of the contents of the initial version.  In addition, a list of bugs fixed and other changes such as noticeable enhancements to performance and security should be provided.  A software professional tracks the reason behind every code change made during a release.  This is often helpful when clients are looking for a feature or fix because you can tell them they need to upgrade to a specific version to get it.

4. Keep the Source Code Related to Each Deployed Version That is in Use.

Modern version control software allows us to see how the code looked at any point in time.  A professional should be able to easily and quickly see how the code looked for any version of the software that any client currently has implemented.  This helps the developer resolve issues reported by clients more quickly, and to create a fix for the software.

5. Keep a Copy of Your Software in Escrow

No one likes to ponder the possibility of their own demise, but as a software professional you should consider this possibility and take steps to ensure that your clients can continue to use your software if something unfortunate happens to you.  Make sure that someone besides yourself can obtain access to your source code and all the artifacts and processes needed to build and support it if the need arises.

6. Use Automated Builds

Building software takes us from the source code to the final output that is deployed to a client or to a production system.  While builds often start out simple they can evolve to become more complicated.  For this reason, a professional developer will set up an automated build process that compiles and combines all the source code to a package ready to deploy.  Minor tasks that can be automated so that they are not forgotten include flagging the code as a release build so that the compiler will optimize the output, or minimize it, or obfuscate it as needed.  Automated builds often update version numbers and also often perform several steps in a specific sequence.  Build processes can be automated initially with simple batch/command files; but many professional use tools specifically designed for building software products.

7. Schedule Builds

Scheduling a build is just the next step, and hopefully a minor step, after a professional has created an automated build.  Many developers schedule a build to automatically run once or twice a day.  This is especially advantageous when multiple developers are contributing to the code.  Some developers even configure the version control system to start a build every time code is checked in.  Frequent builds help developers identify bugs more quickly if an artifact necessary for the build was excluded from their repository checkins, or if code they checked in adversely affected code in another part of the system.

8. Write Regression Unit Tests

I’ll admit that there may be a few cases where it does not make sense to write unit tests for your software.  But I believe that in almost all cases, even for old languages like Cobol and Visual Basic, a developer will write unit tests to validate important logic in the software runs correctly and is not inadvertently altered by a new enhancement or related bug fix.  Getting a unit test framework up and running takes a few hours or even a few days for someone not familiar with unit tests, but once you have it, you find that the tests give you a lot of freedom to make changes and the peace of mind that the changes you make are not breaking existing logic that your customers depend upon.

9. Use a Bug Tracking System

Let’s face it, almost all software has bugs, or things that are not working quite as the user desires.  Software developers need to track when those bugs were discovered and who discovered them so that they can make a careful fix and provide a patch or fix to the affected systems and users.  A bug tracking system can help all customers and users of your software become aware of the bug and sometimes the way to workaround a bug until a fix is applied.  It also can let your customers know that they just need to upgrade to your latest version to obtain the fix for the bug.  In addition, it helps you keep track of when and where you made the fix so that you can manage related issues with the fix, follow up issues, or just to reminder yourself that you already fixed this bug.

10. Use a System to Track Tasks and Features Being Developed

Software professionals understand that delivering quality software includes keeping track of a lot of details.  A system and place to keep track of all of the things that need to be done is very valuable.  Your system could be something as simple as a Trello board.  Ideally you will have a list of everything yet to be completed and everything yet to be done.  Most tasks boards use at least three columns: To Do, Doing, and Done.  Even when you think a task is done you may decide it is not really done if you have not written the documentation related to the feature, or altered your install package to include a dependency for the feature.  The system also helps you remember the status of items and share the status, or share the work to be done, with others.

11. Keep Customers Informed About the Progress of the Software Development

In most software development projects, frequent interaction with the end user, aka ‘the customer’ greatly increases the chance that the software you create meets their needs.  Frequent communication also helps you manage your customer’s expectations about when features will be delivered to them and how those features will behave.  Although the waterfall method of software development describes collecting all of the requirements up front, it is extremely rare that you really can gather all of the details as well as you can if you keep the customer involved as you develop and can show them the product as it evolves.  Professional software developers will not leave their customers in the dark for weeks or months before giving them an update about the progress of the software.

12. Keep third party software used updated regularly

Most developers rely on some third party software to decrease the time it takes to produce the software their clients want.  For web developers, this may include frameworks like Microsoft .Net, Angular, React, JQuery, and Java.  But security flaws and performance problems, and other bugs occasionally get discovered in these frameworks, therefore a software professional regularly updates to the latest version of the frameworks in order to obtain the security patches and fixes to improve the security and performance of the software they pass on to their client.

I recommend you keep a list of all the third party software used by your software and your software development processes, and that you review the list at least twice a year in order to update each dependency to the latest version.

13. Understand the security risks

Professional software developers understand that they generally have more knowledge of software development than the customers that have hired them to write code.  Thus they understand that writing secure code, code that can’t be easily abused by hackers, is their responsibility.  A software developer creating web applications probably needs to address more security risks than a developer writing embedded drivers for an IOT device, but each needs to assess the different ways the software is vulnerable to abuse and take steps to eliminate or mitigate those risks.

Although it may be impossible to guarantee that any software is immune to an attack, professional developers will take the time to learn and understand the vulnerabilities that could exist in their software, and then take the subsequent steps to reduce the risk that their software is vulnerable.  Protecting your software from security risks usually includes both static analysis tools and processes to reduce the introduction of security errors, but it primarily relies upon educating those writing the code.

OWASP (https://owasp.org) is a good resource for developers to learn about possible vulnerabilities and ways to mitigate those vulnerabilities.

14. Ensure Proper Compliance with Industry Standards such as PCI, HIPAA, SOX, and PII

Writing software that complies with industry regulations is also a responsibility of a software professional.  It is the responsibility of the customer asking for the software to tell the developer that they need the software to meet specific regulations such as HIPAA, GDPR, PCI, SOX, or PII.  But is the responsibility of the software professional to understand how those regulations affect the software and software development processes.  A customer may suggest to the developer what impact the regulation has on the code, but if you are a software professional, you will refer to the regulation directly and clarify your own interpretation of the document.

15. Educate Yourself Continuously 

Technology continually changes thus professionals will continually learn new tools, techniques, and software languages in order to improve their efficiency and write software that lasts longer.  Even developers that have been programming in the same language and environment for a decade may discover that there are new tools and processes that can help them write better code.  Perhaps need static code analysis tools can help you find bugs in your code, or perhaps you can learn to write better code by switching from a waterfall methodology to an Agile approach.  Developers writing Visual Basic 6 code may realize they can begin writing more modular code and use classes to facilitate a possible rewrite to Java that is coming, but is still years away.  If you aren’t occasionally doing some research to find ways to improve, I don’t think you can consider yourself to be a professional software developer.  (If you are reading this article, you probably are a professional or planning to be one!)

16. Invest in Your Development Tools

A good carpenter doesn’t use a cheap saw, and a good software developer doesn’t just use the cheapest tools and equipment.  Luckily for software developers, many really good tools are free to use.  Some tools do cost a little bit of money, but the productivity improvement gained by paying for them is often well worth the price.

Besides software tools, developers sitting at computers writing code day after day probably benefit from having good hardware.  A computer with a lot of RAM and a fast hard drive and Internet connection may spare you waiting minutes or even hours each day.  Multiple monitors, a comfortable keyboard, mouse, and chair can contribute to your ability to write code a little more effectively each day.  Take time to invest in the small things that can improve your productivity, even if it is just by a small amount.

17. Properly License Development Tools and Software

Customers expect people they hire to work ethically.  Well, perhaps unethical customers don’t expect ethical behavior or are willing to turn a blind eye, but if you declare yourself to be a professional software developer I believe you are also declaring that you do things ethically, and part of being ethical is paying for the resources you use to develop the software solution.  This means that you will pay the correct price and licensing fees for the tools you use for development.  It also means that you won’t use the free or community version of tools to develop professional solutions that clearly don’t qualify for the rules of use for that version of the development product.  Just as you are probably making a living writing software for your customers, those people and companies that created the tools you are using to enhance your productivity are trying to make a living by writing their software too.

Summary

I wrote this for people that are new to software development, particularly those who have completed a program in writing software and hope to embark on a career as a professional.  Most of the items covered in the article are not covered by formal software education programs, but are an essential aspect of writing good quality software.  Individuals writing software for a living on their own probably want to implement all of these practices in order to give current and potential clients confidence in the quality and professionalism of their work.

Please let me know if you feel something important is missing from this list so that we may improve this article as a good reference for developers.

And please check back for a future article where I cover those practices that make a software development “team” a team of professionals.

Advertisements

Posted in CodeProject, Coding, Process, Project Management | Leave a Comment »

Agile Baby Steps: A Parable To Help You Get Started

Posted by robkraft on March 20, 2016

We often hesitate to take the action that shows we are committed to doing something new. We read about it, analyze it, and try to understand it; but real learning requires that we go beyond reading. We must DO. The goal of this article is to get you to take action toward becoming Agile, without understanding or adopting all of the habits of an Agile development team. I am asking you to try out some new processes in your software development life cycle, without considering whether or not you are doing Agile development.

Side bar: Your ability to implement an Agile technique depends upon the process by which your software is implemented. An Agile development technique that works for one process may not work for another, so be cautious of Agile recommendations that state you must do something specific or you will fail at being Agile. Your goal is not to be “Agile” by anyone’s definition. Your goal is to write better software. 

  • Some software developers write code then send it to a quality assurance environment who then push it into production;
  • Some software developers write code for embedded systems where all the software must be completed before it is written to a chip;
  • Some software developers check in code that that runs through automated tests and gets published to a public web site without further human action;
  • And some software developers follow other models for implementation.

The process by which you take code from development into its final environment greatly affects which agile techniques will work for you.

The Parable Begins

Let me share with you a parable of two teams, each tasked with developing the same software, but each using a different methodology.

Team Agile used an agile methodology, and team waterfall used a waterfall method. Both teams were asked to build a web application to allow users to manage data in an existing client-server application.

Team Agile met with the end users, usually known as the product owners in agile, and learned the high level requirements and features desired for the entire application. Because they did not gather detailed requirements, they spent only eight hours on this task.

Side bar: Delaying the gathering of detailed requirements often adds value in several ways:

  1. You don’t spend time gathering and documenting detail requirements for features that later get cancelled or excluded from the project.  If the team spends ten hours detailing the requirements of a feature that management decides not to implement a month later, then the team wasted ten hours. Eliminating waste is the major focus of Lean and Kanban styles of Agile development.

  2. Another reason to delay gathering detailed requirements is that every team member will be smarter in the future than they are today.  Each will know more about the application and may learn that ideas considered early in development are not as effective as new ideas learned since. Perhaps a developer read about a new programming technique or tool; or perhaps the product owner learned about a better way to design a user interface. You can implement these new ideas and techniques even if you documented detail requirements for the old techniques, but that means the time spent gathering and documenting requirements for doing it the old way was wasted time.

Prioritization

Team Agile also asked the product owners which features were most important. The product owners initially said all of the features were necessary and important, but after more discussion the product owners provided this prioritized list of features:

  1. Users need to be able to log in
  2. To view data
  3. To edit data
  4. To add data
  5. To delete data

Prioritizing features is an important, and necessary feature of agile development, as we shall see later. If you do not prioritize the features you are going to work on, you will probably not receive the benefits that agile development can provide.

Meanwhile, in a parallel universe, the Team Waterfall also met with the end users to gather requirements. They spent much more than eight hours on this task because they needed detailed requirements for all the features. They planned for little interaction with the product owner after this meeting until the product was finished. Team Waterfall spent sixty-four hours on requirements.

The Login Feature – Deploy Early

Team Agile next then did some design for the project. They thought about all of the requirements they had learned about, but they only did a detailed design for the first feature they worked on; and that feature was the ability to log in. They spent four hours on the design of this feature.

Then Team Agile coded the login feature. The coding took eight hours. Next, Team Agile turned the application over to the Quality Assurance (QA) team. Even though the entire application was not completed the QA team found a few problems with the login feature. Team Agile fixed those problems, and the QA team could find no more defects.

Side bar: Agile development does not magically prevent programmers from creating bugs, but it does make developers aware of the bugs sooner, so they can fix them while the code is still fresh in their minds and before some errors might get propagated into more of the code.

Team Agile implemented the application in production. Now, this seemed a little silly to some people, because the application did not do anything other than let a user login; but it turned out to be very valuable. They discovered that the software did not work in production. The production environment had an older version of the web server that lacked some features the application depended on. Team Agile met with IT to discuss the problem and decide if the web app should be re-written, or if the production web server could be upgraded to a newer version. They decided the web server could be upgraded, but it did require two weeks for this to be completed.

Agile Manifesto Principle #1: -“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”

Side bar: Not all software can be deployed in small pieces, such as software embedded on chips or shrink-wrapped software. But some software, like the software in this parable, can be deployed in pieces. By taking the software as far as possible along the path of implementation you may discover problems. It is always best to know about problems sooner so they can be accounted for in the project schedule and possibly used to correct the product requirements, design, and code. A product developed using a waterfall method has a higher risk of failing to discover some problems until all the code is completed and thus incurs significantly higher costs to correct the problem.

Side bar: Agile methodologies reduce the risk of unknown and unexpected problems by revealing them sooner.

The View Feature

While Team Agile waited for the new web server to be implemented in production, they proceeded to work on the second feature, “Allow users to view data”.   They met with the users to get more detailed requirements about how they would like to view the data. They spent eight hours on this task. Team Agile then created a design, including some mockups and reviewed the mockups with the product owners. After this sixteen hours of work the developers were ready to begin coding.

I have not forgotten about Team Waterfall. During all this time that Team Agile did the activities above, Team Waterfall has been gathering requirements. Team Waterfall is now ready to design the application, and they will spend about forty four hours in design, which is a little less than the total amount of time Team Agile will spend on design. In this parable, Team Waterfall benefitted by designing the entire application all at once because it was all fresh in their minds as they did it. Team Agile, on the other hand, did parts of the design spread out of several months, and had to spend part of that time recalling why some decisions were made. However, the advantage still goes to Team Agile, as we shall see, because Team Waterfall will discover that much of their time spent in design was wasted.

Team Waterfall completed their design then started coding. They chose to code the view and edit features first, and at the same time because they believed them to be the most interesting and fun part of the code to write. For both Team Agile and Team Waterfall, the coding phase(s) of the application take the longest; around three hundred hours. At the same time that Team Waterfall is working on the total application design, Team Agile begins coding their second feature, “Viewing Data”.

Communication With Product Owner

For both teams, the time spent coding is the same for all features except for “Viewing Data”. Team Agile spent one hundred and twenty hours coding this feature, but Team Waterfall is going to spend one hundred and sixty hours coding this same feature for the following reasons.

  • In the first week, a developer attempted to implement a list box on a form as had been requested in the requirements. But the developer found that this data would be difficult to display given the list box features. He realized he could easily do this with a grid though.  So the developer brought this up with the product owner during the daily status meeting, and the product owner said he didn’t care if it was a list box or grid, he barely understands the difference between the two, and he would just prefer to defer that decision to the developer. So the developer used a grid instead of a list box and saved an estimated forty hours of work that would have been needed if he had tried to make the feature work using a list box.

Agile Manifesto Principle #4 – “Business people and developers must work together daily throughout the project.”

  • In the second week, another developer was working on a feature to let users pick their own colors for the forms. The requirement called for a text box in which the user could type a hexadecimal value representing the color, but the developer had recently learned about a component that could just as easily provide a color picker to make it much easier for the end user. Instead of adhering to the requirements the developer showed the product owner liaison an example of the color picker and asked if this change would be acceptable and the product owner liaison loved the idea, so it was implemented.

Agile Manifesto Principle #6 – “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”

Side bar: Once again, the ability for frequent interaction between the developers and the end users throughout development facilitates many improvements. Also, developers are often more aware of the capabilities of technology than the end users and can make suggestions for improving the application based on that knowledge. When the discussion for detailed requirements can be delayed and the developer writing the code can be involved, there is a greater chance for a better solution.

Side bar: A good technique for software development, and for many decisions in life, is that it is best to commit to a decision as late as possible because your knowledge later in the life cycle is greater than it will be earlier in the cycle.

Reports Feature – New Requirements

During Team Agile’s development of the “View Data” feature, the product owner realized they had omitted the reports feature from the project. The reports are used by every user and are much more important than the ability to delete, add, or even edit data. The product owner and the developers had a meeting about the omission and decided that the developers would add the report feature next, after they finished View feature.

Agile Manifesto Principle #2 – “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” The ability to accept new requirements and to change the priorities of features developed is one of the most noticeable and valuable aspects of agile development.

The development team finished the view feature and easily deployed it into production. The product owners started using the application, even though all of the features were not available.

Agile Manifesto Principle #3 – “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. “

Deploying the view feature provided several benefits:

  • The company could begin deriving value from the application. In financial terms, the Return On Investment (ROI) starts occurring sooner in Agile projects than in Waterfall projects.
  • The users became more productive because the web application was easier and faster to use than the client server application. It was also easier for the IT staff to make it available to more users.
  • The users found bugs in the application. Finding some of these bugs may prevent similar bugs from being developed in the remainder of the application. For example:
    • The users found that there were no accessibility keys. So the development team planned to add these to the view screens, and were proactive about adding this feature in future development.
    • The users became more productive because the web application was easier and faster to use than the client server application. It was also easier for the IT staff to make it available to more users.
    • Twenty percent of the users found that some features did not work on the particular browser they used, which was different than the browser used by the developers and most of QA.
    • A few bugs were found causing incorrect data to be displayed.
  • Side bar: Teams often desire to fix bugs right away, but in an Agile environment, especially one with short one-week or two-week iterations, this can be counterproductive. It is generally best to let the team complete what they started in an iteration, then make the fixes to the bugs a top priority for the next iteration.

Waterfall Team Progress

Let’s check in on Team Waterfall. At the same time Team Agile is coding and deploying the “View Data” feature, Team Waterfall chose to code the “View Data” and “Edit Data” features, and they are still in the process of doing this. They have nothing yet to show to the product owners, so let’s turn back to Team Agile, because they have some visible progress that we can check on.

Agile Manifesto Principle #7 – “Working software is the primary measure of progress.”

Team Agile finished the “View Data” feature, and started to develop the “Reports” feature next. The requirements and design only took sixteen hours, and by this time the reports of bugs in the “View Data” feature were coming in. However, the team felt they could and should complete the “Reports” feature before working on the bugs so that they did not incur the cost of switching back and forth between tasks. The product owners accepted this decision because the development iterations were short, and the development team said they could start fixing the bugs next week.

After finishing the “Reports” feature and deploying it to production, the team spent a week fixing the bugs in the “View Data” feature. Some of the bugs had made some views unusable, and other bugs, such as accessibility and support for other browsers, would affect how they developed all subsequent features.

Side bar: Agile development does not magically eliminate bugs, nor does it prevent errors in requirements, design, and communication. But Agile development does reveal most bugs sooner so they can be fixed more quickly and cost less to correct than they would in waterfall development.

Team Waterfall is still coding the “Edit Data” feature at this point in time. It took them longer to code the “View Data” feature than it took Team Agile because Team Waterfall made the list box work as documented in the requirements rather than go back and talk to the product owners about using a grid instead. Team Waterfall also spent time explaining to the product owner that they could not add the “Reports” feature to the product because they had already gathered all of the requirements and done the design and they would need to redo some of that for a new feature. Ultimately, the product owner agreed to increase the product budget and provide proper paperwork for a “Change Request” to the requirements of the system. Team Waterfall then spent eighteen hours gathering the requirements and changing their design, which included changing the design of some database tables they had not started coding against yet. Since Team Waterfall still has nothing to show, we will go back and see what is happening with Team Agile.

More New Feature Requests

Team Agile has started the requirements and design of the “Edit” feature. During development of the edit feature, the product owner realized they had omitted filtering and sorting abilities in the view feature and that filtering and sorting was really necessary to make the views more valuable. Team Agile decided they would add sorting and filtering to views right after they completed the editing feature.

The Cancellation of the Project

But in the next week a new project came in and the developers were asked to suspend this project and work on the new project. The new project was very important, of course, and would probably take the team a year to complete. Team Agile was given one week to wrap up this project and begin work on the new project. For Team Agile, the editing feature was almost done, but the date and time pickers only worked on one browser and the developers estimated it would take them three to five days to get new date and time pickers working on all browsers. Team Agile had to choose between these options:

  1. Add filtering and sorting to views and not release the edit feature
  2. Finish the edit feature so the date picker worked on all browsers and users would not have to type in dates, but not add filtering and sorting to views
  3. Add filtering and sorting to views and release the edit feature without a date picker, requiring users to type in the date.

Team Agile desired to complete the edit feature by making the date picker work because they did not want to provide the end users with a subpar, low quality product; and they thought it would make the developers look bad if the app did not have the simple date pickers. But the product owner said that the filtering and sorting of views was most valuable, and that they would take the editing feature even without the date pickers because the ability to edit data from the web application would provide some value to users even if the feature was not finished perfectly.

Side bar: Agile development often gives the product owner insight into the product development processes and decisions. This is almost always a benefit to the business because the product owner can help guide the product outcome to a solution that provides the best ROI for the business. However it can, occasionally, upset some developers when they feel they are asked to cut quality to get the job done. The developers may feel it will reflect poorly on them. It is up to the management teams to convey to the end user the decisions made in this situation were those of management, and not the developers.       Waterfall teams rarely have this dilemma because the product owner is unaware of all the decisions made.

Speaking of waterfall teams, as in the side note, what is going on with Team Waterfall now that this new project has arrived and they must work on the new project instead. Well, one benefit for Team Waterfall is that they can start on the new project right away instead of spending a week trying to wrap up the old project because there is no way Team Waterfall can deliver anything within one week on the old project; they never even started the Login feature of the application. The obvious enormous downside for Team Waterfall is that they will deliver nothing to the end users, and all the time spent on the application can now be considered waste. That is not the case for Team Agile. Even though the project was terminated early, the agile team delivered something of value that could be further enhanced in the future.

Agile Manifesto Principle #10 – “Simplicity–the art of maximizing the amount of work not done–is essential.”

I provide two summaries to this parable. The first, is a summary specific to the tale, and the second is a summary of general conclusions to be made about agile development.

Specific Summary

  • Team Agile delivered some business value, but all the time spent by Team Waterfall was a waste.
  • Team Agile reduced the development time of some features by frequent interaction with end users and by being open to changing the requirements.
  • Team Agile provided a better way for end users to choose their colors than team waterfall because the UI decision was not made until the feature was developed and in that time the developer had learned of a new component.
  • Team Agile accommodated the “Report” feature because they had a prioritized backlog and could easily queue it up to work on next. Team Waterfall did not prioritize their work, so any new development would probably just be added at the end. Team Waterfall would need to alter their existing requirements and design.
  • Team Waterfall never learned that their app would not work in production due to the older web server. It is probable that the team would be rushing to deliver this product by a deadline, only to discover right at the end that additional time would be required. It could have been even worse if IT was unable to upgrade the web server and the development team had to go back and change code to make the application work on an older web server.

General Summary

  • Agile teams often waste less time than waterfall teams.
  • Frequent interaction with end users can produce a better product with less waste. This is not exclusive to agile development, but it is more common to agile development than to waterfall.
  • The willingness to accept flexible requirements can produce a better product with less waste. This is more difficult to do when all requirements have been gathered up front and have been included in a design.
  • Delaying requirement and design details can lead to better decisions at the time the decision needs to be made.
  • Agile teams accept new requests easily by adding them in the backlog. They do not have a lot of time invested in any features in the backlog because they wait and do the detailed requirements and design for them when they are about to code them.

If you want to become more Agile today:

  • Create a prioritized backlog
  • Select features from the backlog that you will complete during your next iteration. A good iteration length is two weeks.
  • Make sure that you don’t just code the features, but that you include testing and deployment, if possible, to be done within your iteration.
  • Do not work on several things at the same time. Complete each feature as much as possible.
  • Finish what you start each iteration. Do not add interrupt what you started in an iteration by working on something new that came in to the backlog. Wait until the next iteration to start it.
    • Sometimes, something very high priority will come in that must be completed right away. Agile developers understand and accept this.

Posted in CodeProject, Process, Project Management, Uncategorized | Leave a Comment »

A Simple Build Summary Email For Your Development Environment. Tests Included!

Posted by robkraft on January 13, 2015

We cobbled together a simple system to provide us a single email each day to tell us if all of our jobs and unit tests were successful on their overnight runs.

An example of our daily build summary email.

An example of our daily build summary email.

 

Given that we have a dozen different jobs that run tests and a dozen that run builds it was helpful to have the results of all the jobs in a single email. To implement this similar to what we did you need the following:

  • A database for storing the results of the jobs. We used SQL Server.
  • A program to write the build results to the database. The C# program we wrote is provided here.
  • A few modifications to your build scripts. Our changes to our Nant builds are shown here.
  • Optionally a testing framework that produces output you can read easily.  We use nunit.
  • A report sent via email. We used an SSRS report to send the email summary each morning.

 

Here is how we did it:

1) We created a table in SQL Server to store the results of each job.

SQL for Creating Table ZBuildJobs

SQL for Creating Table ZBuildJobs

Here is a sample of the table with data:

Sample Of Table ZBuildJobs with data

Sample Of Table ZBuildJobs with data

 

2) We created a program that we could call from a command prompt and pass a few parameters to write into the table.   This is our C# program.  It is comprised of just 3 methods.

The App entry code reads the command line arguments.  If there are more than 2 (> 3 in code) then we also look for a number of tests passed and failed to be passed in.

Code Part 1

Code Part 1

The CallDatabase method simply writes the values passed in to the database table.

Code Part 2

Code Part 2

The ProcessTestResults method uses some example code from the Microsoft URL shown to read the results of the nunit test file and count the number of passed and failed tests.

Code Part 3

Code Part 3

 

3) We modified our nant build files to call the program written in step 2 and pass in a value of success or failure.  For nunit test jobs we also pass in the XML results file created by our nunit tests.

  • Nant includes some built-in properties that we can use to specify a target to call when the nant job succeeds or fails.  So we configured those properties like this:
    • <property name=”nant.onsuccess” value=”BuildSuccess” />
    • <property name=”nant.onfailure” value=”BuildFailure” />
  • For ease of maintenance, we created a property to hold the name of our program from step 2 above that does the logging:
    • <property name=”param.EventLogPgmName” value=”C:\Dev\BuildLogger\BuildLogger.exe” />
  • To include spaces in our job names we need to surround the parameters we pass to BuildLogger with double quotes, and to include double quotes as part of the parameters we needed to create a property to hold the double quotes.  We also have a property to hold our current build version number:
    • <property name=”dq” value='”‘/>
    • <property name=”project.version.full” value=”15.0″/>
  • Then at the start of each build target we add a name for that build, something like this:
    • <property name=”param.BuildNameForLog” value=”${dq}${project.version.full} C# .Net Build${dq}” />
  • If the job runs unit tests, we include a property to specify the name of the unit test output file:
    • <property name=”param.TestResultsFile” value=”${dq}c:\program files (x86)\NUnit\bin\TestResult.xml${dq}” />
  • The final change to our build files is to declare the targets for BuildSuccess and BuildFailure.  These look like this:
    • <target name=”BuildSuccess”>
      • <exec program=”${param.EventLogPgmName}” commandline=” ${param.BuildNameForLog} Succeeded ${param.TestResultsFile}” failonerror=”true”/>
    • </target>
    • <target name=”BuildFailure”>
      • <exec program=”${param.EventLogPgmName}” commandline=” ${param.BuildNameForLog} Failed” failonerror=”true”/>
    • </target>

4) The last step is to build the report.  If you are already familiar with SSRS you will probably find this step to be very simple.  SSRS is easy to use and is free if you have a Standard Edition of SQL Server.  Here I share the more advanced features for the report.

This is the SQL to get the most results of the most recent execution for each job:

SQL for Report

SQL for Report

For each job listed on the report we want to show the record in green (Lime) if it passes and red (Tomato) if it fails.  So I modified the BackgroundColor property of each row and gave it this expression:

=IIF(Fields!Result.Value = “Succeeded”, “Lime”, “Tomato”)

For the two columns with the number of tests that passed or failed, I had to test if the field had null first, then apply the color:

=IIF( IsNothing(Fields!TestsFailed.Value), IIF(Fields!Result.Value = “Succeeded”, “Lime”, “Tomato”), IIF(Fields!TestsFailed.Value > 0, “Tomato”, “Lime”))

We have been very pleased with the results of our Build Summary process and continue to add more things to it.  It was easy to implement and has ran very reliably.

 

You can download the code from here: http://www.kraftsoftware.com/BuildYourOwnSummaryEmailCode.zip

Posted in CodeProject, Coding, Dev Environment, Process | Leave a Comment »

Use Fuzzy Hours for Your Software Task Time Estimates

Posted by robkraft on June 14, 2014

Like many software development shops, we need to provide estimates of the time required to fix each bug and add each new feature.  The primary reason for the estimates is to set expectations for when tasks will be completed and to prioritize what is to be worked on.

As an agile shop, we have tried a few scales for estimating such as:

tiny, small, medium, and large and also the Fibonacci sequence:    1, 2, 3, 5, 8

but the one we have been using for years that we like best is fuzzy hours.

We use the symbols 1,2,3, and 4 where:

  • 1 means 0 to 2 hours
  • 2 means 1 to 8 hours
  • 3 means 4 to 40 hours
  • 4 means more than 30 hours

Notice that our ranges overlap and that is what makes them fuzzy.  When an estimator feels an estimate is in a fuzzy zone it is up the estimator to pick one zone or the other based on their gut feeling.

We have also learned that the velocity for all items sized 1 and 2 are the same, and items with a size 3 take only slightly longer to reach a completed state (development, testing, quality assurance, and documentation all complete).  Items with a size of 4 take longer, but only about twice as long on average as a 3.

Posted in Estimating, Process | Leave a Comment »

There are no I.T. Projects, only Business Projects

Posted by robkraft on October 28, 2012

Businesses often turn to software and computers to solve problems and gain a competitive edge.  When this occurs, both the business analysts and the I.T. staff have a solution in mind that requires computing technology and software.  Sometimes though, non-technical solutions solve problems better and with far less expense.  One current classic example is the use of a white board to track the project of work being done during a software development iteration.  That white board does not need to be electronic, so don’t spend weeks or months creating the white board software especially when all interested parties are located in the same office or room.

Keep in mind these three principles:

  • Can we try out the new process or idea manually, with minimal investment, without spending the time to develop the idea using software.  If we feel the process still requires automation after a few months, can we at least learn some of the requirements from performing the process manually in the first few months.
  • Is the cost of developing the software the best use of our scarce resources?  If purchasing a software solution, is the cost of the purchase worth the investment?  Many companies spend a lot of resources developing software that is not part of their core business.  Doing so is depriving your business of the resources it could be using to better the products that really drive the business.
  • Just because your software developers are busy does not mean you are making progress.  I once learned of a doctor that traveled an hour between two hospitals.  He saw four patients a day because he spent four hours traveling back and forth.  The doctor was constantly busy, but the doctor was not near as productive as he could have been.  Once the doctor changed the schedule of the patients to meet all the patients at one hospital in the morning, and all the patients at the other hospital in the afternoon, the doctor was able to see twice as many patients each day.

Posted in Process, Project Management | Leave a Comment »

Make Time to Learn from the Free Microsoft Security Assessment Tool

Posted by robkraft on August 16, 2012

Today I discovered the Microsoft Security Assessment Tool.  Although the tool came out nearly three years ago I still discovered it to be very useful in helping you to identify the security areas in which your organization or software development shop may be weakest.  It also provides a lot of links and useful recommendations to assist you with improving security.  Security should be near the front of every developer’s mind as the world moves toward cyberattacks that are increasingly more damaging to businesses and people.

Download the free tool here:

http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=12273

Expect to spend 30 to 90 minutes answering the questions.   You must first answer all the Business Profile Assessment questions, then you can move to the more interesting “Defense In Depth” assessment.  Follow that with generating a report of the results and if you’d like, posting your results to Microsoft and see how you compare to other organizations.

 

 

 

Posted in Free tools, I.T., Process | Leave a Comment »

Subversion 1.7 provides a big performance boost!

Posted by robkraft on March 4, 2012

Last week we decided to install the latest version of subversion just to see if it fixed any minor bugs or could speed up performance a little. I’m thrilled to report that it improved performance by a lot, not a little! We can pull data out of the repository three times faster than before, and checkins and all other features of subversion are much faster. Subversion got a major redesign and refactor in release 1.7 that greatly affects performance on Windows. If you are using subversion on Windows, go out and get this great upgrade as soon as possible.

We use the subversion from Collabnet, and they provide a video and other great information free about the changes in subversion that have made it so much better.

Another important action item to take away from this is that all software development shops should examine upgrading all of their tools regularly to the latest release and service pack. Software developers know these upgrades can fix bugs, fix security flaws, offer new features, and, as in this case, provide large performance improvements. Make a list of all of your tools and then make a task to consider upgrading each tool at least once a year.

A big thank you to the team that contributes to subversion at Collabnet!
http://www.open.collab.net/downloads/subversion/

Posted in Coding, Dev Environment, Process | Leave a Comment »

Get started using FxCop against your nightly build without first resolving all the violations

Posted by robkraft on February 13, 2012

FxCop is one of the best known free static code analysis tools available for .Net software. Version 10.0 of FxCop was published in June of 2010 and is available for download here (http://www.microsoft.com/download/en/details.aspx?id=6544). FxCop is easy to install and use to analysze your code for potential problems, but I think the real value of FxCop is when you run it every night against your nightly build and receive a report via email of new violations recently introduced in your code. I have long believed this to be the best use and value of FxCop, but only recently implemented it. And the reason I only recently implemented nightly scans of our code using FxCop is because when I run the tool I get a report with thousands of violations. Ouch! Are all these violations things I really need to fix? The answer for me, and likely the answer for you, is no. However, a lot of the violations should be fixed.

I think of the violations in these groups:

  • Violations like misspelled words that are not misspelled, but just need to be added to a custom dictionary so that they are no longer flagged as violations.
  • Violations that could, and would, cause problems if that bit of code was ever used in ways others than intended, but I know that won’t happen.
  • Violations that would be problems if the software runs in other countries using different languages, different date/time formatting, etc.
  • Violations that really are problems we should fix.
  • Violations that represent waste such as methods that are never called.

So, as I stated we have thousands of violations across all of our code, but I didn’t want to implement the nightly FxCop scan until all of those were resolved. If you look at the categories I used for the problems, many of the violations are not worth our time to resolve because we have more valuable work to do right now. So how could we begin using FxCop regularly, without fixing all the problems? We did it by disabling rule checking for all the rules that were currently failing. Then we set up the automated job to run FxCop every night against our compiled assemblies. Doing it this way gives us a starting point, and it keeps us from violating many of the rules defined in FxCop. Our next step is to begin turning on rules and fixing them one by one. Of course we will start with the important or easy violations and work our way down the list until we have them all re-enabled.

From a high-level perspective, you have to perform just two steps:

  1. Build your FxCop project files that analyze your assemblies and ignore failing rules.
  2. Build your nightly batch job to run against all the assemblies and email results to you.

Some tips before you begin:

  • Tip #1 – Use the same directory path for the DLLs you analyze with FxCop on both your developer PCs and your build machine. FxCop stores the path to the DLLs you select in the .FxCop configuration file. If the path is the same between your developer PC and the build PC, then you can easily configure FxCop on your PC and it will run without modification on the build computer.
  • Tip #2 – Do not plan to use one .FxCop project to scan all your assemblies unless you have just 10 or 20 assemblies. At some point, .FxCop fails to work when too many assemblies are loaded.
  • Tip #3 – FxCop does not work on Silverlight projects.

Here are the steps I recommend:

  • Step #1 – Add all the assemblies for your logical group into the .FxCop project and analyze them. Logically group the DLLs and Executables you analyze.
  • Step #2 – Copy the CustomDictionary.xml file provided by FxCop into the folder you with your FxCop projects, modify the CustomDictionary to recognize some of your acronyms and names as valid – not violations.
  • Step #3 – Sort the results of the analysis by Rule, then from the Rules tab go through and deselect all the rules that you have violations for. Save the .FxCop project file.
  • Step #4-n – Use the .FxCop file you saved as the starting point for the next group of assemblies you desire to analyze (save it with a different name of course). Exclude all the targets from the first analysis and add your new set of targets.
  • Step #5 – We use nant in our nightly processes, and nant includes a task specifically for running FxCop. Here is how we do it. Create a batch file to run nant:
cd\
cd\dev\fxcop
REM This next line appends the FxCop directory to the existing path so we can run FxCop from any folder
PATH %path%;c:\program files\microsoft fxcop 10.0
C:\WINDOWS\system32\nant.bat fxcop -logger:NAnt.Core.MailLogger
  • Step #6 – Configure an nant .build file for executing FxCop. Ours looks something like this:
  • <?xml version="1.0" encoding="utf-8" ?>
    <project name="FXCOP" default="build"  xmlns="http://nant.sourceforge.net/release/0.86-beta1/nant.xsd">
    	<property name="MailLogger.mailhost" value="smtp.ourdomain.com" />
    	<property name="MailLogger.from" value="build@ourdomain.com" />
    	<property name="MailLogger.success.notify" value="true" />
    	<property name="MailLogger.success.to" value="developers@ourdomain.com" />
    	<property name="MailLogger.failure.to" value="developers@ourdomain.com" />
    	<property name="MailLogger.success.subject" value="FXCop Violations" />
    	<property name="MailLogger.failure.subject" value="FXCop Build Failure" />
    	<property name="MailLogger.success.attachments" value="MailLogger.success.files" />
    
    	<fileset id="MailLogger.success.files">
    		<include name="ProjectGroup1.html" />
    		<include name="ProjectGroup2.html" />
    	</fileset>
    
    	<!-- this loads any required dependencies that nAnt needs to do all of our tasks-->
    	<loadtasks assembly="C:\Program Files\nAnt\Contrib\bin\NAnt.Contrib.Tasks.dll" />
    <target name="fxcop">
    	<fxcop analysisReportFilename="ProjectGroup1.html" applyOutXsl="true" includeSummaryReport="true" projectFile="MinimalRulesGroup1.FxCop"/>
    	<fxcop analysisReportFilename="ProjectGroup2.html" applyOutXsl="true" includeSummaryReport="true" projectFile="MinimalRulesGroup2.FxCop"/>
    </target>
    </project>
    
  • Step #7 – Over time, enable rules in FxCop and resolve the violations that arise. Eventually, hopefully, you will be able to get all of your code to pass all FxCop rules.

On a final note, I’d like to make sure you are aware that you can also write your own FxCop rules to ensure the code your team is written follows your shop standards.

Posted in CodeProject, Coding, Dev Environment, Process | 1 Comment »

Use Root Cause Analysis for Defect Prevention in your Software Development Process

Posted by robkraft on February 5, 2012

Recently I decided to review all the bugs we fixed in our last release to determine if they originated from a few common causes.  In software development, there is not a lot of data that can be mined that may provide insight into actions to take to improve software development, but I thought this might be one.  We fixed about 40 bugs in the last release, and I can tell you right now that my analysis has not lead to any significant changes in the way we develop software.  This is not because we didn’t find areas that could benefit from improvement, it is just because, in our case, attempting to make improvements to reduce the source of the bugs is not currently more valuable than other tasks we can be working on.

Of the bugs we found, I considering them to be “coding flaws”,  “configuration flaws”, “design flaws”, “process flaws”, or “requirements flaws”.  I realize that my technique is crude and I have very little data to draw definite conclusions from, but this is an exercise that anyone on a project team can perform if you are tracking the defects fixed in a release.  You may find patterns that help you identify aspects of software development most in need of improvement.  An excellent paper based on a lot of software development data at NASA is online here: http://www.hpl.hp.com/hpjournal/96aug/aug96a2.pdf.  The goal of the study in the paper, and my goal is well, was to use this “root cause analysis” to determine ways to prevent and reduce defects in the software.  Given that excellent article provides much more detail than I enter into here, I recommend that you read it if you are really interested in improving your development processes through root cause analysis of your bugs.

I will add here, a list of reasons that I came up with for the sources of bugs in our code:

18% – Coding Flaws.  The programmer did not think the issue was a bug, or the programmer did not test the issue thoroughly, or the programmer felt it was insignificant and lacked the time to address the problem.

39% – Coding Flaws.  The programmer should have noticed the problem.  But perhaps senior developers or business analysts should have provided a junior developer more training and input so that the junior developer would have recognized this as a problem.

3% – Coding Flaws.  The bug is a bug in 3rd party software we use (in this case Microsoft), that we need to wait for them to fix, or rewrite the feature.

5% – Coding Flaws.  The programmer should have just provided an error message to the user that is easier to understand.  Perhaps the message provided should always be determined by a business analyst.

5% – Coding Flaws.  A known bug was released.  The bug had minor impact and the cost to fixing it was high, as was the cost of delaying the release.

8% – Configuration Flaws.  We use code generation for much of our code.  The business analysts made mistakes that led to improperly generated code.

5% – Design Flaws.  The design did not allow for a feature to work as it needed to in some situations.

8% – Process Flaws.  We don’t test 100% of the User Interface possibilities in our application because the time required makes it impractical.  However, such testing is unnecessary for most of the code as long as we follow our processes for development.  In a few cases, we failed to follow the correct processes.

3% – Requirements Flaws.  The developer was not aware that the feature was desired.

3% – Requirements Flaws.  The requirements provided to the developer were incorrect.

3% – Requirements Flaws.  We removed a feature from the software that we thought was not being used, but it was and we had to re-add it.

We also improved performance in the last release.  Another way to phrase that is that we fixed performance problems.  It is difficult to decide if improving performance is fixing a bug or not when it doesn’t violate a clear service level agreement.  A code change is not always objectively a bug fix or an enhancement.  If the client believes it was in the original specification, the client will say the missing feature is a bug; but if the developer thinks the feature was not in the specification, then the developer will consider the addition of it in the next release to be an enhancement.  Just part of the joys of software development.

Posted in Code Design, CodeProject, Process, Project Management | 10 Comments »

Send SQL Server query results as nicely formatted emails in Outlook

Posted by robkraft on February 4, 2012

I find myself sending the results of a lot of SQL queries I run in SQL Server Management Studio via e-mail to co-workers.  The problem is, that the email content often looks terrible when I paste it into outlook from SQL Server Management Studio.

However, I have found that I can make it look better using the following steps.

  1. Run the query and let the output results go to grid (the default for most people).
  2. Right-click in the results and select ‘Copy with Headers’.
  3. Open Excel and paste the results.
  4. Click in the upper left corner of the Excel grid to select all rows and columns.
  5. Double-click between any two columns (A and B) to resize the widths of all columns.
  6. Select the “Center Text” alignment so that all column values are centered, rather than the default left-justified.
  7. Now copy all the contents of the Excel spreadsheet and paste the results into the body of your email in Outlook.

I find the results usually look pretty decent.  Can you provide me a way to do this that is easier?

Posted in Process, SQL Server | Leave a Comment »