Rob Kraft's Software Development Blog

Software Development Insights

How to Emulate Azure Function Application Settings Read Via Environment.GetEnvironmentVariable on Local Machine

Posted by robkraft on March 25, 2020

I borrowed a bit of code from an existing .Net Framework app that I had used to write an Azure function years ago.  But my new app was based on .Net Core and the code to retrieve an Environment variable (Environment.GetEnvironmentVariable) did not pull a value out of my .json file.

It took me a little googling around to figure out how to make it work so I figured I might as well share my findings in a blog post.  It will probably help “Future Rob” fix the problem more quickly if no one else.

If you have code like the following that pulls values from the “Application Settings” in an Azure function:

data[“refresh_token”] = Environment.GetEnvironmentVariable(“refresh_token”);
data[“grant_type”] = “refresh_token”;
data[“client_id”] = Environment.GetEnvironmentVariable(“client_id”);
data[“client_secret”] = Environment.GetEnvironmentVariable(“client_secret”);

but you want to run that code on your local machine, you just need to put your values inside the “Values” node of local.settings.json as shown here:


{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"client_id": "xxx",
"client_secret": "yyy",
"refresh_token": "zzz"
}
}

 

Posted in Coding | Leave a Comment »

How To Add Azure “Storage Emulator” to an Existing Azure Function Running on your Local Machine

Posted by robkraft on March 24, 2020

I created a new Azure Function App on my local machine in Visual Studio 2019 today.  One of the wizard steps gave me the option to use “Storage Emulator” for storage account and I decided to choose no storage.

StorageEmulator

That was a mistake because I also set my Azure Function App up to run based on a Timer Trigger, and in order to run it locally I need that Storage Emulator.

So I created another Visual Studio Azure Function App and chose the Storage Emulator so I could see what it generated differently, and to my surprise there were no additional dependencies, just one value that needs to be added to local.settings.json to make it work.

“Values”: {
“AzureWebJobsStorage”: “UseDevelopmentStorage=true”,

Once I replaced the empty string value for AzureWebJobsStorage with UseDevelopmentStorage=true, the app ran locally with no problems.

StorageEM2

Posted in Coding | Leave a Comment »

How To Fix Scaling Problem for High DPI PC Running Kali Linux

Posted by robkraft on March 11, 2020

I couldn’t find any posts on the web telling me the best way to fix this problem so I am sharing my solution here.

The problem I experienced is that I downloaded Kali Linux to run in VMPlayer on my High DPI Windows 10 machine but the guest font and screen is too small and hard to read to be really usable.  VMPlayer doesn’t have any settings to fix this, but you can change the VMPlayer.exe settings in the Windows host OS to fix it.

Find your VMPlayer.exe, probably in a folder like C:\Program Files (x86)\VMware\VMware Player.  Right-click on it, select Properties, Compatibility Settings.  Then click on the “Change high DPI settings”.

I checked the checkbox labeled “Use this setting to fix scaling problems for this program instead of the one in Settings”.

I also checked “Override high DPI Scaling behavior.  Scaling performed by System”.

When I restarted VMPlayer and opened Kali Linux, the UI was much more functional and usable, once I opened it full screen.

Posted in I.T. | Leave a Comment »

Free Tool from F-Secure To Check If Your Password Was Compromised

Posted by robkraft on March 11, 2020

F-Secure has an easy online tool you can use to find out what sites leaked your email address, password, and other information.

The URL is: https://www.f-secure.com/en/home/free-tools/identity-theft-checker

F-Secure is a site I trust and recommend.  Just enter your email address and within minutes you will receive an email with information about each breach your email address was part of.

F-Secure has other tools at https://www.f-secure.com/en/home/products#free such as a tool to show your IP address, a tool to check your router for flaws, and an online virus scanner for your PC.

Posted in Home Tech, I.T. | Leave a Comment »

My Favorite New Interview Question To Ask a Potential Employer

Posted by robkraft on March 5, 2020

I am not in a rush to find a new job and plan to look carefully at each potential employer to see if I think the culture of that company is a good fit for me.  But how can we get a good grasp of the culture without working at the company for a while?  I have been in interviews where a candidate asks, “What is the culture like here?”, but responses to the question are ambiguous because it is difficult to define culture and to accurately guess what the questioner is concerned about when they ask about culture.

The question I like to ask to assess culture is more direct.  That question is:

“Have any of the current managers in the company ever yelled at any of the employees?”

I know what you are thinking.  Wow, that is a bold question!  But it partially gets to the heart of the issue.  I want to know that the managers are emotionally mature.  I believe that an emotionally mature manager will never yell at employees.  Perhaps I would expect yelling if I was joining the military.  And perhaps I should expect it in some industries; but I am a software developer and I do not expect it in the software development industry.  If yelling happens, and especially if it is common within the company, then I already know that is not a place where I want to work.

I believe that it is a little rude to spring this question upon my possible employer during the interview, which is why I like to submit this question to the potential employer before I meet with them in person.  In fact, I like to submit all of my questions to the employer in advance to give them time to prepare answers to those questions for me.

Do you have a better question to ask to determine if the company culture is right for you?

Posted in Culture | Leave a Comment »

C# .Net SQL Injection Detection – Especially for Legacy .Net Code

Posted by robkraft on March 4, 2020

When you have an existing .Net code base full of SQL statements, and you want to reduce the chance that there are SQL injection risks in the code, you may decide to perform a review of every SQL statement in order to confirm that they are all coded correctly; or you may hire another company to do this for you. But one problem with this approach is that the code is only “SQL Injection Free” from the moment the review is completed until people start modifying and adding to the code again.

What you should strive for is a way to make sure every past and future SQL statement gets tested for SQL Injection risk before it runs.  That is what this sample code provides you.  If you follow the patterns described here, I believe you can significantly reduce the risk that your code has bugs leading to SQL Injection and it will stay that way going forward.

Using the Decorator Pattern to Provide a Place To Add SQL Injection Detection

The primary technique I recommend in this article for adding SQL Injection detection into your application is to stop using the .ExecuteReader and .ExecuteNonQuery methods.  Instead, use the Decorator pattern to create your own method to be called in place of those two, and that method will include code to do some SQL Injection detection.

Replace:

SqlDataReader reader = command.ExecuteReader();

With 

SqlDataReader reader = command.ExecuteSafeReader(); //provided in sample code

The sample code provided behaves like the Proxy pattern in that it will make the actual call to the database after finding no SQL Injection risk.  The benefit of this approach is that you can then regularly scan your entire code base for the use of .ExecuteReader and .ExecuteNonQuery knowing that there should be no cases of those methods, other than the exception cases you expect.  Thus you can be sure that the majority of your code is running through your SQL Injection detector.

Another benefit of using the Decorator pattern to implement SQL Injection Detection is that you can also easily add other features such as:

  • Logging every SQL that is executed
  • Logging and blocking every SQL that is a SQL Injection risk
  • Altering every SQL on the fly.  One scenario where this could be helpful is that if you renamed a table in the database but had a lot of SQL that needed to change.  You could possibly add a find/replace to every SQL on the fly to change the table name, allowing you more time to find and correct all stored SQL fragments with the old table name.
	public static SqlDataReader ExecuteSafeReader(this SqlCommand sqlcommand)
	{
		if (!sqlcommand.CommandType.Equals(CommandType.StoredProcedure))
		{
			var sql = sqlcommand.CommandText;
			//Options: You could Add logging of the SQL here to track every query ran
			//Options: You could edit SQL - for example if you had renamed a table in the database
			if (!ValidateSQL(sql, SelectRegex))
				return null;
		}

		return sqlcommand.ExecuteReader();
	}

The SQL Injection Detection Code

Warning!  This does not detect all forms of SQL Injection, but it will detect most of them.  Here is what causes the class to throw an exception:

  • Finding a single apostrophe (single quote) that does not have a matching single apostrophe (single quote)
  • Finding double quotes that do not have a matching double quote.  This is only needed if the SQL Server has SET QUOTED_IDENTIFIER OFF.  However, you may also want to use this if your database is MySQL or some other DBMS.
  • Finding a comment within the SQL
  • Finding an ASCII value great than 127
  • Finding a semicolon
  • After extracting the strings and comments, finding any of a specific configurable list of keywords in a SELECT statement such as DELETE, SYSOBJECTS, TRUNCATE, DROP, XP_CMDSHELL

The code is written to be easy to change if you don’t want to enforce any of the rules above, or if you need to add similar rules because you have a special scenario or a DBMS besides SQL Server.

The code uses the regex [^\u0000-\u007F] to reject the SQL if it contains any non-ASCII characters.  This works for the applications I have written, but may need alteration for non American English language support.

The code also uses regexes to check SQL statements for undesirable keywords.  One regex is for SELECT statements and therefore blocks them if they contain INSERT, UPDATE, or DELETE.  Other keywords that may indicate a SQL Injection attempt are also rejected and that list includes waitfor, xp_cmdshell, and information_schema.  Note that I also include UNION in the list; so if you use the UNION keyword you will need to remove that from the list.  UNION is frequently used by hackers attempting to perform SQL Injection.

private static void LoadFromConfig()
{

	_asciiPattern = "[^\u0000-\u007F]";
	_selectpattern = @"\b(union|information_schema|insert|update|delete|truncate|drop|reconfigure|sysobjects|waitfor|xp_cmdshell)\b|(;)";
	_modifypattern = @"\b(union|information_schema|truncate|drop|reconfigure|sysobjects|waitfor|xp_cmdshell)\b|(;)";
	_rejectIfCommentFound = true;
	_commentTagSets = new string[2, 2] { { "--", "" }, { "/*", "*/" } };
}

SQL Server supports two techniques to comment out SQL code in a SQL Statement, two dashes, and enclosing the comment in /* */.  Since it is unlikely that developers write SQL to include comments, my default choice is to reject any SQL containing those values.

Exactly How Is The SQL Injection Detected?

There are basically three steps in the SQL Injection detection process.

First, the code checks for any ASCII values above 127 and rejects the SQL if one is found.

Second, the code removes all the code withing strings and comments.  So an SQL that starts out looking like this:

select * from table where x = ‘ss”d’ and r = ‘asdf’ /* test */ DROP TABLE NAME1 order by 5

becomes this:

select * from table where x = and r = t DROP TABLE NAME1 order by 5

Third, the code looks for keywords, like “DROP” and “XP_CMDSHELL”, in the revised SQL that are on the naughty list.  If any of those keywords are found, the SQL is rejected.

Formatting Methods included in the SQLExtensions Class

The SQLExtensions class provides additional methods to help your coders reduce the risk of SQL Injection.  These methods help coders format variables in SQL when doing so with a parameter is not an option.  The most useful of these methods is FormatStringForSQL and it could be used as shown here to enclose a string in SQL quotes as well as replace any single quotes contained within the value with two single quotes.


string sql = "select * from customers where firstname like " + nameValue.FormatStringForSQL();

Another advantage of using a method like this is that it makes it easy for you to change how you handle the formatting of strings everywhere within your code if you discover that you need to make a change.  For example, perhaps you decide to move your application from SQL Server to MySQL and therefore that you also need to replace double quotes in addition to single quotes.  You could make the change within this method instead of reviewing your entire code base to make the change one by one for each SQL.

Custom .Net Exception Class

I also provided a custom Exception primarily to show how easy it is to implement custom exceptions and because I think it is useful for this extension class.  This provides you more flexibility for handling exceptions.  You can catch and handle the exceptions raised specifically due to SQL Injection risk different than exceptions thrown by the underlying ADO.NET code returned from the database.


[Serializable]
public class SQLFormattingException : Exception
{
	public SQLFormattingException() {}

	public SQLFormattingException(string message): base(message) {}
}

The Rules For Detecting SQL Injection are Configurable

I made enabling/disabling configuration of the SQL Injection detections easy to change so that you could import those rules at runtime if desired so that different applications could have different rules.  Perhaps one of your applications needs to allow semicolons in SQL but the others don’t.  It is a good practice to implement the most stringent rules you can everywhere you can.  Don’t implement weak SQL Injection detection rules everywhere because a single place in your code needs weaker rules.  The rules are “Lazy Loaded” when needed, then cached, to support the ability to change them while an application is running by calling the InvalidateCache method provided.

Below is an example of one of the rules.  You can configure your code to reject the SQL if it contains SQL Server comments.


#region RejectComments Flag
private static bool? _rejectIfCommentFound = null;
public static bool RejectIfCommentFound
{
	get
	{
		if (_rejectIfCommentFound == null)
		{
			LoadFromConfig();
		}
		return (bool)_rejectIfCommentFound;
	}
}
#endregion

Steps To Implement and Use This Code

I suggest you take the following steps to implement this class:

  1. Get the SQLExtensions.cs class file into a project in your code base. You will also need the CustomExceptions.cs class file.  The program.cs just contains a sample usage and there is also a UnitTest1.cs class.
  2. Comment out all the lines in ReallyValidateSQL except for the “return true”
  3. Do a find and replace across your entire code base to replace ExecuteReader with ExecuteSafeReader
  4. Compile and test.  Your app should still work exactly the same at this point.
  5. Review the Customizable Validation Properties and decided which ones you want to implement, then uncomment the lines you commented out in ReallyValidateSQL
  6. Decide if you need to and want to replace dynamically constructed SQL in your application with any of the four FormatSQL… extension methods provided.
  7. Provide me feedback

MIT FREE TO USE LICENSE

This code has an MIT license which means you can use this code in commercial products for free!

A link to the source code example is here: https://github.com/RobKraft/SQLInjectionDetection

Posted in Code Design, CodeProject, Coding, Security | 2 Comments »

Asp.Net Core – HTTP Error 500.0 – ANCM In-Process Handler Load Failure

Posted by robkraft on October 30, 2019

I spent a few hours trying to fix this problem today.  What I thought was strange was that one web site loaded and worked fine, and another web site, on the same server using the exact same files did not work.

Our fix turned out to be somewhat simple.  You can only have one Asp.Net Core module running in process per IIS App Pool.  We have long been sharing an app pool with several web sites.  I guess as we migrate to Asp.Net Core running on IIS, that we will need to create a separate App Pool for each site.

I figured this out by reviewing the Event Viewer Application logs.

“Only one inprocess application is allowed per IIS application pool. Please assign the application ‘/LM/W3SVC/1/ROOT/Nightly/SQL2012/LucityDocumentServerCore’ to a different IIS application pool.”

Eventi ID 1008

 

  • HTTP Error 500.0 – ANCM In-Process Handler Load Failure
  • The specified version of Microsoft.NetCore.App or Microsoft.AspNetCore.App was not found.
  • The in process request handler, Microsoft.AspNetCore.Server.IIS, was not referenced in the application.
  • ANCM could not find dotnet.
  • failed to start application iis aspnetcore module v2 0x80004005

Posted in I.T., Web Sites | Leave a Comment »

SQL Server’s sp_executesql Does Not Protect You from SQL Injection

Posted by robkraft on August 18, 2019

Many coders of SQL have learned we can dynamically construct SQL statements inside of stored procedures and then execute the constructed SQL.  In Microsoft’s SQL Server product there are two commands we can choose for running the constructed SQL:

  • EXEC (EXEC is an alias for EXECUTE, both do the same thing).
  • sp_executesql.

We SQL Server “experts” often advise coders to use sp_executesql instead of EXEC when running dynamically constructed SQL statements to reduce the risk of SQL Injection, and this is good advice.  But it is not the use of sp_executesql that prevents SQL injection, it is the use of parameters with sp_executesql that helps protect against SQL Injection.  You can still construct SQL dynamically and run that SQL using sp_executesql and be affected by a SQL Injection attack.

If you use parameters to substitute all the values in the SQL and then use sp_executesql you have probably eliminated the SQL Injection risk; but as a developer this means you may be unable to dynamically construct the SQL you want to run.

When you use sp_executesql parameters correctly, you can only replace data values in your SQL statement with values from parameters, not parts of the SQL itself.  Thus we can do this to pass in a value for the UserName column:

declare @sql nvarchar(500)
declare @dynvalue nvarchar(50)
select @dynvalue=’testuser’
SET @sql = N’SELECT * FROM appusers WHERE UserName = @p1′;
EXEC sp_executesql @sql, N’@p1 nvarchar(50)’, @dynvalue

But the following code will return an error when trying to pass in the name of the table:

declare @sql nvarchar(500)
declare @dynvalue nvarchar(50)
select @dynvalue=’appusers’
SET @sql = N’SELECT * FROM @p1′;
EXEC sp_executesql @sql, N’@p1 nvarchar(50)’, @dynvalue

Msg 1087, Level 16, State 1, Line 1
Must declare the table variable “@p1”.

If you are dynamically constructing SQL, and you are changing parts of the SQL syntax other than the value of variables, you need to manually write the code yourself to test for the risk of SQL injection in those pieces of the SQL.  This is difficult to do and probably best handled by the application calling the stored procedure.  I recommend that the calling program do the following at a minimum before calling a stored procedure that dynamically constructs SQL:

  1. Validate the length of the parameter. Don’t allow input longer than the maximum length expected.  If the stored procedure allows a column to be passed in that is used for sorting in an ORDER BY clause, and all of your column names are less than or equal to 10 characters in length, then make sure that the length of the parameter passed in does not exceed 10 characters.
  2. Don’t allow a single single quote, make sure to replace a single single quote with two single quotes.
  3. Don’t allow other special characters or even commands such as a semicolon or the UNION keyword or two hyphens that represent a comment in SQL.
  4. Don’t allow ASCII values greater than 255.

That short list is not sufficient to prevent all SQL Injection attacks, but it will block a lot of them and gives you an idea of the challenge involved in preventing SQL Injection attacks from being effective.

If you would like to see for yourself how the EXEC and sp_executesql statements behave I have provided a script you can use to get started with.  Related to this article, the most important query to understand is the last one because it shows a case of SQL injection even though the dynamically generated SQL is ran using sp_executesql.

–1. Create tables and add rows
DROP TABLE InjectionExample
GO
DROP TABLE Users
GO
CREATE TABLE InjectionExample ( MyData varchar (500) NULL)
GO
INSERT INTO InjectionExample VALUES(‘the expecteddata exists’), (‘data only returned via sql injection’)
GO
CREATE TABLE Users( username varchar(50) NULL,[password] varchar(50) NULL)
go
INSERT INTO Users VALUES (‘user1′,’password1’), (‘user2′,’password2’), (‘user3′,’password3’)
GO
–2. Run a test using EXEC with data the programmer expects
declare @sql nvarchar(500)
declare @p1 nvarchar(50)
select @p1 = ‘expecteddata’
select @sql = ‘SELECT * FROM InjectionExample WHERE MyData LIKE ”%’ + @p1 + ‘%”’
exec (@sql)–returns 1 row as expected
GO

–3. Run a test using EXEC with data the hacker used for sql injection
declare @sql nvarchar(500)
declare @p1 nvarchar(50)
select @p1 = ”’ or 1 = 1–‘
select @sql = ‘SELECT * FROM InjectionExample WHERE MyData LIKE ”%’ + @p1 + ‘%”’
exec (@sql)–returns all rows – vulnerable to sql injection
GO

–4. Run a test using sp_executeSQL to prevent this SQL Injection
declare @sql nvarchar(500)
declare @p1 nvarchar(50)
select @p1 = ‘expecteddata’
select @sql = N’select * from InjectionExample WHERE MyData LIKE ”%” + @param1 + ”%”’
exec sp_executesql @sql, N’@param1 varchar(50)’, @p1
GO

–5. Run a test using sp_executeSQL to prevent this SQL Injection – hacker data returns no results
declare @sql nvarchar(500)
declare @p1 nvarchar(50)
declare @pOrd nvarchar(50)
select @p1 = ”’ or 1 = 1–‘
set @pOrd = ‘MyData’
select @sql = N’select * from InjectionExample WHERE MyData LIKE ”%” + @param1 + ”%” order by ‘ + @pOrd
exec sp_executesql @sql, N’@param1 varchar(50)’, @p1
GO

–6. But sp_executesql does not protect against all sql injection!
–In this case, sql is injected into the @pOrd variable to pull data from another table
declare @sql nvarchar(500)
declare @p1 nvarchar(50)
declare @pOrd nvarchar(50)
set @p1 = ‘expecteddata’
set @pOrd = ‘MyData; SELECT * FROM Users’
select @sql = ‘select * from InjectionExample WHERE MyData LIKE ”%” + @param1 + ”%” order by ‘ +@pOrd
exec sp_executesql @sql, N’@param1 nvarchar(50)’, @p1

 

 

Posted in CodeProject, Security, SQL Server | Leave a Comment »

What Makes A Software Programmer a Professional?

Posted by robkraft on June 16, 2019

Many people write code, but not everyone that codes considers themselves to be a professional programmer.  What does it take to be a professional?  This article lists the practices you undertake when you are a software development pro.  From my experience, many of these practices are not taught in coding schools, but they are essential to delivering quality software to customers.

Before covering the practices, let’s first briefly consider different classes of those who write code:

The Tinkerer
A tinkerer is a person that writes a few lines of code here and there, perhaps a macro in Excel, perhaps connecting services in IFTTT or Microsoft Flow, or perhaps a script to run in a database. A tinkerer may also use a language like basic or javascript in order to create a program or web site for their own personal use and enjoyment.

The Amateur

A programmer becomes an amateur instead of a tinkerer when the programmer starts writing software or web sites for others, especially when the programmer is compensated for their work.  Amateur programmers often create good software and write code well.  But to be considered professional, there are several practices a programmer will follow.

The Professional

The following lists represents what I believe are practices that every professional software developer will follow.  As in all things, there may be some cases where it makes sense that one or two of these practices is not performed by a professional; but I believe most would agree that professionals follow all of these practices most of the time.

  1. Use version Control
  2. Back up everything off site
  3. Track the changes, fixes, and enhancements in each release
  4. Keep the source code related to each deployed version that is in use
  5. Keep a copy of your software in escrow
  6. Use automated builds
  7. Schedule builds
  8. Write regression unit tests
  9. Use a bug tracking system
  10. Use a system to track tasks and features being developed
  11. Keep customers informed about the progress of the software development
  12. Keep third party software used updated regularly
  13. Understand the security risks
  14. Ensure proper compliance with industry standards such as PCI, HIPAA, SOX, and PII
  15. Educate yourself continuously
  16. Invest in your development tools
  17. Properly license development tools and software
  18. Write documentation about the software
  19. Keep a journal of your activity

1. Use Version Control

A version control system should be used by all professional software developers.  It is difficult to imagine a solution for which version control would not apply.  Today, most developers use GIT, but many also use Subversion (SVN).  The version control system used should have the following features:

  • Allow people to add comments explaining each check in, and track who checked it in
  • Allow people to view a history of changes checked in for each file
  • Allow people to revert to earlier versions of the software
  • Allow people to compare the code changes made across check ins

Zipping up the files that are part of each product release or check in, instead of using a version control system, is a sign of an amateur programmer.

2. Back Up Everything Off Site

A professional programmer will make sure that all of the code they write is backed up regularly, and that back up needs to be an offsite location to prevent loss of all of the source code in the event of a fire, flood, theft, or some other event.  Given the ubiquity of the Internet today this is usually easily achieved.  Simply using an online repository like github or bitbucket for version control almost fully meets this practice in most cases.  Along with the backups of the source code, a professional will make sure that that the scripts and commands for the tools and processes used to build the software is also backed up remotely.  A professional programmer should be able to start from nothing, acquire a new computer, and reconstruct everything needed to continuing developing the software by recovering all of the software from the off site backup.

3. Track the Changes, Fixes, and Enhancements in Each Release

A software professional provides more than software to their clients, they also provide a list of all the new features included in the latest version.  Of course there should also be a comprehensive list of the contents of the initial version.  In addition, a list of bugs fixed and other changes such as noticeable enhancements to performance and security should be provided.  A software professional tracks the reason behind every code change made during a release.  This is often helpful when clients are looking for a feature or fix because you can tell them they need to upgrade to a specific version to get it.

4. Keep the Source Code Related to Each Deployed Version That is in Use.

Modern version control software allows us to see how the code looked at any point in time.  A professional should be able to easily and quickly see how the code looked for any version of the software that any client currently has implemented.  This helps the developer resolve issues reported by clients more quickly, and to create a fix for the software.

5. Keep a Copy of Your Software in Escrow

No one likes to ponder the possibility of their own demise, but as a software professional you should consider this possibility and take steps to ensure that your clients can continue to use your software if something unfortunate happens to you.  Make sure that someone besides yourself can obtain access to your source code and all the artifacts and processes needed to build and support it if the need arises.

6. Use Automated Builds

Building software takes us from the source code to the final output that is deployed to a client or to a production system.  While builds often start out simple they can evolve to become more complicated.  For this reason, a professional developer will set up an automated build process that compiles and combines all the source code to a package ready to deploy.  Minor tasks that can be automated so that they are not forgotten include flagging the code as a release build so that the compiler will optimize the output, or minimize it, or obfuscate it as needed.  Automated builds often update version numbers and also often perform several steps in a specific sequence.  Build processes can be automated initially with simple batch/command files; but many professional use tools specifically designed for building software products.

7. Schedule Builds

Scheduling a build is just the next step, and hopefully a minor step, after a professional has created an automated build.  Many developers schedule a build to automatically run once or twice a day.  This is especially advantageous when multiple developers are contributing to the code.  Some developers even configure the version control system to start a build every time code is checked in.  Frequent builds help developers identify bugs more quickly if an artifact necessary for the build was excluded from their repository checkins, or if code they checked in adversely affected code in another part of the system.

8. Write Regression Unit Tests

I’ll admit that there may be a few cases where it does not make sense to write unit tests for your software.  But I believe that in almost all cases, even for old languages like Cobol and Visual Basic, a developer will write unit tests to validate important logic in the software runs correctly and is not inadvertently altered by a new enhancement or related bug fix.  Getting a unit test framework up and running takes a few hours or even a few days for someone not familiar with unit tests, but once you have it, you find that the tests give you a lot of freedom to make changes and the peace of mind that the changes you make are not breaking existing logic that your customers depend upon.

9. Use a Bug Tracking System

Let’s face it, almost all software has bugs, or things that are not working quite as the user desires.  Software developers need to track when those bugs were discovered and who discovered them so that they can make a careful fix and provide a patch or fix to the affected systems and users.  A bug tracking system can help all customers and users of your software become aware of the bug and sometimes the way to workaround a bug until a fix is applied.  It also can let your customers know that they just need to upgrade to your latest version to obtain the fix for the bug.  In addition, it helps you keep track of when and where you made the fix so that you can manage related issues with the fix, follow up issues, or just to reminder yourself that you already fixed this bug.

10. Use a System to Track Tasks and Features Being Developed

Software professionals understand that delivering quality software includes keeping track of a lot of details.  A system and place to keep track of all of the things that need to be done is very valuable.  Your system could be something as simple as a Trello board.  Ideally you will have a list of everything yet to be completed and everything yet to be done.  Most tasks boards use at least three columns: To Do, Doing, and Done.  Even when you think a task is done you may decide it is not really done if you have not written the documentation related to the feature, or altered your install package to include a dependency for the feature.  The system also helps you remember the status of items and share the status, or share the work to be done, with others.

11. Keep Customers Informed About the Progress of the Software Development

In most software development projects, frequent interaction with the end user, aka ‘the customer’ greatly increases the chance that the software you create meets their needs.  Frequent communication also helps you manage your customer’s expectations about when features will be delivered to them and how those features will behave.  Although the waterfall method of software development describes collecting all of the requirements up front, it is extremely rare that you really can gather all of the details as well as you can if you keep the customer involved as you develop and can show them the product as it evolves.  Professional software developers will not leave their customers in the dark for weeks or months before giving them an update about the progress of the software.

12. Keep third party software used updated regularly

Most developers rely on some third party software to decrease the time it takes to produce the software their clients want.  For web developers, this may include frameworks like Microsoft .Net, Angular, React, JQuery, and Java.  But security flaws and performance problems, and other bugs occasionally get discovered in these frameworks, therefore a software professional regularly updates to the latest version of the frameworks in order to obtain the security patches and fixes to improve the security and performance of the software they pass on to their client.

I recommend you keep a list of all the third party software used by your software and your software development processes, and that you review the list at least twice a year in order to update each dependency to the latest version.

13. Understand the security risks

Professional software developers understand that they generally have more knowledge of software development than the customers that have hired them to write code.  Thus they understand that writing secure code, code that can’t be easily abused by hackers, is their responsibility.  A software developer creating web applications probably needs to address more security risks than a developer writing embedded drivers for an IOT device, but each needs to assess the different ways the software is vulnerable to abuse and take steps to eliminate or mitigate those risks.

Although it may be impossible to guarantee that any software is immune to an attack, professional developers will take the time to learn and understand the vulnerabilities that could exist in their software, and then take the subsequent steps to reduce the risk that their software is vulnerable.  Protecting your software from security risks usually includes both static analysis tools and processes to reduce the introduction of security errors, but it primarily relies upon educating those writing the code.

OWASP (https://owasp.org) is a good resource for developers to learn about possible vulnerabilities and ways to mitigate those vulnerabilities.

14. Ensure Proper Compliance with Industry Standards such as PCI, HIPAA, SOX, and PII

Writing software that complies with industry regulations is also a responsibility of a software professional.  It is the responsibility of the customer asking for the software to tell the developer that they need the software to meet specific regulations such as HIPAA, GDPR, PCI, SOX, or PII.  But is the responsibility of the software professional to understand how those regulations affect the software and software development processes.  A customer may suggest to the developer what impact the regulation has on the code, but if you are a software professional, you will refer to the regulation directly and clarify your own interpretation of the document.

15. Educate Yourself Continuously 

Technology continually changes thus professionals will continually learn new tools, techniques, and software languages in order to improve their efficiency and write software that lasts longer.  Even developers that have been programming in the same language and environment for a decade may discover that there are new tools and processes that can help them write better code.  Perhaps need static code analysis tools can help you find bugs in your code, or perhaps you can learn to write better code by switching from a waterfall methodology to an Agile approach.  Developers writing Visual Basic 6 code may realize they can begin writing more modular code and use classes to facilitate a possible rewrite to Java that is coming, but is still years away.  If you aren’t occasionally doing some research to find ways to improve, I don’t think you can consider yourself to be a professional software developer.  (If you are reading this article, you probably are a professional or planning to be one!)

16. Invest in Your Development Tools

A good carpenter doesn’t use a cheap saw, and a good software developer doesn’t just use the cheapest tools and equipment.  Luckily for software developers, many really good tools are free to use.  Some tools do cost a little bit of money, but the productivity improvement gained by paying for them is often well worth the price.

Besides software tools, developers sitting at computers writing code day after day probably benefit from having good hardware.  A computer with a lot of RAM and a fast hard drive and Internet connection may spare you waiting minutes or even hours each day.  Multiple monitors, a comfortable keyboard, mouse, and chair can contribute to your ability to write code a little more effectively each day.  Take time to invest in the small things that can improve your productivity, even if it is just by a small amount.

17. Properly License Development Tools and Software

Customers expect people they hire to work ethically.  Well, perhaps unethical customers don’t expect ethical behavior or are willing to turn a blind eye, but if you declare yourself to be a professional software developer I believe you are also declaring that you do things ethically, and part of being ethical is paying for the resources you use to develop the software solution.  This means that you will pay the correct price and licensing fees for the tools you use for development.  It also means that you won’t use the free or community version of tools to develop professional solutions that clearly don’t qualify for the rules of use for that version of the development product.  Just as you are probably making a living writing software for your customers, those people and companies that created the tools you are using to enhance your productivity are trying to make a living by writing their software too.

18.  Write documentation about the software

Professionals write documentation about the software to assist future developers that may need to take over maintenance and enhancements of the code.  Sometimes, that future developer could be yourself.  System diagrams, flow diagrams, use case diagrams, and comments in code explaining the complicated bits all go a long way toward helping future developers maintain the software if you are not around to do so.

19.  Keep a journal of your activity

Professionals often keep notes of their activities during software development.  The notes can serve several purposes, but most often they benefit yourself.  Perhaps you record why you chose one approach over another.  Perhaps you list expected benefits or drawbacks of a decision.  Perhaps you keep track of how often you perform IT maintenance or how you fixed some problems.  You may keep track of interactions with others and a record of tasks and responsibilities.  Professionals also use these notes to help explain where they spent their time and to explain why development is behind schedule, for those times when that happens.

Summary

I wrote this for people that are new to software development, particularly those who have completed a program in writing software and hope to embark on a career as a professional.  Most of the items covered in the article are not covered by formal software education programs, but are an essential aspect of writing good quality software.  Individuals writing software for a living on their own probably want to implement all of these practices in order to give current and potential clients confidence in the quality and professionalism of their work.

Please let me know if you feel something important is missing from this list so that we may improve this article as a good reference for developers.

And please check back for a future article where I cover those practices that make a software development “team” a team of professionals.

Posted in CodeProject, Coding, Process, Project Management | 1 Comment »

A Great Collection of Free Tools for Web Developers – JSON, REST, and CSV – ExtendsClass.com

Posted by robkraft on June 15, 2019

Cyril has put together a great collection of free tools particularly for web app developers working with JSON, XML, Encoding/Decoding, REST APIs, SQL, and CSV files.

You can find them all on his website at: https://extendsclass.com/.

Extends Class provides several online tools:
– fiddles:
* Xpath tester (https://extendsclass.com/xpath-tester.html).
* SQLite fiddle (https://extendsclass.com/sqlite-browser.html). It is open source (https://github.com/cyrilbois/Web-GUI-for-SQLite).
* RegEx Tester (https://extendsclass.com/regex-tester.html), which helps to test and debug regex.
* JavaScript fiddle (https://extendsclass.com/javascript-fiddle.html).
* JSONPath Tester (https://extendsclass.com/jsonpath-tester.html).
– checkers:
* PHP code checker (https://extendsclass.com/php-tester.html).
* Python code checker (https://extendsclass.com/python-tester.html).
– API tools:
* REST client (https://extendsclass.com/rest-client-online.html).
* SOAP client (https://extendsclass.com/soap-client-online.html).
* Web service testing (https://extendsclass.com/web-service-tester.html). It is open source (https://github.com/cyrilbois/RestesterUI).
* Mock REST API (https://extendsclass.com/mock-rest-api.html).
– formatters & beautify:
* JSON formatter and validator (https://extendsclass.com/json-validator.html).
* XML formatter and validator (https://extendsclass.com/xml-formatter-online.html). It is open source (https://github.com/cyrilbois/Web-xml-formatter).
* SQL formatter (https://extendsclass.com/sql-formatter.html).
– Converters / Encoders:
* XML to JSON (https://extendsclass.com/xml-to-json.html).
* JSON to CSV (https://extendsclass.com/json-to-csv.html).
* CSV to Excel (https://extendsclass.com/csv-to-excel.html).
* Base64 image encoder / decoder (https://extendsclass.com/image-to-base64.html).
* Base64 encoder / decoder (https://extendsclass.com/base64-decode-encode.html).
– minifiers
* CSS minifier (https://extendsclass.com/css-minifier.html).
* HTML minifier (https://extendsclass.com/html-minifier.html).
* JavaScript minifier (https://extendsclass.com/javascript-minifier.html).
– diff:
* Text diff (https://extendsclass.com/text-compare.html).
* XML diff (https://extendsclass.com/xml-diff.html)
* JSON diff (https://extendsclass.com/json-diff.html)
* CSV diff (https://extendsclass.com/csv-diff.html)
– other:
* CSS Generator (https://extendsclass.com/css-generator.html).

Posted in Free tools, Resources and Tools | Leave a Comment »