How to benchmark C# code using BenchmarkDotNet

BenchmarkDotNet is a lightweight, open source, powerful .NET library that can transform your methods into benchmarks, track those methods, and then provide insights into the performance data captured. It is easy to write BenchmarkDotNet benchmarks and the results of the benchmarking process are user friendly as well.

You can take advantage of BenchmarkDotNet to benchmark both .NET Framework and .NET Core applications. In this article we’ll explore how we can work with BenchmarkDotNet in .NET Core. You can find BenchmarkDotNet on GitHub.

To work with the code examples provided in this article, you should have Visual Studio 2019 installed in your system. If you don’t already have a copy, you can download Visual Studio 2019 here. 

Create a console application project in Visual Studio

First off, let’s create a .NET Core console application project in Visual Studio. Assuming Visual Studio 2019 is installed in your system, follow the steps outlined below to create a new .NET Core console application project in Visual Studio.

1. Launch the Visual Studio IDE.
2. Click on “Create new project.”
3. In the “Create new project” window, select “Console App (.NET Core)” from the list of templates displayed.
4. Click Next.
5. In the “Configure your new project” window shown next, specify the name and location for the new project.
6. Click Create.

This will create a new .NET Core console application project in Visual Studio 2019.

Note that when you create the console application project, the resulting Program class (generated automatically in the Program.cs file) will look like this:

class Program
{
   static void Main(string[] args)
  {
      Console.WriteLine("Hello World!");
  }
}

We’ll use this project and Program class to work with BenchmarkDotNet in the subsequent sections of this article.

Install the BenchmarkDotNet NuGet package

To work with BenchmarkDotNet you must install the BenchmarkDotNet package. You can do this either via the NuGet Package Manager inside the Visual Studio 2019 IDE, or by executing the following command at the NuGet Package Manager Console:

Install-Package BenchmarkDotNet

Why benchmark code?

A benchmark is a measurement or a set of measurements related to the performance of a piece of code in an application. Benchmarking code is essential to understanding the performance metrics of the methods in your application. It is always a good approach to have the metrics at hand when you’re optimizing code. It is very important for us to know if the changes made in the code has improved or worsened the performance. Benchmarking also helps you to narrow in on the portions of the code in the application that need refactoring.

Steps for benchmarking code using BenchmarkDotNet

To run BenchmarkDotNet in your .NET Framework or .NET Core application you must follow these steps:

1. Add the necessary NuGet package
2. Add Benchmark attributes to your methods
3. Create a BenchmarkRunner instance
4. Run the application in Release mode

Create a benchmarking class in .NET Core

Open the Program.cs file and write the following code in there.

  [MemoryDiagnoser]
   public class MemoryBenchmarkerDemo
    {
        int NumberOfItems = 100000;
        [Benchmark]
        public string ConcatStringsUsingStringBuilder()
        {
            var sb = new StringBuilder();
            for (int i = 0; i < NumberOfItems; i++)
            {
                sb.Append("Hello World!" + i);
            }
            return sb.ToString();
        }
        [Benchmark]
        public string ConcatStringsUsingGenericList()
        {
            var list = new List<string>(NumberOfItems);
            for (int i = 0; i < NumberOfItems; i++)
            {
                list.Add("Hello World!" + i);
            }
            return list.ToString();
        }
    }

The above program illustrates how you can write methods for benchmarking. Note the usage of the Benchmark attribute on top of each of the methods that are to be benchmarked.

In the Main method of the Program.cs file you must specify the initial starting point — the BenchmarkRunner class. This is a way of informing BenchmarkDotNet to run benchmarks on the class specified. So, replace the default code of the Main method in the Program.cs file using the following code snippet.

static void Main(string[] args)
{
   var summary = BenchmarkRunner.Run<MemoryBenchmarkerDemo>();
}

Run the benchmark in your .NET Core application

If you run the application in debug mode, here’s the error message you’ll see:

IDG

When benchmarking you should always ensure that you run your project in release mode. The reason is that during compilation the code is optimized differently for both debug and release modes. The C# compiler does a few optimizations in release mode that are not available in debug mode.

Hence you should run your project in the release mode only. To run benchmarking, specify the following command at the Visual Studio command prompt.

dotnet run -p BenchmarkDotNetDemo.csproj -c Release

For best results, you should make sure that all applications are closed and all unnecessary processes stopped prior to running benchmarks.

Note that if you don’t specify the configuration parameter then the runtime will attempt to do benchmarking on non-optimized, debug-mode code. And you’ll be presented with the same error shown in Figure 1.

Analyze the benchmarking results

Once the execution of the benchmarking process is complete, a summary of the results will be displayed at the console window. The summary section contains information related to the environment in which the benchmarks were executed, such as the BenchmarkDotNet version, operating system, computer hardware, .NET version, compiler information, and information related to the performance of the application.

A few files will also be created in the BenchmarkDotNet.Artifacts folder under the application’s root folder. Here is a summary of the results. 

IDG

As evident from the summary shown in Figure 2, for each benchmarked method, you will see a row of data that specifies the performance metrics such as mean execution time, Gen 0, Gen 1, Gen 2 collections, etc.

On examining the results shown in Figure 3, you can see that the ConcatStringUsingGenericList is much faster than the ConcatStringUsingStringBuilder method. You can also see that there are many more allocations after running the ConcatStringUsingStringBuilder method.

IDG

Now add the RankColumn attribute on top of the MemoryBenchmarkerDemo class. This will add an extra column to the output indicating which method was faster. Run the benchmarking process again using the following command.

dotnet run -p BenchmarkDotNetDemo.csproj -c Release

When you run this command, the benchmarking process kicks off and displays the output after the benchmarking process has been executed successfully. Figure 4 below shows the output with RankColumn added. 

IDG

BenchmarkDotNet is a nice tool that provides a simple way to make an informed decision about the performance metrics of your application. In BenchmarkDotNet, invocation of a method that has the Benchmark attribute set is known as an operation. An iteration is the name given to a collection of several operations. 

You can explore a demo ASP.NET Core application that illustrates several ways to benchmark the code. You can get the application from the ASP.NET repo on GitHub. 

How to do more in C#: