Getting Started

Posted on:

This section guides you through the process of installing SpecFlow and SpecFlow+ Runner and setting up a simple project in Visual Studio. The sample project is intentionally kept very simple and consists of a basic “calculator” that adds two numbers and returns the result. You will define a test scenario to automate the testing of the calculator’s code.

This guide contains the following sections:

Installation and Setup

Installing SpecFlow consists of two steps:
  1. Install the IDE integration
  2. Set up your Visual Studio project to work with SpecFlow

Installing the IDE Integration Packages

The method used to install the IDE Integration packages depends on your IDE:
  • Visual Studio 2010+ (including Community Edition but not Express Edition): The easiest method is to select Tools | Extensions and Updates from the menu in Visual Studio, switch to the Online search on the left and enter “SpecFlow” in the search field at the top right.
    ExtensionsAndUpdatesDialog
  • Visual Studio Express: Refer to Install IDE Integration for direct download links and information on installing SpecFlow for other IDEs.

Setting Up your SpecFlow Project

This section guides you through the first steps of setting up a Visual Studio project with SpecFlow and defining and executing your first test scenario. In this example, we will be using SpecFlow+ Runner, but you can use a number of other test execution frameworks, including NUnit, Visual Studio Test Window and Team Foundation Server (TFS) Build integration for SpecFlow. SpecFlow+ Runner’s advantages include integration with Visual Studio Test Runner and extensive integrated reports available from within Visual Studio.

SpecFlow tests are usually placed into one or more separate projects in your solution, and these projects are referred to as a “specification project” below. The easiest and most convenient way to set up these projects is to use our SpecFlow NuGet package or one of the specific helper packages, like SpecRun.SpecFlow or SpecFlow.NUnit. For a detailed project setup guide, check the Setup SpecFlow Projects page.

To set up your specification project:

  1. Add “Unit Testing Project” to your solution (e.g. “MyProject.Specs”).
    Note: Creating a “Unit Test Project” is the recommended procedure, as it reduces the number of steps required to set up your project.
  2. You can choose to remove the UnitTestX.cs file, as it is not required.
  3. Add SpecFlow+ Runner to your specification project using NuGet:
    1. Right-click on your specification project (e.g. “MyProject.Specs”) and select Manage NuGet Packages.
    2. Search for “SpecRun” and install SpecRun for SpecFlow 1.9.0.

      Alternatively, you can install the package from NuGet’s console (Tools | NuGet Package Manager | Package Manager Console) as follows:
      PM> Install-Package SpecRun.SpecFlow
  4. Installing SpecFlow+ Runner automatically downloads the SpecFlow runtime from NuGet and adds it to your specifications project.
Note: Instead of SpecFlow+ Runner, you can also use other test engines, like MsTest, xUnit, MbUnit or Nunit. These packages are installed in exactly the same manner as SpecFlow+ Runner. However to follow all the steps in this guide, you need to install SpecFlow+ Runner.

Adding a Feature File

You now need to add a feature file to your specifications project that outlines a specific feature and includes a test scenario:
  1. Right-click on your specifications project and select Add | New Item from the popup menu.
  2. Select SpecFlow Feature File (restrict the entries by entering “SpecFlow” in the search field), give it a meaningful name (e.g. “Calculator.feature”) and click on Add.

  3. The feature file is added to your specification project. It includes a default scenario for adding two numbers. You can leave most of the content as is, but add the word “also” to the scenario on the line beginning with “And” (see red text below):

Feature: Calculator
In order to avoid silly mistakes As a math idiot I want to be told the sum of two numbers @mytag Scenario: Add two numbers Given I have entered 50 into the calculator And I have also entered 70 into the calculator When I press add Then the result should be 120 on the screen

We will use this scenario to demonstrate the first development iteration. By adding “also” to the second Given statement, we have two separate statements that will be parsed differently (otherwise both statements would match the regular expression “I have entered (.*) into the calculator”). This both simplifies the code in our example and illustrates how different statements linked with “And” are handled.

You can also assign tags to scenarios (e.g. “@mytag” in the example above), which can be used to filter scenarios, and control how scenarios are executed and automated. For more details on using tags, see Hooks, Scoped Bindings, FeatureContext and ScenarioContext in the documentation.

Generating Step Definitions

In order to test our scenario, we need to create step definitions that bind the statements in the test scenario to the application code. SpecFlow can automatically generate a skeleton for the automation code that you can then extend as necessary:
  1. Right-click on your feature file in the code editor and select Generate Step Definitions from the popup menu. A dialogue is displayed.
  2. Enter a name for the class, e.g. “CalculatorSteps”.
    GenerateStepDefinitionSkeleton
  3. Click on Generate and save the file. A new skeleton class is added to your project with steps for each of the steps in the scenario:
    InitialSkeleton

Executing Your First Test

SpecFlow+ Runner integrates with Visual Studio Test Explorer. After adding your first specification and building the solution, the business readable scenario titles will show up in Visual Studio Test Explorer:
  1. Build your solution.
  2. Select Test | Windows | Test Explorer to open the Test Explorer:
    TestExplorerInitialStatus
    Scenarios are displayed with their plain text scenario title instead of a generated unit test name.
  3. Click on Run All to run your test. As the automation and application code has not yet been implemented, the test will not pass successfully.
    Note: The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a demo license key to remove this restriction (please include your name and company name).

Implementing the Automation and Application Code

In order for your tests to pass, you need to implement both the application code (the code in your application you are testing) and the automation code (binding the test scenario to the automation interface). This involves the following steps, which are covered in this section:
  1. Reference the assembly or project containing the interface you want to bind the automation to (including APIs, controllers, UI automation tools etc.).
  2. Extend the step definition skeleton with the automation code.
  3. Implement the missing application code.
  4. Verify that the scenario passes the test.

Adding a Calculator Class

The application code that implements the actual functions performed by the calculator should be defined in a separate project from your specification project. This project should include a class for the calculator and expose methods for initialising the calculator and performing the addition:
  1. Right-click on your solution in the Solution Explorer and select Add | Project from the context menu. Choose to add a new class library and give your project a name (e.g. “Example”).
  2. Right-click on the .cs file in the new project and rename it (e.g. “Calculator.cs”), and choose to rename all references.
  3. Your new class should be similar to the following:
    EmptyCalculatorClass

Referencing the Calculator Class

  1. Expand your specification project and right-click on the References Select Add Reference from the context menu.
  2. Click on Solution on the left. The projects in your solution are listed.
  3. Enable the check box next to the Example project to reference it from the specifications project.
  4. Click on OK. A reference to the Example project is added to the References node in the Solution Explorer.
  5. Add a using directive for the namespace (e.g. “Example”) of your Calculator class to the CalculatorSteps.cs file in your specification project:
    using Example
  6. Define a variable of the type Calculator in the CalculatorSteps class prior to the step definitions:
     
    public class CalculatorSteps
    {
        private Calculator calculator = new Calculator();
        [Given(@"I have entered (.*) into the calculator")]
    …
    
    Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps and ensures the variable is persistent between steps.

Implementing the Code

Now that the step definitions can reference the Calculator class, you need to extend the step definitions and implement the application code.

Binding the First Given Statement

The first Given statement in the scenario needs to initialise the calculator with the first of the two numbers defined in the scenario (50). To implement the code:
  1. Open CalculatorSteps.cs if it is not already open. The value defined in the scenario is passed as a parameter in the automation code’s associated function, e.g.:
  2. Rename this parameter to something more human-readable (e.g. “number”):
    public void GivenIHaveEnteredIntoTheCalculator(int number)
  3. To initialise the calculator with this number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
    
    public void GivenIHaveEnteredIntoTheCalculator(int number)
    {
        FirstNumber = number;
    }
    
  4. Switch to the file containing your Calculator class (e.g. Calculator.cs) and add a public integer member to the class:
    public int FirstNumber { set; private get; }
You have now determined that the FirstNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the Second Given Statement

The second Given statement in the scenario needs to initialise the second number with the second value defined in the scenario (70). To implement the code:
  1. Open CalculatorSteps.cs if it is not already open.
  2. Locate the function corresponding to the second Given statement and rename the p0 parameter to “number”, as before.
  3. To initialise the calculator with the second number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
    
    public void GivenIHaveAlsoEnteredIntoTheCalculator(int number)
    {
        SecondNumber = number;
    }
    
  4. Switch to the file containing your Calculator class and add another public integer member to the class: public int SecondNumber { set; private get; }
You have now determined that the SecondNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the When Statement

The step for the When statement needs to call the method that performs the actual addition and store the result. This result needs to be available to the other final step in the automation code in order to verify that the result is the expected result defined in the test scenario.

To implement the code:
  1. Open CalculatorSteps.cs if it is not already open.
  2. Define a variable to store the result at the start of the CalculatorSeps class (before any of the steps): private int result { get; set; } Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps.
  3. Locate the function corresponding to the When statement and edit it as follows:
    
    public void WhenIPressAdd()
    {
        result = calculator.Add();
    }
  4. Switch to the file containing your Calculator class and define the Add() method:
    
    public int Add()
    {
        return FirstNumber + SecondNumber;
    }
You have now determined that the Add() method of the calculator class is called once the initial Given steps have been performed.

Binding the Then Statement

The step for the Then statement needs to verify that the result returned by the Add() method in the previous step is the same as the expected result defined in the test scenario. To implement the code:
  1. Open CalculatorSteps.cs if it is not already open. As the result will be verified using Assert, you need to add “usingVisualStudio.TestTools.UnitTesting;” to the top of your automation code.
  2. Locate the function corresponding to the Then statement. Rename the p0 parameter in the function call (this time to “expectedResult”) and edit the step definition as follows:
    
    public void ThenTheResultShouldBeOnTheScreen(int expectedResult)
    {
        Assert.AreEqual(expectedResult, result);
    }
You have now implemented the final piece of the jigsaw – testing that the result returned by your application matches the expected result defined in the scenario.

Final CalculatorSteps.cs Code

Your CalculatorSteps.cs code should be similar to the following:
using System;
using TechTalk.SpecFlow;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Example;

namespace MyProject.Specs
{
    [Binding]
    public class CalculatorSteps
    {
        private int result { get; set; }
        private Calculator calculator = new Calculator();

        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int number)
        {
            calculator.FirstNumber = number;
        }

        [Given(@"I have also entered (.*) into the calculator")]
        public void GivenIHaveAlsoEnteredIntoTheCalculator(int number)
        {
            calculator.SecondNumber = number;
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            result = calculator.Add();
        }
        
        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int expectedResult)
        {
            Assert.AreEqual(expectedResult, result);
        }
    }
}

Final Calculator.cs Code

Your Calculator.cs code should be similar to the following:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Example
{
    public class Calculator
    {
        public int FirstNumber { set; private get; }
        public int SecondNumber { set; private get; }
        
        public int Add()
        {
            return FirstNumber + SecondNumber;
        }
    }
}

Executing the Tests Again

Now that the test steps have been bound to your application code, you need to rebuild your solution and execute the tests again (click on Run All in the Test Explorer). You should see that the test now passes (green).
TestExplorerInitialStatus
Click on Output in the Test Explorer to display a summary of the test:
TestSummary
This example is obviously very simple; at this point you would want to refactor your code before proceeding with the implementation of your remaining scenarios.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner includes a number of features that make working in Visual Studio easier.

Grouping Tests by Category

When using SpecFlow+ Runner to execute your tests, you can group scenarios according to various attributes:
  • Select Group By Traits to group the SpecFlow scenarios by the tags defined at the scenario or feature level:
  • Select Group By Class to group the SpecFlow scenarios by feature (with the feature title in plain text:

SpecFlow Reports

SpecFlow+ Runner generates an advanced execution report for each test run. To view the report, select Tests in the Show output from field in the Output window:

Click on the link to the report file to view the report in Visual Studio:
You can configure the name of the report file in the SpecFlow+ Runner settings (ReportFile element in the .runsettings file). More details are available here.
The report provides a convenient side-by-side view of the individual scenario steps and the trace output for each step:

Step definitions can write to the trace output to provide a lower level technical representation of the executed step (e.g. a more verbose representation of data structures that are processed in a step).

Running SpecFlow Tests from the Command Line

You can also run runtests.cmd from the command prompt to execute your specification tests.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner has a lot more to offer:
  • Parallel test execution
  • Detect flickering (intermittently failing) scenarios
  • Smart test execution order (run failed tests first)
  • Execute tests in different runtime environments
  • Automatic deployment configuration
  • SpecFlow integration with Team Foundation Server