Azure Test Plans jargon for Data Platform professionals

Reading Time: 7 minutes

In this post I want to cover some Azure Test Plans jargon for Data Platform professionals. Because I understand it can be confusing.

In addition, I did say I would explain some jargon in my last post about using Azure Test Plans for Data Platform deployments. Of course, these explanations will help with other kinds of deployments as well as Data Platform ones.

By the end of this post, you will have a better understanding of some of the jargon involved in Azure Test Plans. Plus, a good recommendation of a lab to use.

Below are some quick links in case you want to go to a particular section.

• Test Plans structure
• Types of test suites
• Configurations
• Recommended lab

Test Plans structure

Now, I will quickly explain the structure of test plans here as far as test plans, test suites and test cases are concerned. Because the best way I can explain the structure of test plans to Data Platform professionals is that I like to think it as semi-structured.

Below is a diagram to help visualize how the structure can look depending on your needs.

At the lowest level of the structure, you have your test cases, which you tend to test on one part of a deployment. For example, a test case that contains steps to check that a database update worked.

Next you have your test suites. Which can contain one or more test cases that can be grouped together. For example, a test suite can be for a database deployment which contains various test cases for separate databases.

In reality, you do not always create the structure of your test plans manually. Because test suites can be created either manually or automatically depending on what you do in Azure Test Plans.

For example, you can create a static test suite manually. However, a requirements-based suite can also be created for you if you create a test case using the Boards feature in Azure Boards. I cover this in more detail later on in this post.

Finally, you have test plans, which can refer to a collection of both test suites and test cases. For example, a test plan can be for an ARM template deployment. Consisting of multiple test suites for different databases. Alternatively, it can contain some test suites for various applications and some test cases about individual table updates as well.

SQL Server Test Plan example

To help understand this how you can use this structure, say you are doing a SQL Server database deployment.

If you are deploying to just one SQL Server database, you can simply have a Test Plan that contains a test case for the deployment to that database. Containing various steps to check that it was deployed properly.

For a more complex solution the same test case might be part of test suite instead. Like the below example.

In reality, the above structure can be expanded for a more complex environment. For example, the solution idea proposed by Microsoft for an Advanced Analytics Architecture.

Types of test suites in Azure Test Plans

In Azure Test Plans there are three different types of test suites that you can use. Adding them to your test plans introduces some interesting options. Which I will demonstrate below.

Static test suites in Azure Test Plans

Static test suites are where you have full control, and you can add any test cases you want manually. Including the ability to drag existing test cases into new static test suites.

For example, say I wanted to move the test case I created in my post about using Azure Test Plans for Data Platform deployments. First, I create a static test suite.

Afterwards I can move the test case easily using drag and drop as below.

Requirements-based suites

Requirements-based suites are test suites that are dedicated to specific work items. Usually, you assign a test suite to specific work items by using an easy-to-use query designer. Which will feel very familiar to those of you who have used query designers in the past. Or the Query feature in Azure Boards.

Basically, you run the query and then select which work items you want to create test suites for. For example, say I select the two work items shown above and click ‘Create suites’ button. Doing this creates the below test suites in my sample test plan.

In addition to the above, sometimes requirements-based suites are created automatically. In my previous post about using Azure Test Plans for Data Platform deployments I showed you how you can create a test case directly within a work item. Like the example below.

One of two things can happen if you create test cases this way. Depending on whether or not you have existing test plans.

If there are no existing test plans, then a new one is created along with a new requirements-based suite. Within the test plan the new test suite is given the same name as the work item that the new test case was created in. Which contains the new test case.

However, if you have an existing test plan created in the same sprint as the work item it will create a requirements-based suite there instead.

In fact, I tested this in a bit more detail. If you already have multiple test plans created for the same sprint it appears to create the test suite in the last one that you created.

Query-based suites in Azure Test Plans

Something tells me a lot of Data Platform professionals who enjoy writing queries will appreciate query-based suites.

It’s a special suite where you create a query in Azure Test Plans and any that test cases that meet match that criteria will automatically be added to the test suite.

However, the other good thing is that it will automatically add new test cases that meet that criterion to the test suite. To save you adding them to existing test suites yourself.

For example, I can create a query-based suite and add and extra clause for any test cases where the tag contained Azure Data Factory I would at first get the below. Because I do not have any test cases with that tag in it.

However, if I create a test case that contains the tag ‘Azure Data Factory’ and then refresh the test plan I get the below.

Doing this can save you a lot of administration. In addition, it provides an easy way to group test cases together.

Configurations

I will briefly mention configurations here. Because it can be useful for those of you who want to test that dashboards and reports work the same in different browsers.

Basically, it allows you to set different configurations that you want to test things on. For example, you can create a configuration for using Windows 10 with Chrome. Afterwards, you can apply that configuration to a test case to test a Power BI report. Microsoft provides a guide on how you can test different configurations.

Recommended lab

If you want a better understanding of Azure Test Plans I recommend going through the lab in Azure DevOps Labs called Test Planning and Management with Azure Test Plans.

Bear in mind that this lab was created a few years ago. So, some of the things have changed since then. However, it will give you hands-on experience as well as a better understanding of Azure Test Plans.

Plus, you can get a populated Azure DevOps project created in your own organization for you. You can get one setup in your own Azure DevOps organization by going through the prerequisites.

I really like the fact that you can get these test projects in Azure DevOps Labs. Because it saves having to create a new project and populate it when I want to test things in Azure DevOps.

Just in case anybody needs somewhere to host these projects you get from the labs, I wrote a post before on how to create your own Azure DevOps organization.

Final words about Azure Test Plans jargon

I hope this post about Azure Test Plans jargon explains a few things for Data Platform professionals. In addition, I hope it gives some of you ideas of what you can do within Azure Test Plans.

Of course, if you have any comments or queries about this post feel free to reach out to me.