Another way to check Power BI reports are using the right template with Azure DevOps

Reading Time: 7 minutes

In this post I want to show another way to check Power BI reports are using the right template with Azure DevOps. Like in the below diagram.

You can consider this a follow-up to my previous post. Where covered how you can automatically check Power BI reports are using the right report template with Azure DevOps with the aid of PBI Inspector.

Since I mentioned it was possible in my previous post, I want to show you how to do it with Python. For the benefit of those who want a Python alternative to working with PBI Inspector.

As a bit of a bonus, I also show how you can publish the test results back into Azure DevOps. Along the way I share plenty of links.

If you need help with any jargon, then I recommend that you read one my other posts. Which is a Microsoft Fabric Git integration jargon guide for Fabricators.

Key points about this post

Some key points I want to highlight about this post before I go any further:

• For the benefit of this post, I am showing how to work with a YAML pipeline in Azure DevOps.
• This post covers how to automatically check that Power BI reports are using the right template by checking the theme specified for the report. You can change the logic to check for separate items include in your template like logos instead.
• Logic to run the pipeline shown in this post was adapted from the Power BI Project (PBIP) and Azure DevOps build pipelines for validation guide by Microsoft.

Prerequisites

In order to follow along with this post you will need the following:

• A Git repository created in the Azure Repos service in Azure DevOps.
• Either a Fabric capacity or a Power BI Premium capacity so that you can configure Microsoft Fabric Git integration on workspaces that link to your created repository.
• Power BI reports that are saved as Power BI Desktop Projects locally when working with Power BI Desktop. Alternatively, a report created in a workspace with Microsoft Fabric Git integration configured.
• If working with Power BI reports locally, then recommend you install both Git and Visual Studio Code.

Ideally, you should also create a report template if you intend to follow along with this post. Like the one I created in my previous post.

Python script to check Power BI reports are using the right template

To test that the template exists for all Power BI reports I had to create a test that checked that the theme used in the template existed in all reports.

I decided to implement a test that would go through every report folder and check in each one that the “theme” value in all the “report.json” files started with “PlentyOfGreen”. I needed to check the start of the value due to the additional identified added to the end of the theme name.

In addition, I also decided to implement the test with the Pytest library.

To do this I went into Visual Studio Code and opened up the Power BI Desktop Project that I had created for my previous post. I then created a new “Tests” subfolder, and a Python script file called “test_report_theme.py”. Once done, I added the below code to the Python script.

import os
import json
import pytest

# Set WORKSPACE_DIR to the root folder of the project
WORKSPACE_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))

def find_report_jsons(workspace_dir):
    """Yield (report_name, report_json_path) for all Report folders."""
    for entry in os.listdir(workspace_dir):
        if entry.endswith(".Report"):
            report_dir = os.path.join(workspace_dir, entry)
            report_json = os.path.join(report_dir, "report.json")
            if os.path.isfile(report_json):
                yield entry, report_json

@pytest.mark.parametrize("report_name,report_json_path", list(find_report_jsons(WORKSPACE_DIR)))
def test_report_theme(report_name, report_json_path):
    with open(report_json_path, "r", encoding="utf-8") as f:
        report = json.load(f)
    theme = report.get("theme", "")
    assert isinstance(theme, str) and theme.startswith("PlentyOfGreen"), (
        f"Report '{report_name}' theme value starts with 'PlentyOfGreen': {theme}"
    )

I did what any good developer does and reused logic from a script that I shared in a previous post about Data Pipelines. However, this one was a bit simpler as it is checking if every “report.json” file has a “theme” value that starts with “PlentyOfGreen”.

Once I had synchronized my Python script to my Git repository in Azure Devops it was time to create the YAML Pipeline.

YAML Pipeline to automatically check Power BI reports are using the right template

I started a new YAML Pipeline by creating a YAML file in my dev branch. First, I added the below code to the file which sets the trigger to none and specifies a Microsoft-Hosted Azure Pipelines Agent.

trigger: none

pool: 
  vmImage: 'windows-latest'

I then created a new stage and job as below.

stages:

- stage: TestReports
  displayName: 'Test Reports'
  
  jobs:
  - job: 'TestReports'
    displayName: 'Test Report content'

I was then able to add the below steps to my job within my YAML Pipeline.

Use Python 3.12

First task to run is to select the version of Python to work with via the Python version task. I had previously encountered issues when selecting Python 3.13 so I opted for Python 3.12 instead.

- task: UsePythonVersion@0
  displayName: 'Use Python 3.12'
  inputs:
    versionSpec: 3.12

Install necessary libraries

Afterwards I had to install the Pytest library in a PowerShell task with the below code. In order for the test I specified in my Python file to work.

- task: PowerShell@2
  displayName: 'Install necessary library'
  inputs:
    targetType: 'inline'
    script: | 
      pip install pytest
    pwsh: true

Run sample pipeline test

Once the library was installed the next task runs the Python script that I cover earlier in this post. In addition, I specify to output the test results into a special xml file so that I can display the results in Azure DevOps.

- task: PowerShell@2
  displayName: 'Run report tests'
  inputs:
    targetType: 'inline'
    script: |
      pytest Tests\test_report_theme.py --junitxml=test-results-all-reports.xml
    pwsh: true

Publish Pipeline Test Results

Last task that I configured in the new stage was a Publish Test Results v2 task. Which publishes the xml output that contains the test results into Azure DevOps. I specified to look for the below file(s) with the JUnit result format.

- task: PublishTestResults@2
  displayName: 'Publish Report Test Results'
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/simple-*.xml'
    failTaskOnMissingResultsFile: true
    testRunTitle: 'Report Tests'
  condition: always()

I set the condition to be always so that the task always runs. Even if there is a failure in the previous task.

Doing this means that when there is a test failure this task will still publish the results. However, the stage itself will fail and the YAML pipeline will not continue.

I then went into Azure DevOps and created a new YAML Pipeline. Specifying the new YAML file that I had created in my dev branch. I performed an initial run to check it worked properly.

Afterwards, I defined a branch policy the same way as described in the Microsoft article that covers Power BI Project (PBIP) and Azure DevOps build pipelines for validation.

Once the development workspace was in-place I was ready to start testing everything.

Test results to check Power BI reports are using the right template with Azure DevOps

First of all, I used the Branch out to new workspace functionality to create a new feature branch and workspace.

First of all, I used the Branch out to new workspace functionality to create a new feature branch and workspace.

Doing so populated the feature workspace with my original report. I made a basic change to report in my feature workspace and committed the change.

Afterwards, I went to the repository in Azure DevOps to create a Pull Request to merge the update from the feature branch to the dev branch. It ran my pipeline as part of the check, which completed successfully.

Afterwards, I looked at completed pipeline.

I then clicked on Tests to view the test result, filtered for passed tests and zoomed out to view more details.

Testing for Power BI reports that are not using the template

Of course, my testing would not be complete without testing for failure. So, I uploaded a PBIX file of a report that did not use the theme. Which automatically created a new Power BI report in my feature workspace

When I went to complete a Pull Request from the feature branch to the dev branch it failed with the below error.

I went to the failed pipeline run and clicked in Tests to view the test results. Filtering for both passed and failed tests and zooming out to view more details. Which clearly indicated that the new report caused the test to fail.

I then performed other checks. Including publishing another report that was created with the same template which passed. In addition, I also changed the theme name prefix to PlentyOfBlue in my rules which caused the test to fail with the below error message.

AssertionError: Report 'Power BI Template Check.Report' theme value starts with 'PlentyOfBlue': PlentyOfGreen2438104580276158.json
assert (True and False)

After multiple checks I concluded that the tests were working as they should.

Final words

I hope this way to check Power BI reports are using the right template with Azure DevOps helps those of you looking for a more developer-based way to check Power BI reports. In addition, I hope it inspires some of you to think about how to improve your testing strategies.

I suspect the method in this post will be more popular with those with a Data Engineering background who want to implement a quick check that the report theme is consistent. As opposed to delving into the finer details of PBI Inspector.

Of course, if you have any comments or queries about this post feel free to reach out to me