Run a Microsoft Fabric notebook from Azure DevOps

Reading Time: 7 minutes

In this post I want to share one way that you can run a Microsoft Fabric notebook from Azure DevOps.

You can consider this post a follow-up to my last post about unit tests on Microsoft Fabric items. Since somebody asked me about automating notebooks and I wanted to show it in action.

Please note, currently the ability to call the API that runs a notebook on demand does not support service principals.

Therefore, in this post I show two different ways to authenticate when calling APIs. One way to authenticate with a service principal and another with a Microsoft Entra ID user account that has multifactor authentication (MFA) disabled. As per the below diagram.

I strongly recommend working with this in a test environment until service principals are supported. Unless working with a regular account with multifactor authentication disabled is acceptable in your production. Which is unlikely.

By the end of this post, you will know one way that you run a Microsoft Fabric notebook from Azure DevOps. Which you can customize how you see fit to align with any security policies your company has. Along the way I share plenty of links.

Preparing my Microsoft Fabric notebook

I made some changes to my notebook that I created for my pytest example in my previous post. In the notebook I now focus solely on the publicholidays table.

For those looking to do more with your notebooks, you can setup the Spark session configuration magic command in the first cell of your notebook.

Allowing you to specify parameters such as default Lakehouses and location of custom jar files for items such as libraries. As per the below example.

%%configure
{
    "defaultLakehouse": {  
        "name": "{LAKEHOUSE NAME}",
        "id": "{LAKEHOUSE ID}",
    }
}

If you start typing in %%configure in the first code cell it will autocomplete with all the available parameters. However, as you can see above you can select which parameters to specify.

One thing I did for benefit of this post is rotate freezing the below cell to verify the cause of failed notebook runs. I show why later in this post.

test_no_missing_values(pd_publicholidaysdf)

Anyway, in Azure DevOps I created two variable groups. One for sensitive values and another for non-sensitive values. Which contain the below variables:

• servicePrincipalId – The clientid of a service principal.
• servicePrincipalKey – The secret of a service principal.
• tenantId – Tenant id of the tenant hosting Microsoft Fabric.
• resourceUrl – Main API URL to authenticate against for a token, in this case “https://api.fabric.microsoft.com”.
• workspaceId – Id value of workspace that contains notebook.
• notebookName – Name of the notebook.
• user – Name of Microsoft Entra user account with MFA disabled. Please remember my warning about this!
• userpw – password for Microsoft Entra user account.

Once that was done I created my YAML Pipeline in Azure DevOps to add the below tasks.

Get notebook id in Azure DevOps

I decided to create one job in my YAML pipeline with multiple tasks. With my first task calling an API to return the id value of a specific notebook. Which can be done with a service principal.

So, I first authenticated with the service principal and returned a token I can use to call the API with the below code.

$SecureStringPwd = ConvertTo-SecureString $(servicePrincipalKey) -AsPlainText -Force
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(servicePrincipalId), $SecureStringPwd
            
Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(tenantId)

# Get authentication
$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

Afterwards, I called the API.

$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

$apiUrl = "https://api.fabric.microsoft.com/v1/workspaces/$(workspaceId)/notebooks"
$headers = @{
"Authorization" = "Bearer $fabricToken"
"Content-Type"  = "application/json"
}

$list = Invoke-RestMethod -Uri $apiUrl -Headers $headers -Method GET
$notebookId = ($list.value | Where-Object {$_.displayName -eq "$(notebookName)"}).id
Write-Host "##vso[task.setvariable variable=notebookId;]$notebookId" 

pwsh: true

I performed this with the Invoke-RestMethod PowerShell cmdlet. Due to its flexibility. As you can see, I also stated pwsh to be true at the end of the task. Which allows me to work with newer parameters supported by this cmdlet.

One other key point I want to highlight is that the returned notebookId value is added as a new variable in my pipeline.

Running a Microsoft Fabric notebook from Azure DevOps

Out of the three tasks this is the most relevant one for this post. Since this is the one that runs the Microsoft Fabric notebook from Azure DevOps.

As I mentioned before, in order to do this, I setup a Microsoft Entra user account that has MFA disabled.

Note: Please remember my warning and try to avoid doing this in a production environment. My advice is this to test this in a test environment. Ideally, your own Microsoft Fabric environment.

If your company allows this in production it is a judgement call on your part and I take zero responsibility.

Anyway, I authenticated the user and got a bearer token with the below code.

$SecureStringPwd = ConvertTo-SecureString '$(userpw)' -AsPlainText -Force
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(user), $SecureStringPwd
            
Connect-AzAccount -Credential $pscredential -Tenant $(tenantId)

# Get authentication
$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

I then built up the required parameters and called the API with Invoke-RestMethod.

$startnotebookUrl = "https://api.fabric.microsoft.com/v1/workspaces/$(workspaceId)/items/$(notebookId)/jobs/instances?jobType=RunNotebook"
            
$headers = @{
"Authorization" = "Bearer $fabricToken"
 "Content-Type"  = "application/json"
}

Invoke-RestMethod -Uri $startnotebookUrl -Headers $headers -Method POST -ResponseHeadersVariable headers

On a side note, you can also build up the same configuration as the Spark session configuration magic command in URL for the API call. Providing you the flexibility to specify various settings like additional jar files for libraries dynamically.

You can see a good example of this in the documentation that covers how to run a notebook on demand.

Another key point I want to highlight about the above code is that I added the -ResponseHeadersVariable parameter. So I can get a returning URL which allows me to call an API to check the progress of the notebook.

I added the returned value as a variable in the Azure DevOps pipeline so that I can call it in the next task.

$statusUrl = $headers.location[0]

Write-Host "##vso[task.setvariable variable=statusUrl;]$statusUrl"

You can read about this in-detail in the guide to working with REST APIs and PowerShell’s Invoke-RestMethod.

Warning in Azure DevOps task

It is worth noting that when I checked the status of this task in Azure DevOps I got the below warning.

WARNING: Authentication with a username and password at the command line is strongly discouraged. Use one of the recommended authentication methods based on your requirements. For additional information, visit https://go.microsoft.com/fwlink/?linkid=2276971.

Which emphasizes my warning about usernames and passwords.

Anyway, after adding this task I added one final task. Which checks on the progress of the running notebook.

Checking the progress of the Microsoft Fabric notebook in Azure DevOps

Because I can use a service principal with the API that checks the progress of the running notebook, I authenticated the same way as I did with the first task.

Install-Module -Name Az.Accounts -AllowClobber -Force

$SecureStringPwd = ConvertTo-SecureString $(servicePrincipalKey) -AsPlainText -Force
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(servicePrincipalId), $SecureStringPwd
            
Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(tenantId)

# Get authentication
$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

From there, I built up the header for my API call before calling the API to check the current status of the notebook in Microsoft Fabric.

$headers = @{
"Authorization" = "Bearer $fabricToken"
"Content-Type"  = "application/json"
}
            
# Write-Host $(statusUrl)

$status = (Invoke-RestMethod -Uri $(statusUrl) -Headers $headers).status

I then configured the below logic. Which polls for the status of the running notebook every ten seconds and will inform me if the notebook has either completed of failed.

while ($status -ne 'completed' -and $status -ne 'Failed') {
Write-Host "Current status: $status. Waiting for notebook to be completed..."
Start-Sleep -Seconds 10 
$status = (Invoke-RestMethod -Uri $(statusUrl) -Headers $headers).status
}

if ($status -eq 'completed') {
  Write-Output "Notebook has completed running."
} else {
  Write-Output "Notebook has failed, probably due to an assertion failure. Check the stdout log for assertion failure."

  exit 1
}


pwsh: true

Testing the pipeline to run Microsoft Fabric notebooks in Azure DevOps

I first tested by disabling the final cell in my notebook which runs the assertion test. Which worked as expected in Azure DevOps as below.

However, when I enabled the final cell in the notebook the pipeline failed. Due to the logic that I put in the if statement. As you can see below the below example. Which also shows my custom failure message.

I verified this in Microsoft Fabric by selecting the recent runs of the notebook. In order to view its history.

I then clicked on the last entry and navigated to the Logs tab. From there I went to Driver (stdout) -> Latest stdout and searched for the word assertionError. As you can see below it found the assertion error in the log and showed the text to confirm it was due to a failed test.

Of course, this is just one way to manage failures. You can customize the above to suit your requirements.

For example, you can remove my logic that fails the task if a test fails. However, just because you can does not mean you should. Because this is a quick way to identify if a unit test has failed.

Final words

I do hope that this demonstration of one way to run a Microsoft Fabric notebook from Azure DevOps has been of interest to you. In addition, inspired some of you to look at APIs further and customize them to suit your needs.

Personally, I am looking forward to service principal support being made available for the API that runs a notebook on demand.

Of course, if you have any comments or queries about this post feel free to reach out to me.