Deploying Scheduled Triggers In Data Factory v2 With PowerShell

So in ADFv2, scheduled triggers have been overhauled to become their own deployable components, able to kick off multiple parameterised pipelines together in each trigger.

This new version of the trigger seems far more modular and flexible than the functionality in v1, however there are a couple of catches and easy-to-miss steps during implementation if you choose to use PowerShell rather than the brand new visual authoring tool, so I’ll break down the process below.

First, you need to build your trigger, as a separate JSON object to the pipeline. No additional code is needed in the pipeline itself. There is a more complete example in the documentation Microsoft has provided, but on top of that here is a working example I’ve written below:

{
  "name": "ST_OrchestratorTrigger",
  "properties": {
    "type": "ScheduleTrigger",
    "typeProperties": {
      "recurrence": {
        "frequency": "Day",
        "interval": 1,
        "startTime": "2017-12-20T04:00:00-05:00",
        "endTime": "2099-11-02T22:00:00-08:00",
        "schedule": {
          "hours": [ 4 ],
          "minutes": [ 0 ]
        }
      }
    },
    "pipelines": [
      {
        "pipelineReference": {
          "type": "PipelineReference",
          "referenceName": "PL_MasterOrchestration"
        },
        "parameters": {}
      }
    ]
  }
}

As you might guess, this trigger runs the PL_MasterOrchestrator pipeline (with no input parameters) at 4am each day.

A caveat on the recurrence – it seems that when running, the trigger doesn’t strictly adhere to the “startTime” after the first time it runs, so despite starting at 4am, the subsequent trigger could be minutes or possibly hours off on the next day.

To ensure it always triggers at exactly 4am, you need to add the “schedule” recurrence attribute as shown above.

Deployment of the trigger (as of writing this blog) is still only done through PowerShell scripts, using the cmdlet below:

Set-AzureRmDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName `
                -DataFactoryName $DataFactoryName `
                -DefinitionFile $TriggerFile `
                -Name $TriggerName `
                -Force; 

Each of these parameters are exactly as you would expect from the v1 parameters for setting other objects. -Force shouldn’t strictly be necessary, but sometimes is worthwhile putting in depending on the exact circumstances, -DefinitionFile is asking for the .JSON file itself, etc.

So, at this point you’ve created the scheduler, and deployed it to your data factory. Scheduled triggers in v2 need to be started, once they are deployed. You can do this with the following:

Start-AzureRmDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName 
-DataFactoryName $DataFactoryName 
-Name $TriggerName 

Just another quick note here: you will need to stop the trigger again if you want to redeploy it. Trying to overwrite a trigger while it’s running will just result in an error message.

As well as finding the necessary cmdlets 2/3rds of the way down the Microsoft document linked above, there is a list of all relevant cmdlets found here.

Hope this helps!

The documentation again:

https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-schedule-trigger

https://docs.microsoft.com/en-us/powershell/module/azurerm.datafactoryv2/start-azurermdatafactoryv2trigger?view=azurermps-4.4.1