Deploying Azure Data Factory using Bicep

hey everyone my name is wilverleader i'm a software engineer and microsoft data platform mvp based in auckland new zealand and today i'm going to show you how we can deploy an azure data factory instance using bicep now for those of you who don't know what bicep is essentially bicep is a domain specific language for deploying resources to azure essentially it's like the next generation of arm templates so anything that can be done in arm can be done in bicep there are a few known limitations but that's just generally the idea now when i'm using infrastructure code to deploy resources to azure i usually opt for terraform because we used to use this a lot in my last company so when it came to doing my own personal projects i thought well i know terraform anyway i just i'll just use that however recently i've changed jobs it will change companies and now i'm finding that in my new role instead of terraform i'm using arm a lot more than than i used to um because to be honest with you i never really liked arm i didn't really want to spend my time messing about with jason um and also the syntax was almost like too complicated to work with when things really should be as simple as possible so when i first saw bicep i thought it'd be really good to give this a go because it looks a lot more simpler looks looks very terrifying to be honest um and you know since it's the next generation of arm it's probably good to learn it so i can use it um use it for my work anyway so coming back to this video i thought i'd keep it real simple and just we're not gonna be you know doing a deep dive into bicep we're not going to even going to be doing like a deep dive into azure data factory this is just going to be really simple really straightforward and just deploy an instance of azure data factory uh i want to do some more content in the future around data factory since it's a really nice mix between integration and data it's one of those technologies that sits quite nicely between the two and since i'm already learning biset i thought it'd be a really good opportunity to actually provision a data factory resource that i need for that future content and do a little bit of bicep at the same time so um hope you guys enjoy this video okay so before we get started into writing bicep code there's just a couple of things i want to note so the first things first is with um what microsoft have done is they've provided some documentation for template references that we can use um for various different type of resources so here we've got aks um we've got one for cosmos db databricks and the one we'll be using data factory and what we can do with this documentation we can actually view the differences between arm templates and bicep templates so here we can see what it would look like in jason here's what we um what how we would write it using bicep um and it looks like it's fairly straightforward as well there's also different api versions that get supported luckily for us there's just one um for for data factory if i use the latest uh hopefully it will show me yep so there's 2018 uh the first of june uh 2018 um i hope i'm i'm saying it hope that's in the right format hopefully it's not a weird like a like a strange american format that i'm not used to and as we can see here within the template we need to provide it some property values um some are required so we need to give it a name uh define the type of api that we or type of resource that we're deploying sorry the api version that we're going to be using and there's also some optional ones location we can provide it tags for data factory we can supply a managed service identity for the data factory itself uh some properties let's have a look at that uh so this is related to uh so some get repo information for the data factory you can actually hook up your git repositories um to azure data factory which is quite cool there's global parameters i'm not really sure what that is um but we're not going to be using it so it doesn't really matter for this one so in terms of documentation um there is some really good documentation around the template references for different types of resources depending on what api versions you're using it can differ in quality and also depending on the on the resource so just keep an eye out for that but when it comes to actually writing your bicep code um one of the things i really really recommend is if we go into visual studio code if i just go into my extensions and search oh there's bicep so there's an extension for bicep there's also an extension for arm templates as well so if you're still working with arm templates um but what this extension does it actually provides some validation for your bicep code also provides a little bit of intellisense as well which is really really straightforward and well so not straightforward but really really handy when you're writing bicep for the first time you just need a little bit of help um understanding what definitions are available to you so with that what we're going to do is we're just going to crack into creating our data factory instance so i've got a bunch of folders here i've created one for data factory i'm just going to name this file name main dot bicep so telling visual studio code this will be a bicep file what i'm going to do is i'm going to cheat slightly just take this documentation put it on my other screen so i can refer to it as i'm going along okay so first things first oops first things first go back to visual studio code what i need to do is define a resource block and then we need to give it a symbolic name so say if we're working with more complex bicep files and we need to create child resources and or we need create parent resources then refer to them in the child we give it a symbolic name so for this i'm just going to use the leader data factory we're not going to be doing that in this video but that's just what we can use it for and then we need to specify the resource types so what we need to do is microsoft dot data factory dot factories and then we need to specify an api version i'm just going to use the api version that we saw before which is that 2018 and open up my resource block now so the required resource required properties for this resource so we need a name obviously i'm just going to call that the leader data factory give it a location i'm going to choose australia east because that's the closest assist uh data center to me um here in new zealand hopefully the new zealand one is coming out in 2022 hopefully fingers crossed and i'm not going to define any tags i'm just going to leave that as an empty object but i also need to specify some properties as well cool so that's all the mandatory ones that i need to provide apart from tags that's optional and also location that's also optional as well the api version is defined here and the type is also defined here as well sweet so what i'm going to actually do with these properties name and location we can actually define these as parameters and these are very straightforward to do so we just need to indicate that this is a parameter uh give the parameter a name so data factory name specify the type which is a string i'm just going to use this for leader data factory here and then just copy this data the name of the parameter here is the name and the idea is um well the purpose of this is if you are deploying say data factory to multiple different environments so you've got a dev environment test environment uat and prod what you can do is you can actually specify your parameters in a different file name or you can set them via the command line so if you're doing deploying this via devops and you're just running like a bash script as part of your release pipeline you can actually specify um different parameter names as part of your azure command line command but for this we're just going to be using it within the same file keep it nice and straight forward i'm also gonna do one for location as well uh with location what you i can what i have seen some people do is use this resource group dot location um so you can deploy in the same resource location as your resource group um what i'm going to do is i'm just going to say australia east for now and then use this parameter as my location value there sweet that looks all good so that pretty much in like nine lines of code seems almost too too good to be true that it's that simple to define a azure data factory resource using bicep anyway let's have a look at how we can actually use this um to deploy this uh bicep file so bicep files can be directly deployed let me just zoom in a little bit so you guys can read that as well via the azure command line interface or powershell az module so the standard deployment commands will just work inverted commas with a bicep files you'll need az cli version 2.2 0.0 or over and powershell is a model uh 5.6.0 later than that uh what i'm going to do is i'm going to use the azure command line interface tool so if i go back into visual studio code if i just go a z wonder if atv will show me the version okay cool so yeah that's the shorthand version so i can see that azure cli uh version is 2.24.1 so we should be good to go i do have some updates available but what would do just clear that for now to clear the clutter okay so if i go back to this az cli command what i'll do i'll just copy and paste that the proper way not drag it across go back into visual studio code um oh yeah terminate batch job all i wanted to do i actually just typed that out so easy deployment because i don't want to deploy into that resource group so group create dot f um i use the main bicep file g and i've got this user a resource group that i use for a variety of different um different generic resources that i use across multiple um multiple resources for a variety of different projects so i'm just going to go ahead and deploy it into that there you are go back into visual studio code i'll just copy that so valida azure engine rg let's hit enter and see what's happening so that will give it okay so new bicep releases available you can upgrade it using the asic bicep upgrade and here we've got running so the idea is if i go back into if i just go back into that resource group if i look at let me just zoom in to you as well oh i can see my valida data factory there already but if i look at deployments so this is the one that we just kicked off so it looked like it succeeded let's seriously take one second okay that's that's pretty cool so if i go to that resource here's my azure data factory already and raring to go the idea is if i just go into author and monitor i should be redirected to my azure data factory ui it looks pretty good so far let me just check a couple of things so if i just go to essentials i can see it was deployed to the location australia east it was deployed to that resource group that i provided in the azure command line interface command that i ran you can pass other parameters as well so i could have passed in a little could written up my bicep code in such a way where i could actually provide the location uh within that an az cli command i go into my azure data factory suite that's everything all set up so i can create pipelines data flows um all i've done is really set up the the data factory instance i haven't set up any um code repositories with it i haven't defined any workflows if i just go back to this template reference there's a bunch of different things that we can actually do of it so if i just go back into data factory here you can see we can define some data flows data sets integration run times some linked services i can do all of that using bicep code but all we've really done is just set up azure data factory using just nine lines of bicep code which is really pretty pretty cool i think okay so yeah i hope you enjoyed that video well i hope you enjoyed this um again like i said we kept it really really simple and there's obviously quite a bit more that we can do when it comes to using bicep and writing up our infrastructure code and obviously this is really really basic um we can again define parameters to different files and we can change that depending on which environment we're going to be deploying it to but hopefully the main takeaway that you get out of this is we can write some really simple bicep code um and use it to deploy azure resources such as azure data factory um i hope you guys enjoyed this video if you have any questions feel free to pop them in the chat if you want you can like and subscribe i don't know where the icons are youtube keeps moving them around so i'm not going to do the whole hand point thing that people do at the end of their videos um hope you guys are staying safe and i'll see you all next time

Channel: Will Velida

Views: 2,053

Keywords: Azure, Azure Data Factory, Bicep, IaC, Software Development, Software Engineering, Tutorial

Id: mtn58qKwJxc

Length: 14min 56sec (896 seconds)

Published: Sat Jun 05 2021