Sending Cosmos DB Diagnostic Logs to Log Analytics with Terraform

hello everyone my name is will valida i'm a software engineer at microsoft data platform mvp and in this video we're going to take our existing azure cosmos db account and send diagnostic logs to log analytics using terraform diagnostic settings in azure are used to collect logs about resources azure resource logs are admitted by resources that produce data about the operations of those resources these are captured per request the content of these logs will vary by resource type activity logs and platform metrics are collected automatically but to collect resource logs we have to create diagnostic settings we can create these diagnostics settings and send them to log analytics event hubs or storage accounts cosmos db has a variety of log categories that we can monitor some of these are available across all apis such as data plane requests that capture requests that are executed to create update delete or retrieve our data others are more api specific such as query runtime statistics which detail query operations that are executed against a sql api account for more detail on these log categories i've checked out the article that i've posted in the description now in our demo we will create our diagnostic logs for our cosmos db account and send them to log analytics i already have a logs at log analytics workspace created so in order to send my diagnostic logs i'm going to be using a terraform data source data sources allow terraform to use information to find outside of terraform or defined by another terraform configuration providers in terraform may offer data sources alongside its own set of resource types a data block in terraform will read from a given source and export the results under the given local name we will then use the local name to reference the resource in the same terraform module so in this demo we're going to use a data source in our cosmos db terraform file to import our log analytics workspace we'll then create our diagnostic logs in cosmos db that will be configured to be sent to our log analytics workspace we'll then deploy our changes to azure so here i am in the azure portal and as you can see we can actually set up our diagnostic settings in our azure cosmos db account right here in the portal so we can decide what metrics we're going to send and where we're going to send them to now we're going to do the center form but as you can see here we can send it to our diagnostic settings to log analytics we can send it to a storage account we can stream our logs to event hub or we can send it to a partner solution but for this demo we're going to send it to log analytics using our terraform code let's have a look at the terraform docs to see how we can create our diagnostic logs so in this example here we can see that our diagnostic settings has a name and it has a target resource id and looking further down at the argument references here we can see that these fields are required looking a little bit further down we see that we can also provide our diagnostic settings with a log analytics workspace id which specifies the id of a log analytics workspace where the diagnostic data should be sent now if we jump into the terraform docs for log analytics we can see by looking at the attributes reference here that we can retrieve the id of this workspace when we import it into our terraform file so in order to make this work we're going to use a data module to reference our log analytics workspace and get and retrieve the id so i'm going to go ahead and go into visual studio code and just at the top of my file just before i even declare my azure cosmos db resource i'm going to create my data module here use the azure rm log analytics workspace give that a name of log analytics and in here i'm just going to give it a name which will be the name of my log analytics workspace and then the resource group name which is where my resource group which will be the resource group where my log analytics workspace has been provisioned i'll just use variables for this not being very consistent with my variables that's okay for this make this type equal string and i'll get the default value a little bit later cool now what i'll do is just head into my azure portal i'll actually search for my log analytics workspace so this will be the name of my log analytics workspace go back into my variable file stick it in there and i'll just get the resource group as well stick it in there so now that we have our log analytics workspace imported into altera for well referenced in our terraform file as this data module we can actually go ahead and start to create our diagnostic settings so create a resource block of azure rm monitor diagnostic settings i'm just going to give this a name of cosmos db diagnostics cool i'm going to give this a name i'll use a variable for this just say cosmos log name or settings settings name that's better uh we need to actually target it target this towards our cosmos db account so i'm going to use my cosmos tv account that i defined earlier and use the id see i need to use the target resource id so i'll use the resource id of my cosmos db account and then i'm going to use my log analytics workspace let autocomplete do it for me so for my log analytics workspace id i'm using the data module of the azure log analytics workspace that i've imported into my terraform file and i'm using the resource id as the log analytics workspace id so now i'm in a position to create my logs that i'm going to send to my log analytics workspace so in order to start creating our logs i'm just going to go back into the terraform documentation as you can see we've got this uh sub um block of code here for log so if i go down a little bit so we can provide one or more log blocks as defined below so scroll down a little bit so we're going to need a couple of things we're going to need to provide a category which is the name of the diagnostic log category for this resource and again this will depend on the resource um that you're um that you're using in your terraform file so we'll look at the cosmos db documentation to see how which logs we need to send to log analytics we can also set up retention policies and whether this is enabled or not now we're going to be enabling our log so we're going to set it to true so we can actually send logs to our log analytics workspace this retention policy is really good if you're sending your logs to a storage account but we're not going to be doing that so we'll just leave that one out for now so if i bring up the cosmos db documentation on diagnostic logs let's choose the log categories that we're going to be sending to log analytics now remember depending on which api you've enabled for your cosmos db account um some uh log categories will be specific to particular apis and we've created a sql api account so for this demo i'm going to send data plane logs i'm going to do a queue runtime statistics petition key statistics petition key ru consumption statistics and control plane requests so what i'm gonna do is just go back into visual studio code i'm going to cheat a little bit here and just copy and paste something i've done earlier sweet so now that what we've done here is we've just essentially copied and pasted all of these log categories and we're enabling them so when we actually provision this we'll be enabling these log categories to be sent to our log analytics workspace so that should be everything that we need to send our logs to our log analytics workspace let me just check i've got that cosmos tv log settings name i don't so i'll just create my variable for that make this a type of string i'm going to give it a default setting of cosmos log settings there we go so now we have everything that we need to um send our logs to log analytics let's start to deploy this so what i'm going to do is just open up a new terminal and i'm going to run terraform emit to initialize my terraform configuration i'll initialize my back end initialize the provider plugins cool so that's been successfully initialized i'm now going to validate my terraform code running teleform validate looks like everything is valid so now i'm going to plan my execution by running terraform plan so this will generate my execution plan and show me what's going to be deployed cool so we've got free resources that are going to be added so we can see our log categories our diagnostics are going to be created and i've deleted my database and container from the previous one um but since i haven't removed it i've deleted it manually but since i didn't remove it from our terraform file it's going to create it again but that's okay we don't mind that the thing that we're looking out for here so our log analytics workspace id so there's the log analytics workspace that i've created before so that's what it's going to send our diagnostic settings to we've given it our name and also there's our cosmos db account as our target resource id and here are the logs that i'm going to send or configure to send from cosmos db to my log analytics workspace so that looks all good so now we just need to apply our configuration and this will generate our plan again and to confirm our plan all we need to do is type yes once it's generated the plan again it's always good idea always a good idea just to verify just before you type in yes you don't want to deploy something that you don't mean to or even remove or change something you don't need to but for this we'll just enter yes and let us apply cool so through the power of video editing we can see that happened really really quickly and our resources have been successfully applied so if i go into my cosmos db account i scroll on down to diagnostic settings i can see that my cosmos log settings has been created so if i just click edit i should see all of the logs that i enabled in my terraform file are being sent to my log analytics workspace so i just remove my terminal so as you can see in just a few lines of code we've actually enabled logging on our cosmos db account and sending those logs to log analytics workspace using a data source without having using a data source instead of actually having to define or create a log analytics works workspace within the same terraform file that's the power of the data source module here we can actually refer to other bits of infrastructure that haven't been necessarily created in terraform or it's been created in another terraform configuration we can actually reference those resources that have already been created in other terraform files such as this one which i think is really really powerful cool so that's the end of the demo let's recap so to recap in this video we learned what diagnostic logs are and what kind of logs we can generate from azure cosmos db we then learnt what data sources are in terraform and how we can use them to import resources that haven't been defined in our terraform configuration files and finally we learned how we can send diagnostic logs from azure cosmos db to log analytics using terraform so that's the end of the video thanks for watching i hope you found that useful um as always if you have any questions leave them in the comments if you like this video give it a like and make sure you subscribe for more content coming in the future i have a bit more time on my hands these days and i'm trying to generate more video content so if you have any suggestions or topics that you'd like to see me cover also yeah just let me know in the comments so thanks for watching and i'll see you all next time

Channel: Will Velida

Views: 74

Rating: 5 out of 5

Keywords: Azure, Azure Cosmos DB, NoSQL, Database, Data, Software Engineering, Software Development, Tutorial, Terraform, DevOps, Analytics, Technology

Id: 09L0vOxtPps

Length: 15min 14sec (914 seconds)

Published: Thu Sep 09 2021