What's Wrong With The Terraform Remote State Data Source?

when you're trying to chain together multiple terraform configurations the remote State data source might seem like the perfect solution but Hashi Corp recommends against using it why and what should you use instead that is what we'll cover in this episode of terraform [Music] [Music] Tuesday what's up everybody I'm Ned bellance Ned in the cl.com and welcome to another installment of terraform Tuesday like I said in the introduction we're going to look at the remote State data source and why Hashi Corp recommends against using it we'll also look at some of the Alternatives you can use instead but first two quick things one I want to thank spacelift for sponsoring this episode of terraform Tuesday more about them at the break and number two I want to recommend a newsletter for all you terraform fans out there it's called terraform weekly or weekly. TF and it's put together by the excellent Anton babenco if you like to stay on top of the latest terraform news and open- Source projects I highly recommend subscribing I'll put a link down in the description below now let's get down to business what does the remote State data source do it's all about sharing values between configurations when you first start building out your infrastructures code there's a good chance that you'll put everything into a single configuration and a single repository and that's okay you're just learning terraform you're getting the lay of the land and figuring out what goes where but as you're use of terraform matures you'll want to start refactoring that code and one of the first things you'll end up doing is breaking things into modules this allows you to reuse code across multiple configurations you can take that module put it in a terraform registry and let others make use of that code inside the module you can enshrine best practices add some scene defaults and version your module independently of the configurations that use it it's pretty cool stuff the next step in the process is to break up your monolithic configuration into multiple configurations each with their own repository and instance of State data why would you do that there's a few reasons I'll outline here but you should also check out Lee Brigg's excellent blog post on the subject first of all it allows you to group together Resources with a similar change rate meaning that resources that seldom change won't be affected by resources that change frequently this can limit the blast radius of Any Given change the second reason is to group things by life cycle for instance network security groups have Network in their name but but their life cycle is usually tied to the thing they're actually securing so you might have a network configuration that a bunch of things share but your network security groups are going to go in the configurations for the things they're securing so you might have a network security group for your web servers and another one for your database servers you would put those in separate configurations that include your web servers and your database servers the people responsible for managing say your web servers are probably a different team than the database Folks by breaking up your infrastructure as code into separate configurations it allows you to separate the responsibilities of managing those resources to the teams that actually manage the resources so if my web team wants to make a change to their infrastructure they don't have to consult with the network or database team about the changes they're in different configurations the web team can just make the change to their config fig and push it through their cicd pipeline there's probably lots of other reasons to break up large configurations including the time it takes to refresh State and the time it takes to apply changes but I think you get the idea breaking up configurations is a good idea but it also introduces some challenges one of which is how do you share values between configurations when it was all one monolithic configuration you could simply reference the attribute of another resource or module that's in the same config but now they're living in separate repositories and SE separate State data how can my web team get the v-net ID that's stuck in the network configuration that's where remote State data sources come in the remote State data source is one of only two resource types that are part of the built-in terraform provider this is the only provider that you don't have to download plugin for it is literally just built into the terraform binary the other resource type is the terraform data resource and if you're curious about that I have a whole other video about it the remote State data source allows you to pull values from another configurations State data when you configure a remote State data source you gain access to any outputs that are defined in the remote configuration you can then use those outputs in your configuration let's take a look at an example to see how this works okay in vs code I have my terraform Tuesday repository open and we'll be working in the directory 2023 1024 remote State data now here in the network folder in the main.tf I have defined a resource Group a virtual Network and a subnet below that I've also defined outputs for the resource Group name and the subnet ID now I'm using local state for this example but it would work the same for a remote State backend as well I've already deployed the resources so if I pull up the terminal and I run terraform output at the command line it prints out the resource Group name and the subnet ID this is the kind of information I might want to share with a configuration that uses this network speaking of which over in the app folder I have just such a configuration in the main. F I am creating a resource Group and a network interface now I want to place that Nick on the subnet I created in the network configuration I can do that by using the remote State data source to get the subnet ID in the remote data source block I specify the backend type I want to access in this case local and for the config argument I specify the path to the state file the arguments for the config portion will be be specific to the backend type specified and since I'm using local state it's just the path to the state file once I've configured the remote State data source I can access the outputs from the remote configuration inside my network interface block for the subnet argument I can use the Expression data. terraform remotest state. network. outputs. subnet ID to retrieve the subnet ID that's stored in that output remember that outputs are the only values that the remote State data source exposes as attributes if I run terraform apply it will create the resource Group and the network interface and place the Nick on the subnet I created in the network configuration that's pretty cool but there are some problems with the remote State data source let's take a look at those the configuration that has the remote State data source requ requires read access to the remote State data in question now unfortunately there is no native way to Grant access to just the outputs of State data the process running your configuration will have read access to the entirety of the source State data one thing you're probably aware of is that state data can contain sensitive information things like passwords API keys and application secrets you don't want to expose that data to just anyone so you need to be careful about who has access to your state data even if you got your state data encrypted at rest and in transit by granting the target configuration read access to your state data you are exposing the contents of that state data to the process running the configuration if that process is compromised the attacker could gain access to your state data and all the sensitive information it contains in the world of information security we try to follow the principle of least access meaning only Grant access to the things that are absolutely necessary for a person or a process to accomplish its goal the remote State data source doesn't allow you to do that that is the main reason hashicorp recommends against using it another issue that's not brought up by hashicorp but I think is very important is now you've tightly coupled your configurations together if you make a change to the source configuration you have to make sure that you don't break any Target configurations that reference it if you want to refactor or break up that Source configuration you have to make sure that all the dependent configurations are updated as well and that can be a real pain so what are the other options let's take a look after the break spacelift is an infrastructure as code platform that helps you automate your terraform workflows it's a hosted solution that provides a web UI for managing your terraform configurations and it integrates with your version control system to automatically run terraform plan and apply when you push changes to your repository H that sounds pretty useful even more than that spacelift includes a policy engine that allows you to Define policies that will be enforced when running terraform plan and apply this allows you to enforce best practice and security policies across your entire organization and it can even prevent you from deploying infrastructure that violates those policies space lift also takes care of Hosting your state data includes a private registry for your modules and supports Dynamic credentials for the major cloud service providers managing multiple environments with terraform as we've seen is a complex task and spacelift helps you focus on managing your environments and not the platform if that sounds interesting to you head on over to spacelift doio to learn more and sign up for a free trial I've included a link in the description below and thank you to spacelift for sponsoring this video if your primary concern is granting read access to State data and you happen to be using terraform Cloud then you have access to the TF outputs data source as implied by the name the data source allows you to pull output from a terraform cloud or terraform Enterprise workspace when you grant permissions you can restrict it to read access of only the outputs and not the entirety of the state data this is a great option if you're using terraform cloud or Enterprise but it's not available for other remote state backends it also doesn't address the second issue of tight coupling between configurations instead you can choose one of many options for sharing config configuration values depending on the cloud you're most invested in you can use something like Azure KEYT to store your sensitive values and Azure app config for less sensitive things the source configuration would write those values to keyal or app config and the target configuration would read those values if you want to refactor or change the source configuration that's fine just make sure it's still writing the values to the same key Vault or app config and the target configurations will just continue to work now if Azure isn't your bag then there's similar services on all the major clouds and even some independent Solutions you can use AWS Secrets manager or parameter store Google secret manager or Hashi Corp Vault and console you could also use a database an object store or even good old DNS as long as there a there's a terraform provider and a data source that supports it you can use any of these options and a bonus feature is that other tools that aren't terraform can also read from these services so your application code can read from the same KEYT or app config that your terraform configurations are using this is a great way to keep your secrets in one place and avoid [Music] duplication so there you have it the remote State data source is one way to share values between configurations but it's not without its problems Hashi Corb recommends against using it because it requires access to the entirety of State data I'd also add that it tightly couples your configurations together and it prevents other services from Reading those values instead you can use a variety of other solutions to share values between configurations check out the example code in my terraform Tuesday repository for some examples that use app config I'll put a link in the description below that's going to do it for today's terraform Tuesday like I mentioned at the Top If you're looking for more terraform content check out Anton Banko's terraform Weekly Newsletter I'll put a link in the description below and if you're a vendor who wants to sponsor an episode of terraform Tuesday reach out to me on LinkedIn or on my website ninthe cloud.com thanks everyone for watching until next time stay healthy stay safe out there bye for now but I really got right up on the mic didn't I so I got this cool Alpha tester sticker while I was at hashy comp what was I Alpha Testing well I can't tell you exactly what it was but it has something to do with terraform

Channel: Ned in the Cloud

Views: 2,268

Keywords: terraform tutorial, infrastructure as code, terraform remote state, terraform state file, remote state, terraform state file management, remote state terraform, infrastructure as code (iac), infrastructure as code azure, infrastructure as code with terraform, remote state file terraform, terraform tutorial azure, terraform remote state data source example, terraform remote state azure, terraform remote state file

Id: YRsqXppXu3s

Length: 14min 48sec (888 seconds)

Published: Wed Oct 25 2023