Go Programming - JSON Encoding & Decoding in Golang

handling Jason is a vital part of development nowadays and in this video we're going to dive into the utilities that are offered by go when working with Json data we're going to see how to decode the Json that's coming in from an API into gold structures and we're also going to see how to use Json tags within those structures and we'll also see the opposite process of encoding a goal structure to adjacent data and that data that you encode can then be sent over the network and consumed by other applications and finally we'll dive into some final details such as the decoders disallow on known Fields function as well as private Fields within a structure and also how to exclude certain fields from being serialized to a Json when you're performing that encoding process so let's get started I have the documentation here for the encoding Json package which is built into goes standard Library we're going to work with this package throughout the video and we're going to see some of the functions and utilities that it offers and we're also going to work with the Json placeholder API which I'll leave a link to below the video and we're going to work with this to Do's API it's a dummy API that's provided by this website and it returns a to-do containing some dummy data and you can see the fields that are returned here the user ID and ID field as well and also the title and completed what we're going to do in this video is create a goal structure around this data and we're going to see how to send requests to this API and then convert the Json data that we get back in this format into the go stopped and that's where we're going to start in this video so I'm going to open the vs code and we have a very basic file here called main.go and this is in the main package that we've declared and also we have our main function here so let's start by declaring a URL where we want to send an HTTP request in order to get the Json data from this API so let's copy this API endpoint and we're going to paste that in as the value of this URL and once we have that URL we can use the go HTTP package and we're going to send a get request to the above URL which is going to return a response and potentially an error as well so we're going to send a get request to the URL I'm we save the fail that's going to bring in the net HTTP package from the go standard Library so now we potentially have that response what we're going to do is check to make sure we don't have an error so if the error is not nil then what we're going to do is we're going to use the log dot fatal function and we're going to pass the error into that function and what log dot fatal does if we just hover over here it's equivalent to just printing out the error and then calling the OS dot exit function so the execution of this program will stop here if indeed there has been an error if there's not been an error what we want to do is go down here and we're going to get that response content and explicitly what we're going to do is read the body from the response of this get request so what we're going to do to start with is we're going to defer a particular function and that function is response.body.close so when we read the response body what we're doing is we're reading that content and we need to explicitly close that when we are done reading to prevent things like memory leaks and we can defer the closing of of this and what that means is when this function completes when it returns any deferred functions such as this one are going to be executed at that point and this is important because if we write any further code in this function below and any errors or exceptions occur that cause the program to exit without this body being closed that can as I said cause a memory leak in your application let's now go on and write the code to actually read the content from the body so we're going to use an if statement here and we're going to check the response status code and if the responsive status is equal to the HTTP dot status OK response that means we have a response here and we're going to go into this if block and we're going to read that response and we're going to do that using the i o module in go so let's get the bytes from the body with a variable or body bytes and we're also getting the error and what we're going to call here is the io.read all function and we're going to pass the response dot body into the io.redol function in order to read that into this variable of body base and again when you save this it's going to Auto Import the i o package from the standard library and now we can process these bytes that we've got from the body now again this might cause an error if there was any issues reading the body here so we're going to paste this code in where we check if the error is now and we exit the application if we indeed have an error if we don't have an error though we are going to set up a variable called data and let me scroll down here and we're going to use the string function in go I'm going to pass the body bytes into that string function and then we're going to use the format module to print these out as a new line to the terminal so we're going to print out that data and before we execute the program remember the Syntax for declaring a variable like this we have a colon and an equal sign so that was my mistake what we can now do is bring up our terminal just below here and I'm going to execute this program with the go run command and the file that we're passing in here is the main.go file so let's see if this works and you can see the response that we're getting on the terminal it contains the Json data that's been brought back from the API and after we send the get request using the http.get function we are then reading the response body when we get that response and what we're doing is we're converting it to a string and just printing it to the terminal so that is pretty useful we're getting back the data as a string but let's say that we want to model this with a structure that contains fields and that will give us a better representation of the data within the application so what I'm going to do is make the terminal a bit smaller here and we're going to scroll up to just above the main function I'm going to define a new structure up here and because this is a to-do API let's call the type to do and that is going to be a struct and then we can declare indestruct whatever fields we want to be part of this particular structure so we have a user ID from the API and that is our type integer we also have an ID again that's an integer title is going to be of type string and the final field is completed which is of type Boolean now again when we save this we're going to get some Auto formatting from go and you can very cleanly read the different fields that are in this structure as well as the data types that are associated with those fields now we can also add something called Json tags to the structure and the Json tags can be used to determine which fields in the Json map to which fields in this structure and vice versa now you can see I'm declaring this variable called user ID if we look at the Json placeholder we can see that it has a lowercase first letter now we can explicitly tell go that we are going to be reading this particular field if we go back to the structure here and tab over we can use the backticks and we can declare these Json tags here and for this particular tag it's going to be user ID that's going to match what we have on the API here and we can do the same for the other three Fields as well if we go back here note that we do need to surround the tag itself with the double quotes and we can copy these tags for each of the other fields for example the ID field it's just a lowercase ID coming back from the API and we also have a title field here and that's going to be title and also a completed field as well and when we save that again some Auto formatting is applied so each of the fields in our struct now has a Json tag that tells the Json package a bit more information about what the key should be when it's serialized to Json and when the data is coming from Json now that we've written that we can write some more code just below here and instead of Simply converting the response to a string what we're going to do is decode the response into that to do structure that we've just created now to do that let's start with this function here it's the on Marshall function and if you're not familiar with this terminology you can just read the documentation and as it says on Marshall it parses the Json encoded data and it stores the result into the value that's pointed to by the parameter V and this is what you would use to pass a Json encoded string such as the one we're getting back from this API into a go structure we would provide that the response data the bytes as a first argument here and then the second argument can be a pointer to a particular structure that we want to decode that data into if we go back to vs code here let's remove these two lines of code at the bottom what we're going to do now is create a variable called to do item that's going to be representing where we're going to store this data that's coming back and we're going to instantiate or initialize an empty structure here what we can do underneath that is use the Json dot on Marshall function and like we saw in the documentation that takes some bytes as the first argument I'm going to pass that body bytes that we're extracting from the response and then we can pass in a pointer to the empty to do item that we created above and that is then going to take the bytes that have been extracted from the body and it's going to try and create a structure and fill the fields that are in that structure with the data that's coming back from the API so if we scroll back down here just below the on Marshall function I'm going to print to the terminal this statement here where we look at the data from the API and this syntax here is going to print the structure and it's going to print each field in this structure to the terminal when we run this application so let's go back to the terminal and again we're going to run the go run command and we're going to pass min.go to that and if we make the terminal a little bit bigger you can see we get back a different format now we're getting the data from the API and you can see the structure containing the user ID which is set to one the ID which is also one and we get this title here and completed is equal to false if we go back to the Json placeholder API this data now matches the data that we have here and if we fetch a different to do by changing the URL to number two here you can see the ID has changed to two and we get back some different data so if we go back to the application and change the url to fetch this second to do and rerun we now get back the second to do it below here so we're able to fetch these to those and we're able to convert them into a go structure that we've called to do here and once we have that structure you can then process the data in your application in the normal way that you would use a structural angle and that's a lot better than working with that data as a string now we're about to see a different way of writing this code that's a little bit shorter so I'm I'm going to remove the cord at the top here and we're going to create this empty to do and we're going to remove the Json dot on Marshall function and replace it with a call to the new decoder function and as it says the new decoder function returns a new decoder that reads from the parameter that we pass in are and that has to be an interface that matches the io.reader interface and it turns out the response.body is something we can pass directly in here and we need to make sure we call that response so the new decoder function is going to take the response body and it's going to decode that Json and what we can then do is call the decode function and as it says decode reads the next Json encoded value from its input and stores it in the value pointed to it by V so here we need to pass in a pointer to a structure which is the to-do item that we created above so let's pass that in to the decode function and again we'll print that to do item to the terminal below so let's clear the terminal and rerun the application and you can see we get back the same data as we had before but this time we've done it in three lines of code we initialize an empty structure here for the to do we then use the new decoder function in the Json package and pass the body of the response to that function and that will return a new decoder which has this decode function and we can pass the pointer to the structure that we created above that's going to then decode the string from the Json data into that structure now one benefit of the new decoder function is that we have some additional methods that we can take advantage of that we didn't have with the on Marshall function so if we go to the Json packages documentation we have the type decoder here and remember that's what's returned from the new decoder function and you can see all of these receiver functions that are defined on the decoder now the function we're going to look at is this one here it's called disallow unknown Fields so let's go back to vs code and we're going to see an example of this now but before we do it what I'm going to do is comment out one of the members of this structure so let's comment out the completed member now if we rerun the code below what we get back if we look at the response as a structure that contains the user ID the ID and the title of course we don't get the completed field because we've removed that from the structure but importantly the program does run so even though we have a response body that contains some data we didn't expect from the structured Fields the application will not crash it will run it just won't look at data that's not part of the structure now we might not want this we might not want our application to process a response that contains data that we don't expect so let's scroll down to where we're defining this code and what I'm going to do is I'm going to remove the call to decode or rather I'm just going to bring it below here and what we're going to do is we're going to set up a decoder variable by calling the Json dot new decoder function and passing the body once we have that decoder we can call the receiver method on that so it's the disallow unknown Fields function that can just be called on that line of code and then below that we can use the decoder that we're getting back and we can call the decode function and pass the pointer to the structure and we're going to surround this with an if statement here so if we don't get an error back when we call decoder.d code then we're going to process the line below where we print the structure to the terminal but what we can do is put our colon here or a semicolon and then we can check if the error is not equal to nil and if it is not equal to no then we're going to again call log dot fatal and I'm going to pass a string in here decode error and we can also pass in the error itself so let's just quickly go over this we're calling the json.new decoder function and this time we're storing the return value in a variable called decoder and then we take that decoder and we call the decelo unknown Fields function and finally we take the decoder we call its decode function and pass in the pointer to the structure and we're checking here if we get an error in that process so remember at the moment we have this completed field commented out so if we go back to the terminal and rerun this code you can see that this time we are getting an error and that's that we have an unknown field called completed and we have the exit status of one from the log dot fatal function so this time because we have an unknown field coming back from the data and that's the completed field that we've commented out of the struct this time you're getting that error because we have called the disallow unknown Fields function on the decoder so that's how you tell your application that you do not want to accept a response that contains data that you're not expecting now if we go back up to the structure and uncomment this completed and bring it back into the structural definition if we rerun the application even with the disallow unknown fields we are now getting back the response and that's because the response body contains all of the data that we expect when we decode it into the to-do structure so let's move on to something different now what we're going to do in this structure is we're going to show that go has certain rules about the casing of its Fields within the structure now it turns out that you can have private Fields within the structure and a private field will start with a lowercase letter in go and what it means when it's a private field is that it is not visible to anything outside of the package where the structure is defined and you can see now that we have this yellow line under it we have a struct field called user ID which has a Json tag but it is not exported so we have Json tags which means that we are trying to convert this to Json but we have a private field and because the Json module is not part of the main package where we're defining this it's not going to be able to read this and decode it to a Json so what I'm going to do is remove this Json tag here and we're going to save the structure and if we scroll down what I'm going to also do is remove the call to disallow unknown Fields I'm going to try and run this application as it is now and see what happens here with this private field so let's run the code and you can see from the response here that we have a user ID but it's set to 0 here and that is the default value of an integer field in a structure so for example we don't have a value being passed in from the body because the Json module is unable to do that because this is a private field if we change it back to uppercase you can see when we rerun the code that we're going to get a user ID below here here that is correctly coming back from our jsonplaceholder API so this highlights that if you have a response and you need your data to be converted into fields in a structure you need to have these fields start with a capital letter so the casing is important when you define fields in a structure if you want to define a private field you can Define that field for lowercase first letter and that might be useful if you have things are used internally within the goal structure but in the context of Json when you have a structure that represents something you want to convert from Json data into go you need to Define these with the capital letters if you want that to take on the value from the response so that's a little bit about private fields and goal let's now move on to encoding a goal structure to Json now of course if you have data objects like to do here it's very useful to convert these to Json and then send them on over the network to other applications for example you may have react apps or vue.js apps that are consuming that data and showing it in a user interface so the process that we're going to use is the opposite of decoding Json data into a structure it's now going to encode the structure to adjacent string now decoding we also saw the on Marshall function when you encode the data that is called marshalling that data so let's make the terminal smaller here and after we've read the data and decoded it into our to-do item structure what I'm going to do is remove this printf statement and we're going to convert this data back to a Json just to show how you would take a goal structure and convert it back to Json and later we're going to see how we can omit certain fields from the data that's serialized so let's get started now there's a function called json.marshall This is the opposite of the on Marshall function and this will return as it says the Json encoding of the argument V that we pass in so we're going to pass in the to-do item and that's this to-do item here that we populated within the decoder.d code function now the Marshall function as the signature says here it will return our bait slice and an error so we need to handle that now so we're going to call the bait slice to do and also we might get back and error here and we need to check again if the error is not equal to nil and in that case we will call the log dot fatal method again so let's copy this code down to this section and the final thing I want to do is use the format dot print line function and we're going to again convert the byte slice to a string by using this string function so let's save the file and we're going to rerun this application in the terminal and you can see the data that we get back now we have a user ID that's set to one we have the ID of 2 we have the title and the completed so all the fields in the structure are here we've taken the data from the Json placeholder API we converted it or decoded it into our structure and then we have encoded that structure back to Json now one thing you might note here is that all the fields have the lowercase representation the ID the title and completed but user ID here has some uppercase characters now the question is why is this happening if if we go back up to the structure you can see that before we removed the Json tag from user ID so we don't have a value that the Json module knows to encode the user ID to so it's just going to encode it using whatever the name is in the structure itself so to fix that I'm going to tab over and we're going to Define another Json tag here and let's use camo case for what we want to convert this to in the serialized Json data let's now rerun the application and hopefully this time we're going to see that the key and the return data now matches that Camel case format and you can see that it does now here we've used the Json dot Marshall function what we can also do is pass another handy function in here and it's the Marshall indent function and that takes in not just the item that we're serializing to Json but it also takes in a couple of extra arguments we have a prefix and we can just set that to an empty string and we also have the indentation that we want to set in this data so when we serialize the data we want to indent all the fuels using the tab character so we can use this character here let's now rerun the application and we're going to check the output here now we have a bit of a cleaner representation of the serialized goal structure and we can very clearly see the different fields as well as their values so it's worth knowing that the Marshall function has a marshall indent sister function that will change the formatting of your output let's now see how we can use private Fields within the Json tags themselves what I'm going to do is go down to the main function here and I'm going to remove all the code that's sending an HTTP request and looking at the response body so I've now cleaned up the main function where we are creating an empty to do we're using the Marshall indent function and we are printing the result to the terminal but now we are not performing any requests to the API what I'm going to do is directly create the to-do here with some values so let's do that now we're creating a to-do and we're going to set the IDS equal to one let's give it a title of release video on all and finally we can give a completed status of false so let's save that code and run the program below and you can see we get back the indented and marshalled data so let's now change some of the fields in the structure so that they will not be converted to Json we already saw how to create a private member which go will keep private within the package but you can actually explicitly tell the Json module not to include a particular field and the response or rather in the serialized representation of that data and you can do that by specifying this syntax here where we just put a dash as the tag for the Json field so I'm going to do that for the user ID and also for the ID field as well and let's now try and rerun this code and we'll see what output we get so if we make this bigger you can see that now we only have the title and the completed fields in the Json data and that's because we have now specified these Json tags here for the user ID and the ID and that will essentially make the these private fuels that are not included in the serialized Json and I'm going to go to a digitalocean article on this and there's a header here called ignoring private Fields And as it says here if you have sensitive Fields you might like the Json encoder to ignore that field entirely and that is done as it says using the Special Value that Dash as the value argument to the Json tag and you can see that they've done that for a field called password within the user structure and of course if you have sensitive data like that you would not want to serialize that to Json and return that as any kind of response so you can use the dash to indicate that this is a field that should not be serialized to Json when you call the encoder functions so these are not private Fields with the lowercase first letter as we saw before those fields are not actually exported from the structure so they can't be read by anything outside of the package that they're finding but in this case the field is exported from the structure other packages can read the password field it's just not serial lies to Jason when you call the encoding function so that's the subtle difference there between these two methods and just to wrap this video up I'm going to show one other technique here now what if we have an empty value for a particular field let's go down and we're going to remove the title that we have for this structure that we're initializing and we're going to set that equal to an empty string if we run the code again we're going to see the output below and you can see we get back the title with an empty string so what if you don't want to include these empty values in the output well the Json tags contain another flag that you can use here and that's the omit empty flag so when you specify the Json tag for the title field here we're giving it the name that we want in the serialized output but we also add this omit empty flag and that is going to tell the Json module that if the value that we're passing in has an empty value if that's the case then we are not going to include it in the output so let's try and rerun this program and we're now going to see that title field is not included in the data so this structure here shows some features that we can use when we want to control what is output to adjacent from a go struct if we want to denote a field as private and we want to always exclude it from the outputed Json we can use the dash as the value for the Json tag and if we don't want to include a field that has an empty value we can use the omet Mt flag within the Json tag in order to do that so that's all for this video it's been an introduction to the Json module we've seen how to decode Json data from an API into a goal structure we've also seen how to encode a goal structure to Json and we've also seen how to use private Fields how to exclude a field from the output to Json and also how to exclude Fields when they're empty and finally we've also seen the disallow unknown Fields function this can be used to cause an error when you get data that you're not expecting from an API response so that's all for this video if you've enjoyed it please like And subscribe to the channel and we'll see you in the next video

Channel: BugBytes

Views: 6,519

Keywords:

Id: Vr63uGL7NrU

Length: 25min 3sec (1503 seconds)

Published: Mon Jul 17 2023