Building Microservices with Go: 6 JSON Validation

[Music] hello and welcome back to billing microservice with go and episode six and I'm going to apologize because yeah I got a four plane and I've got a little bit of a cold so if I sound kind of like Barry White with a cold then that's why today's video I'm gonna do a little a little bit shorter maybe the normal and I do apologize but you definitely know want to be listening to me croaking for too long but what we're gonna look at is kind of adding some validation to our API that we've been building so we're gonna start looking at a validation system and we're going to use a tool called go validator validator just allows us to add strut validation and then things like that you'll see how all of this works but it's it's really nice for for kinda you know just doing this this kind of stuff adding is a nice validation to our API is make them a little bit user friendly but also got to think about ourselves so kind of one of the most common things and you will find when when looking at vulnerabilities and if we look at say a wasp's website you'll see that well let's zoom matter but the top ten vulnerabilities on a wasp and I'm just looking at a blog post here from CloudFlare but we've got things like injections so sequel code injection data explosion broken access control security miss configuration cross-site scripting insecure D serialization so things like that and combined with being able to do injection vulnerabilities around that it's really really important that you sanitize your your data input so too good too good reasons why validate one actually makes your application secur because we're validating and sanitizing our input and number two well it gives us a really nice way of kind of managing our data strokes and also been able give nice error messages back to people so let's go so what we're going to do so this is the example that we were we were looking at last time and again I'll put the the link down there and if you missed last week's video I'll put something up there but we've we've got this kind of data model and and again just to kind of recap for folks this model isn't actually connected up to a database we're doing some some pretty basic stuff here we've literally just got a list of object but in terms of how it would function pretty much exactly the same as if I was wiring this up to a database and defining my my strut which is my my data type and I'm just adding some methods to return the lists of products add products and things like that so I'm writing an abstraction around the underlying data implementation so this is the the product that we we would like to to validate so how do we do this so what we're going to do is we're going to use it very basic we're gonna use some strut tags and we can specify which are valid fields and and things like that what we will also do is we're going to add some custom validation types and I think it's gonna be pretty pretty neat so we're gonna do all of that with go validator so let's take a look at go bad little data and again I'm gonna put the link down here so you don't need to worry about this too much right now so go validator so it's very basic what we we do is we have this validate method and we guys through strut method on the validator and we passiveness and so our strut and what it's going to do is it's going to return us a list of errors which relate to the validation on that stroke in order to to kind of determine what those validations are well we can we can just use these tags here so we can say for example we can add the validate tag for required so in this instance the user strut first name will be required say my last name but we can also do some more complicated validation so for example with age what I'm saying is that ages a an unsigned integer so it's a positive number and it's going to be greater than zero GT it's greater than equal to zero or it's gonna be less than equal to a hundred and thirty for validating say an email field well you know the validator also has this this special strut type called email and we can validate this string corresponds to the correct RFC email type and we've got things like IP addresses and all sorts we will we'll dig through a few of these but it's it's really really nice really really nice package so let's go let's take a look all right so what we need to do is we're gonna kind of add a validate method on our data type so we can say func P and we're gonna add it to our product I'm going to call it validate and it's just gonna return an arrow so for the time being we're just gonna return nil now what I need to do in order to do the validation is I need to construct a validator now I can construct that validator outside is kind of a global kind of data object because I might want to register a lot of custom stuff makes sense to keep it in one place or for convenience I'm just gonna put it into my my validate function right now okay so so we've got a validator and we're gonna say let's have a look at what the package name is okay validator so I'm gonna copy this input as well so it's gonna be validator dot new and there we go excellent so then what we can do is we can file a date a strut so I can say validate doffed strut and then the strut that I want to validate well I'm just gonna pass it a reference to myself and because strut returns me an error message I can actually just return that so now let's look at how we can add those validation structures to our product strut so name we want name of the product to be a required field so I can add a strut tag so in the same way as I'm doing my strut tags for Jason I can add these validate data tags and I'm just gonna say that this is required we'll leave the others blank for for the time being so let's let's test this out now I'm just going to write a very quick unit test just literally as a proof of concept to kind of these things out I am gonna look at testing in more depth so I'm not gonna write full unit tests it's literally just some stuff that'll help me show you this example and and write the code okay so products tests off go that's packaged data and then we're gonna do is create a simple test so func test checks validation or something it doesn't really matter as a sake I'm not I'm not really interested in writing a full unit test for this right now I'm just going to use this to test out my validation function so if I create a product and then what I want to to be able to do am I on my product is I want to call validate so I'm gonna get an error so I can say error is P dot validate like that right and then I'm gonna write in my TAS I'm gonna say if error is not equal to nil then T dot fatal error just really really really simple so let's run that test so we run that test and it fails as expected because you can see here that it says error field validation for name failed for the required tag this is pretty cool so I've I've not had to do anything really let's look at what I've wired up there but I've literally just wired up the validator and I've just said validate a struct which is myself and then adding these tags so we can add some some more tags so let's have a look well I want my name is required I don't really have any requirements for name but you know I could set a minimum or a maximum length but let's just keep it simple for now and then price well prices is is a required field and I'm gonna say that price has to be greater than zero because I make sense price has to be has to be greater than zero and let's let's give give that a give that a sort of a well I suppose run our test again and you can see that we've got two validation error messages now so we've got the validation for name failed and we've got the validation price failed so I can just check that this is all working so I'm gonna add name thing we say Nix doesn't really matter and I'm gonna add a price and I'm just gonna say one dollar one pound one euro and we'll run that test okay so everything is passing now right so that's the validation in practice is really really easy but what about this what about this school so ask you is is kind of a like a product ID I suppose it's not like a an object ID like the the database ID it would be like the the I unique ID for the product which could contain very certain things like color codes and stuff like that so how do we validate ask you because the SKU is well it's very custom you know it's different things so what we can do with this file a data package is we can add our own custom validation functions so I'm gonna add validate and I'm going to say required because it is and I'm well actually just gonna use SKU now skew doesn't doesn't exist but we can create our own validation functions and we can wire this up so let's have a look how do we create our own validation function well to create our own validation function again we can do validate and then if we we look at the validate it has a function called register validation so let's take a check of that in the docs register validation so this is gonna use a custom tag name and it's gonna use this custom function I am so I can define my own function which is gonna do the validation and all I have to do is return true or false so let's let's see how how we can wire that up for our skill so I'm gonna whoa I'm not gonna do that I'm gonna do that and pasting it in from the docs and I'm gonna call this validate ste you okay so then what I want to do for my skew is my skew is gonna look like this it's gonna have let's say a number of different letters separated by more letters and all liners so very very basic so we want to ensure that the skew is in this is in this format so how do we do that well what we can do is we can just use a regex so reg X so re of course reg X dot must compile and we can define our reg X so our reg X is gonna be a to Z plus - a tizzy whoa paper Zed plus - okay - Zed plus so what we're doing is we're saying that we've got these three parts separated by this dash here and the parts must be in the format of a alphabetic so between a and Z and there must be one or more so it's a pretty pretty straightforward regex and then what we can do is we can just check that so we're gonna say re dot find all strength and it's going to be FL dot field strict so FL field string is the the validator here is returning this field level and if I quickly just jump into the definition for that you can see that it's an interface and it has all of these things such as the parent field name etc etc so to get the the value what I do is I do this field to get the value and then I'm just casting that into a string of course field could be very very many different types so what this is going to do is this is going to return me a slice of string so I'm just gonna call that matches now with this what what I'm Xbox re and we're just gonna say the cow so matches why am I getting this all as a as an array well what I'm doing is that I could actually you pass this a string it could be multi-line it might have multiple skews in it when I get the matches if there's more than one what I can do is I can say so if Len matches does not equal one whoops-a-daisy return false otherwise return true so if we don't have exactly one match then the validation is gonna fail and we're just gonna return return true on there so let's bounce over to our test and give this a little bit of a whirl so let's run my test and what have we got not enough arguments and called a register validation yeah well yeah would help if I wired it up wouldn't it so we're gonna wire it up what is the name of the custom validation function we've called it skew and what is the function - violet - execute when it finds this tag we're doing validates queue so now let's run our test so running our test and it says it's failed so validation failed for skew and that's because well we don't have a skew so let's add a skew for let's have a bad skew ABS well let's just try that for a stop and we can run that and again it's failing because the the skew is not on the right format so I'm let me put something in the right format run our test and now it's working so this is a really really kind of neat way very very quick way that I've just written that sort of that validation so we've got we've got all of that nice validation now so let's start wiring this up onto our API that we created last week so our API if we kind of bounce back then we had these put and these post methods and what we were doing was we added this middleware for both put and post and that's going to validate the object which is passed to it so we go in there and look at our middleware you can see that what we're doing is we're creating a product what converting that product from Jason and if that works and is all is good then we're returning the context but now what we can do is we can validate the product so we can sanitize that input right so we can just do prod dot validate that's gonna return us an error and then we can say if error is not equal to nil well I don't want to call the next thing through in the chain I just want to fail so let me just do a little copypasta because always the quickest way we can do I'm just going to do error validating product in my logs and I'm going to print out the the actual validation message and then I can do HTTP error and I can say error validating product but let's let's actually pass the the error message to the output because that's it's a pretty sane error it's nothing but it's gonna cause any any sort of risks on there exactly the sort of the definition that we'll have in our swagger API which is a nice segue to the next video which will be coming mid week because I'm going to show you how to start documenting your API is with with swagger but for now let's let's just write that error message out so format s printf blah blah and I want the error alright let me just for my lap it does make any functional difference but it makes difference to my sanity and the readability of the code so if the validation fails on our product we're are going to return an HTTP error it's gonna be a bad request because the the input is is invalid what else could we have used no I think bad requesters is pretty much the right thing to use in this instance and I suppose we should probably just give this a go maybe let's see if it works go run go okay so create myself a new terminal so I'm going to curl localhost 99d let's just check our let's check our API of course I don't recall what we wired those up as okay just on the root path that okay so our API is working so now let's let's try posting something to to our API and see what happens so I'm gonna I'm just gonna post a a new I'm gonna create a new a new product and it's going to be let me see let's just call a will add a name and we just can call it new product because imagination is flowing wild and let's see what happens when we do this so let's have a look what are we complaining about curl oh of course my bad data but look at this now so we were getting the the error message back let me just put Kirtland verbose mode but we're we're getting an error message back from our API service and saying bad request so what what are we getting back well say an error validating product key product price I revealed validation for price okay so my far price failed on the GT tab it's you know it's better than nothing it would have been nicer if we we could have had an error message we says price should be a minimum of X amount and we can do that if we did all custom validation so we can we can do that I mean let's just add a price so price for our new product and if I make sure that that Jason just a quick mention of Harrison in the chat base saying that go Lang's ID intelligence goal and 100% agree that is a really really nice really nice ID everything's a little bit broken the new version of go please which just dropped this weekend 0-3 seems to be improving things a lot but still having a few problems here and there anyway so price so let's add our price and it's going to be one pound 23 and we can run that so now well we're getting still getting a bad request but it's telling us that the product SKU error validation SKU is required okay so I can add my my scoop and well let's first add a bad SKU and it's still saying that skew is bad okay don't you make sense so we can say the e f g h i and run that and then that's all working so now we've we've added that that validation to our product api very very simple just a very few kind of steps there by using the the validate package and just kind of using these these strut tags which i think is is really nice really neat again recapping validating the input is good for your security because it ensures that the values are within the bounds of what you're expecting but also it's really nice to be able to present a useful error message to your users because you know it's easy to make mistakes if you've been watching the series you'll see i make our plenty but i mean we're we're all done that we're we're pretty good so let me just check in that code like you so added push it and what I'll do as soon as I hit the stop on this video is I will make sure that I update the link just down below there with the the source code but it's in the same github repository as normal just type in a password for my SSH key and we are golden so I want to say thank you as always for watching I'm looking forward to taking you on to the next part midweek will do kind of am a p.m. GMT on Wednesday I think and we're gonna look at swaggin how can we start out documenting our restful api is using the open api standard okay the swag of a standard some really really nice tools and go to to be able to do this as always i want to say thank you for tuning in if you like these series please like subscribe share them with your friends and colleagues it's I really appreciate all of that and until next time I'm gonna say goodbye

Channel: Nic Jackson

Views: 15,567

Keywords: go, microservices, JSON, RESTful

Id: gE8_-8KoOLc

Length: 27min 36sec (1656 seconds)

Published: Sun Feb 02 2020