Pydantic - Validators (Root Validators, Pre-Item Validators, Each-Item Validators)

in this video we're going to dive deeper into validators in pedantic we're going to understand the order in which pedantic applies validators to our Fields we're going to learn what a root validator is in pedantic and we're going to learn how to apply functions that run before the default pedantic validation finally we're going to learn how to apply a validator to each item in a collection of data which might be a list in python or a set or a dictionary we're going to learn how to define validators that work on each item in those objects so let's get started I've written a blog post on this topic I'll leave a link to this below the video and what we're going to do is recap what we did before so what I'm going to do is open vs code and I have the codes from the previous video open here and what we have is a pedantic model called student and the student model has two validator functions applied to particular fields on the model we have a validator applied to the modules field and the modules field as you can see in line 23 it's a list of module objects and module is a pedantic class as well it's a pedantic model so this is a list of nested modules that are applied to a student so a student will have many modules and this is the data structure that creates that relationship we have a validator function which is defined using this validator decorator that you can import from pedantic and below that one we have another validator that's applied to the date of birth field and as said before and these functions you can apply any logic you want so this is the basic way of defining validators in pedantic it's going to cover most use cases you just Define a python function and you decorate it with this validator decorator and the argument is passed to the validator is the field to which the logic will be applied now if we look at the signature of this function you can see that the first parameter to the function is the class so that implies that validator functions in pedantically are class methods they're not instance methods so we don't use self as an argument here we're referring instead to the actual class itself and the second argument to the function is the value that we're actually validating so when we use a pedantic model to validate some incoming data for the field that we're applying the validator to in this case date of birth this is the value of that field and the incoming data so that is the second parameter to the function but it turns out these validator functions they can accept an additional argument that we're going to see now and on the pedantic documentation it gives you some things to note on these validators for example you can add any subset of the following arguments to the signature and the one we are going to look at is values and that is a dictionary containing the name to Value mapping of any previously validated Fields now if we go back to the classes here it turns out that the validation in pedantic is applied in the order that we Define these fields so for example any validators on the ID field are going to be applied first if the ID field is validated successfully we'll move on to the name field then date of birth and so on so it's applied top to bottom in the order that you define the fields and we can see this if we Define another validator so I'm going to Define one for the GPA field so let's go just below the config class and we'll write a validator decorator and we're applying this to the GPA field and we can write the function called validate GPA here now like the other functions it takes the class as its first argument and the value being passed in as a second argument we're now going to add a third argument called values here and that's going to contain all the previously validated values so this is just a dummy function we're just going to print out the values dictionary that's coming in and then we're just going to return the value for the GPA now if we go back to the main.pi file this is where we Define a URL we get the data from that URL and convert it to a Json and I'm going to remove this statement here and we have a for Loop that is basically converting each record in the data to a student pedantic model and then it's printing out a particular field in this case I'm going to change that and just print the model out now I'm going to run the main.pi file and I'm going to bring a terminal over to do that and we have a virtual environment activated with pedantic here and we can run python main.pi in order to actually run that file and what we're going to get is the models printed out so instead of printing out all the models what we actually want to see here as we want to see the values that's coming through for the GPA field for each record so I'm going to actually remove this print statement to the model and let's go back to the terminal clear that out and rerun this script now what you can see is for each record in the data we are getting the values that have already been validated printed out to the terminal so you can see the ID which is a uuid object we get the name of the student and we also get the student's date of birth if we go back to our models.pi file you can see that this validator for the GPA that's the one that prints this out this is printing out these values for the records that come before the GPA and The Ordering of the fields so this is just emphasizing that the validation is occurring from top to bottom with the order that you define in these fields now that can be fine for some use cases but sometimes your validator functions they require the values of other fields for example if the GPA required the department value then we would need to re-enter these fields so that GPA comes after our department but then the department field may also will be dependent on the value of another field for example the name so this can get very tricky if you have a lot of fields and there's got to be a better way than simply relying on the ordering now one solution to this is to use root validators and root validators they run validation on the entire model's data after the individual validator functions have all run so what we're going to do is create a contrived the example here and let's say that we only want to accept students in the science and engineering department whose GPA is greater than or equal to three so for science and engineering students unless your GPA is above three we're going to reject them what we can do is we can import at the top the root validator decorator so that comes directly from pedantic we're importing the root validator and let's scroll down and we're going to Define another function here and we're going to call this function validate GPA and department and that takes the class again as the first argument as well as the values and the values is a populated dictionary containing all of the validated values for every single field in our pedantic mode so that's the function for the body of that let's just pass for now and we're going to decorate this with the root validator decorator and that will tell pedantic that this will be run after all of the validators for the individual Fields so what we're going to do is we're going to extract from the values dictionary the GPA and the department so let's create a variable called department and we're going to use the values.getfunction and we can get the Department value from that dictionary and then I'm going to copy that line below and we can rename this to GPA and we're going to get the GPA field from that dictionary and now we can perform the validation here let's now create two Boolean variables that we're going to use in an if statement first of all we're going to check if the student is part of the science and engineering department we can do that by checking if the department that we've extracted is equal to the department enum.science and engineering.value so remember the department has a value that's set to an enum here and we're accessing the science and engineering field on that enum and we're getting the text value from that I'm not checking whether or not what we've extracted is equal to that value so that's the first Boolean variable we'll create another one below and we're going to say valid GPA and we can set that equal to GPA being greater than or equal to 3.0 so that's our 2 billion variables we're now going to check if the student is part of the science department and if they are we're going to reject them if they do not have a valid GPA so let's now add another if statement here and if they don't have a valid GPA what we're going to do is we're going to actually raise a value error here and we can pass a message to the value error and we're going to say GPA not high enough for Science and Engineering now that's the case where we have a problem and we want to raise a value error if we get to the end of this function without raising that value error what we want to do for a root validator is just return all of the validated values we can return values and that's the end of this root validator function if we now go back to main.pi we're going to keep this code largely the same but in the for Loop when we convert the student to a pedantic model we're going to surround that with try statement and I'm going to tab these across but trying to convert the data that's coming in to a student pedantic model and if we manage to convert that student we're going to print a message out with the student's GPA and also the department but we also want to catch the exception so we're going to use the accept statement here and we can pass a value error as e and what we want to do is just print the message that's coming from that exception we can do that with the print e statement and we can now run this to see what kind of output we get before we do that I'm actually going to remove this print statement for the GPA validator that we used a second ago and we can close models.pi and I'm going to bring up the terminal now if we run the main.pi script we should see the output and you can see for the first record we get a GPA of three and we get the science and engineering department but for the second record we're getting a validation error and that's that the GPA is not high enough for Science and Engineering now let's have a look at the source data now this is the source data set here I'll leave a link to this below the video you can see the records in this list of data have the first record which remember had a GPA of three and also had Department of Science and Engineering for the second record we also have science and engineering but the GPA is 2.5 and that's too low to actually pass the test that we've just defined so that is why that second record is being rejected and we get the message from our value error so the root validator allows us to compare different values for different fields in the pedantic model and we can access those previously validated values within this values argument that's passed to the root validator and again we can perform any logic we want within the body of that function now if you want a comparison to djangle this is kind of similar to a Django form and Django forms we can validate individual fields and for example if we had a field called name we would Define a function called clean underscore name and that would allow us to validate the values that are coming in for that field so the clean field name functions are similar to The pedantic validators on the other hand the root validator is similar to the jungle clean function so the Django clean function has access to all of the values coming for the fields of the form and that allows you to perform comparisons based on more than one field so if you know Django you can think of root validators as being kind of analogous to the clean function so let's move on we're now going to cover pre-validators in pedantic now sometimes you want a function to run before pedantic applies any validations to a field an example of this is that you might want to join up the elements of a list or a tuple before you apply a validator to that field or conversely you might want to take a CSV delimited string for example and split that into elements in a python list now these kind of transformations of data they must be applied before the pi identity validator if the validator for example is expecting an array you must split the string before you send the value to that function now defining a validator function that runs before normal validation you can actually achieve this with the normal validator decorator and we can do that by passing our keyword argument to that call pre and setting that equal to true and as you can see in vs code when you apply this pre-parameter it defines whether or not the validator should be called before the standard validators now to get an example of this what I'm going to do is go back to the GitHub repository and we're going to now use the student version 3.json file and you can see if we scroll down that we've added a new field to each record in this data and that's the tags field and this is a comma separated bit of text containing different tags that you might want to apply to a student for example motivated skilled and hard working and if you scroll down you might have other tags such as hipster lazy and slacker so each student now has a set of tags and we're going to change our main.pi script to actually fetch that new data set so instead of student version 2 we're going to set it to version three and of course because we have a new field we need to add that to our pedantic model so let's add the field code tags and this is just going to be a list of strings now what we're seeing here is that we want the tags to be a list of strings in the pi dancing model but as you can see on the API or on the GitHub data set the tags are actually coming through as a comma separated string so if we want to use a list of tags here we need to split that string on the comma so that we get back a python list of strings instead so let's go down here and what I'm going to do is remove this GPA validator and I'm going to validate this tags field now instead and what I'm going to do is Show an example of this that's not going to work but it is going to demonstrate the use of pre-validators and we're going to fix this in a second so let's call the function split tags and because this is a normal validator it takes the class and the actual value as parameters and let's say we want to take the string of tags coming back and we want to split that by the comma so what we can do is return value dot split and what we're going to split on is the comma here and before we run this let's also just comment a root validator now we don't need that anymore and we can bring up the terminal and run this script now when we get the results you see that we have a validation error for all of our records and it's on the tags field the problem problem is that the value is not a valid list so if we go back to models.pi you can see that we are expecting a list of strings here but again we're getting back some text comma separated text that we need to split into a string so the splitting does not work on a normal validator function and the reason for this is that pedantic's normal validators that check the actual types they will run before your custom validator functions so if you want to apply some logic like this that will run before pedantic's default validation you can pass the pre equals true parameter to the validator if we go back to the terminal now and rerun this you can see that we no longer get those errors and we're getting back the student pedantic models so just by passing pre equals true to the validator function we're then able to apply this split call before the pedantic default validation and that will then return a list which is what's expected by the default validator and we can then proceed as normal and just to see that this is actually a list of strings we can go back to main.pi I'm going to change the print statement here I'm going to print out model dot tags let's execute that now and we get back all of the tags for each student now one thing you might notice about the tags is that we have a leading white space at the tags that are not the first tag so for the first tag in each list it's totally fine but for the others we have this leading white space and that's because of the way we're splitting this but we can add our field to the pedantic config class for the student model and we're going to use a field here called any string strip white space and we can set that to true and what that's going to do is then remove any leading or trailing white space and because it's in the config class that's going to be applied to any field where it makes sense to strip white space so any kind of string field if we go back to the terminal and rerun this you can see that that fixes the problem with that leading white space so that shows how we can perform some logic before the pedantic default validation of course we're now going to finish this video by showing how to run pair item validators so as you can see in this list we have some students that have a tag of slacker let's see that we want to throw a validation error if we get a student who's a slacker in this data set and we want to reject that student so let's go back to vs code and I'm going to scroll down and we already have this validator for the tags field but this is the one that's applied before the default validation but what we're going to do now is Define another validator function and again it's going to be applied to the tags field but we're going to pass a second keyword argument here and that's going to be the each item equals true keyword argument so let's scroll down and Define the body of this function we're going to call the function remove Slackers and just as above it takes the class and the value coming in as arguments so like the name implies each item equals true we'll apply the validation logic to each item within the field that we're passing in so we can simply write an if statement here and check if the value is equal to Slacker and if it is what we can do is raise another value error and we can pass a message into that value error if not we can simply return the value and proceed as normal so each item equals true will result in this function being applied to each tag in this list one at a time whereas a normal validator that doesn't have each item will simply receive the list of tags as an argument and then well you could write a for look to go through these but that each item equals true is a little bit cleaner in my opinion and it is true that often you will be dealing with lists sets dictionaries tuples in your data so when you need to validate each item in those data structures you can use this construct here so let's now run this code and we can see if this works I'm going to clear the terminal and rerun main.pi and you can see now that we are getting validation errors for students and that's for any student who is a slacker in this data set so these are a couple of different techniques you can use in pedantic when you need to perform validation but you're not simply validating the row value of a field if you use the root validator that we defined here you can access all of the values for each field in the data so this is important when the value of one field might be tangent on the value of another and you need to perform validation based on that logic we've also seen the validator for tags where we have this pre equals true argument which applies the validator before pedantic's default validation and that allows you to perform data Transformations and any other kind of logic you want to apply before the default validation in this case we perform the transformation where we took a comma separated string of tags and we split them into a python list where each element in the list was one of the tags and as well as pre-validators with the pre-equals true keyword argument we can also pass each item equals true to The validator Decorator and this is then going to apply the logic of the function to each item in a sequence of data or a mapping of data so that's all for this video thank you for watching and if you've enjoyed the video please like And subscribe to the channel we've done four pedantic videos now I've not got any more planned on just using pedantic alone but hopefully in the future we'll do some videos using fast API and other technologies that rely on take under the hood but if you do have any more requests for videos just let me know in the comments thanks again for watching we'll see you in the next video

Channel: BugBytes

Views: 5,774

Keywords:

Id: nQJKkY8XnRg

Length: 18min 6sec (1086 seconds)

Published: Mon Jun 05 2023