Pydantic - Field Customization, Advanced Data Exports and Model Config classes

let's dive deeper into the features offered by pedantic in this video we're going to look at the field function and how we can use that to set default values how we can use it to create aliases and also Define the fields that we want to export in our models we're also going to see some more advanced use cases for exporting the models to Json anti-dictionaries and finally we're going to see how to define a model config classes in order to define model-wide configuration of a pedantic model so let's get started we have here a blog post that was written we're going to cover the content of this post but I'll link it below the video and this is part of a playlist on pedantic if you want to check out the previous two videos feel free to do that they introduce some basic concepts in pedantic and we're going to build from that in this video now the first thing that we're going to do here is we're going to explore how we can use the field function in pedantic I'm going to cross over to the pedantic documentation here and it's the schema page on the left hand side here and this page is the one we looked at in the previous video for exporting your pedantic models to Js skimmers but we're going to scroll down or rather on the right hand side we're going to click field customization and you can see here that the field function can be used to provide extra information about the field and about validations and you can see here that there are a number of arguments you can pass to the field function these are all optional arguments and we have values such as default and default Factory we're going to look at both of these in this video as well as things like aliases and titles so let's start by using the default argument and we're going to define a default value for one of the fields in our pedantic model so I'm going to go back to vs code and we have here a models.pi file containing our pedantic models from the previous video and it's the student model that we're going to amend here and we're going to look at the modules which is a list of module models currently that has a default value of an empty list but we can also specify default values using the field function so before we do that we need to go to the top and from pedantic we can actually import the field function at the top here and then we can change the statement here on the modules and instead of an empty list we can use the field function and we can pass a default value of that empty list and that's going to do exactly as the code before did if there are no modules coming in from a source data set then this is going to set that equal to an empty list by default now on its own there's no benefit to using the field function like this instead of Simply using an empty list but there is a benefit if you need to use the field function anyway to Define some other Arguments for example we might want to put a constraint on the length of this list of modules and we can do that using the max items and that can be set to a number for example 10 and that will then for a validation error if we try and receive our modules object that contains more than 10 modules in the case where we're using the field function for some purpose like this we also can use it to define the default value of an empty list so you can see here that we're building up a field function that contains a validation and that's the max items validation on a list as well as a default value so that works fine for the modules it can just be an empty a list if that data is not present but what happens if you need a dynamic value as your default for example you might want to specify the current date or time as a default value for a date field and that's exactly what we're going to do now this doesn't really make sense but we're just going to do it for demonstrative purposes and we're going to Target the date of birth field here and we're going to use the field function again and we're going to specify a default Factory in this case and the fact that it is actually a python function it's going to be a Lambda function here with no arguments and we're going to see if there's no date of birth in the data we're going to set that to datetime.today.date in other words that is the current date that we're going to set that to so the default Factory allows you to define a dynamic default value for a field so let's test this out and we're going to see if it works we'll go back to main.pi I'm going to remove this comment here that we had in a previous video and we're getting data from a GitHub URL that contains Json data and what we're going to do here is we're going to get the last record from this data dictionary by indexing in at -1 and then we're going to get the date of birth field from that particular object and what I'm going to do is actually remove that from the key value pairs by using this keyword Dell in Python and that's going to remove the key of date of birth from the last object in the data and what we can do after that is go down to the for loop at the bottom here and for each model in the data I'm going to print out the date of birth so let's save that file and I'm going to bring a terminal over here and what we're going to do is run this code I'm going to see the date of birth for each student in the data and you can see that the folder at the top all have this date of birth in the 90s and that's coming from the GitHub data but because we removed this date of birth from the last user that particular date of birth doesn't exist for that student it's replaced with the current date which at the time of recording is the 5th of April 2023 so you can see that this default value coming from a model from the field function it's a dynamic default Factory and it's being applied whenever that date field does not exist in the source data so that covers the default and the default Factory attributes of the field function we're going to move on now and we're going to see how to use the Alias keyword now let's go back to the blog post that we have here and I'm going to go down to this section on using the Alias here now the use case for this is if the field that we want to create on the pedantic model if its name does not match the name of the field coming from our API or our external data for example we have a student model here and it has a field called name and that matches the name of the key in this Source data set which you can see in GitHub here the student has a name in this case it's Eric Travis but what if we wanted to call this field something different on a pedantic model for example we might want to call the field student name in that case we need to use the Alias keyword to tell pedantic what is the name of the source field in our data and in our case the name of the key containing the student's name is just name as you can see on GitHub here so if we're calling it something different on the pedantic model for example student name we need to use that Alias to tell pedantic what to look for or in the source data and this technique is very handy if you want to map the names of fields in your Source data into different names on the pedantic model and that's a very common use case because you don't always want to be confined to whatever the names are on the source data set so you can use the Alias for that we're not going to show a demonstration but it's very useful to know about that field let's move on and we're going to see some Advanced uses for exporting models in pedantic now we saw in the first video of this series how to use the dot dictionary and Dot Json functions to export a model to python dictionary or a Json string Now by default that dumps the entire pedantic model and all of its fields to that dictionary however you can actually control this Behavior with a field function as well if you want to exclude a particular field from the dumped data for example there are two additional keyword arguments to this field function and they both default to false the first one here is the include argument and that's a Boolean that indicates that only this particular field should be included when you call those dictionary and Json methods and the other is the exclude argument which indicates that that field should not be included when you call those functions so for example let's say we wanted to exclude the list of modules from the dictionary or Json what we can do is go back to vs code here and we have this list of modules here it has a default value of an empty list and it has the max items constraint that tells pedantic that this list cannot have more than 10 items we can add another keyword argument here and we're going to set the exclude equal to True here and that is going to exclude the list of modules from the dumped data when we call one of those functions so let's see that in action if we go back to main.pi and within this for loop we're going to print out the model that we get back and just to print one of them I'm going to break out after the first iteration of that for Loop and what we're actually going to print out here is the model.dict function and that's going to dump the model to a dictionary in Python so we can run that code and we're going to see if it includes or excludes those modules so I've opened the terminal here we're going to rerun the main.pi and you can see in this output that we do do not get the list of modules if we go back to the models.pi file and remove that exclude equals true keyword argument if we save that and go back to the terminal and rerun this you're going to see that we get more data and that's because the modules are printed out so the effect of adding the exclude equals true keyword argument to a field function is it's going to tell pedantic when we call model.dect or model.json we want to exclude this particular field from the output and that can be useful if for example you're working with sensitive data or if you're trying to reduce the amount of data you're sending over the network or the amount of data you're transferring between different python processes you can use this exclude keyword argument in order to control that using the field function and we can set as many of these field functions with the exclude equals true as we want for example at the top here of the student model we have an ID that is a uuid object we may not want to expose that to any other processes so we can add the field function here and say exclude equals to true if we bring up our terminal and rerun this code again we are going to see that we don't get the uuid and the output so this way is useful for security sensitive Fields however when you're flexibly dumping data you might not want to have a field definition for every single field that you want to exclude you can imagine some pedantic models might have an awful lot of fields in that case you can customize your call to the model.duct function to that you can pass and exclude keyword argument just as we did in the field function and that is equal to a set in Python of fields that you want to exclude from the output so for example the ID field and the modules list of modules we want to exclude those we've already done that in the field definition here but we can remove exclude equals true from that and we can actually remove the field function from the ID itself if we save that file and go back to main.pi when we call model.dect with this exclude keyword argument that's another way that we can exclude those two fields from the output without having to write the field definition on the model so let's test this out if we go back to the turn terminal here and we rerun main.pi we should see the same output as we had before but the difference now is that we're controlling what's excluded through a keyword argument to the model.dext function and this will work too with the model.json function the same keyword argument can be applied if we rerun this you see we get the output containing the Json ready data and we can add as many output Fields as we want to the exclude function so for example the fees paid we might not want that in our output so we can add that down here and then we can rerun the code and you'll see that that is then excluded from the model.json output so that works well for excluding fields from a student model but what if we wanted to exclude something from a nested model for example from the module objects in this list of modules we might want to exclude the registration code for example what we're going to do is go back to main.pi and I'm going to define a dictionary under this model object now before we Define this dictionary I'm going to go back to pedantic's documentation and this section here is on Advanced include and exclude I'm going to link this below the video and you can see there's somebody at the top here that the dictionary Json and the copy methods they support those two keyword arguments which can then either be sets or dictionaries now we've seen how to do this with sets when we pass the exclude keyword argument here and set that equal to a set of fields we wanted to remove from the output but we can also Define a dictionary and we're going to see why we might want to do that if we go back to the documentation and we scroll down here and it's this paragraph here it says that special care must be taken when you're including or excluding fields from a list or a tuple of sub models or dictionaries and that's what we're going to do now we're going to exclude a field the registration code from a sub model that's the module we have in the student class a list of these modules and we want to exclude a particular field from that module and that's the registration code if we go back to the documentation we can see this line here and that says that to exclude a field from every member of a list or a tuple the dictionary key all can be used as follows and if you scroll down to the bottom here we have here a dictionary of exclude keys and you can see that when you use addictionally you set the name of the field and the value is a Boolean which determines whether or not the field should be excluded from the output now that works when you have the fields defined directly on the pedantic model but when you also want to exclude fields from a subclass or a list of sub models for example we do that here for the address and we Define another dictionary for that key and that dictionary tells pedantic which fields from the sub model should be excluded from the output and you do that using a nested dictionary but when you have a list of such models if we scroll to the bottom if we look at this line of code here we have the user object and we're using the dictionary function on that user object and we're passing the exclude keyword argument to that with a dictionary here and the key at the top level is the hobbies and that contains a list of sub models hobby objects and to exclude the info field from all of those we use this all key so let's see an example now of how we can do that our own application let's go back to vs code and we're going to go back to main.pi and I'm going to Define that dictionary here called excludes and we're going to set that equal to a structure that matches what we had on the documentation now let's say we want to exclude the ID field that is defined directly on the student model so we can just set that to a Boolean value of true and that will then exclude this uuid from the output but if we wanted to exclude the registration code from each of the modules in this list of modules what we need to do is we need to use that nested structure here so what we're going to do is We'll add another key here and it's going to be for modules and this refers to the fields on the model that contains the list of those objects and we're going to set that equal to an inner dictionary and the key we're going to use is that underscore underscore all and then two more underscores and that's going to map to a set of fields that we want to exclude from all of the objects in the list of sub models so in our case we want to exclude the registration code here so we're going to copy that I'm going to put it into this list as a string and on the line below that to the exclude here we're just going to pass that dictionary that we've defined above and let's now see if this works if we go back to our terminal we're going to clear that out and rerun main.pi and actually what I'm going to do so that we can see this better at the top of main.pi I'm going to bring an import in here and that's the P print function from the p-print module and we can use that function here to pretty print the model.jsoncode and what we can also do is change the model.json to a model dot dict call and that's going to Output that as dictionaries instead of Json data so we can now go back to our terminal and we're going to rerun main.pi and we're going to see what the output is for this you can see the list of modules is defined now and the structure of this data is quite a bit better than it was before but you should note that for each module in this list of modules we're not getting the registration code and that is because of this line here in the exclude statement we take that list of nested modules and to all of them we are excluding the registration code if we remove this line of code mode and go back to the terminal and rerun the script you can see that the registration code is present in that output so we need that kind of line here if we're going to exclude a particular field from a list of nested sub models so we've covered some Advanced ways we can dump data to dictionaries and Json and how we can exclude certain fields from that data let's move on now to model config classes in pedantic I'm going to go back to the pedantic documentation and on the left hand sidebar we can click model config and you can see here that behavior of pedantic can be controlled via the config class on a model or a pedantic data class so in the example below that we have a class called Model that inherits from pedantic's base model and it defines a field called V that is set to the string data type and then we have a class an inner class here called config and that contains some configuration that's applied throughout the pedantic model itself for example at the top here we have Max any string length set to 10 and that is going to say that for every field on the class the length of those strings cannot be greater than 10 characters let's see an example of how to use config classes in our code here if we go back to the models.pi file we're going to focus on the student model here and you can see that we have a department field and that's set to an enum and we Define that enum and the enums.pi file and it contains three values what we're going to do is go back to models.pi and within the student model we're going to define a class called config and within there we can set any configuration we need for this pedantic model and one of the configuration Fields is called use enum values and we can set that equal to true now what I'm going to do is I'm going to print out each department for every student that we're getting back from GitHub so let's go back to main.pi and we'll remove the exclude statement here and what we're going to print out for each model is the model dot department and we can remove the break statement and we're going to print all of those out now what we're going to do is go back to the terminal and we're going to rerun the python main.pi script and you can see the output here for each department for the students and our data and these departments are coming back as string data types and they're not coming back as those enum values and that's because we're setting this model configuration within the config class we're going to say that when we have an enum as a field type we want to use the raw value of the enum when we're outputting that particular field now if we remove this config class and comment it out and go back to the terminal you can see that the output is explicitly referring to the department enums field when it's outputting those values so the effect of using this config class with the use enum values set to True is that it's going to take those enums and it's going to Output them as raw values in this case raw strings and that can be useful for things like serializing your objects and it's one of many configurations that you can apply in a pedantic model config class now we're going to move on to another field we can use in the config class let's go back to the pedantic documentation and for model config we have on the right hand side under the table of contents we have options that we can apply within that config class for example the title we can define a title for our pedantic model and that's going to be reflected in the Json schema output so for example if I set this title to student model that will be the title and the Json schema output now we're going to see a particularly useful field here if we go back to the documentation it's this one here it's called extra and this field tells pedantic whether to ignore allow or forbid extra attributes when you initialize a pedantic model and when that value is set to forbid it's going to cause validation to fail if you try and add extra attributes to the model ignore is going to silently ignore any extra attributes and allow will actually assign the attributes to the model so we have three behaviors here let's demonstrate how this works now what we're going to do is go back to models.pi and I'm going to remove this field here called phase paid let's delete that from the model so now the student model is missing that particular field but the GitHub data that we're using if we go back to the GitHub data set that contains this field called phase paid and it's set to a Boolean value so you can think of the fees paid field as being extra data that's coming into the pedantic model and when we take this data and we instantiate a pedantic model as we're doing in main.pi here we're passing that extra field that's no longer on the model to that model and the question is what should pedantic do when it sees that extra field and that's what they ignore parameter to the config class does so let's go back to the class here and the config class we're going to define the extra keyword argument I'm going to set it equal to ignore now this is the default and it's just going to say to pedantic if we see any extra data in the instantiation of this class we're just going to ignore any fields that are not defined within the model if we save this file and go back to main.pi we're going to try and print out model dot phase page which remember we've removed from the model so if we go back to the terminal and run python main.pi you can see that the student object has no attribute of phase it so even though the pedantic model when we instantiate it on line 13 here is receiving that fees paid as one of the arguments it's not going to do anything with it it's just going to ignore it and when we try and access that attribute we get an attribute error in that case but we don't get a validation error when we actually create the object now if you want to be more strict about this and actually throw a validation error if you receive data you're not expecting you can go back to models.pi and we can set extra equal to forbid and in that case when we try and rerun this code we're going to get a different error and that's the extra fields are not permitted and you can see that the name of the field that we're passing that's raising this error is fees paid it's no longer part of our pedantic model definition it's extra data that we're trying to pass into the model when we're creating that model and that is no longer allowed when we set the extra parameter to forbid within the config class now this can be useful for example if you want to avoid any data being serialized that you don't expect but on the other hand you may actually want to be quite flexible about what you allow on your pedantic model so you have a base set of fields here but you might want to allow extra fields to be added to the model and in that case you can change for bid to allow and if we now go back to the terminal and try to rerun this code you can see me get back the Boolean values that are coming from the GitHub data and even though we don't have a phase paid field on our model anymore that extra data is allowed to come into the pedantic model and it's set on the attribute called fees paid so we've covered a few different fields on the config classes some are used for Json schemas such as title we have things like use enum values that we can set to true and we also have this very useful extra field that we can use to determine whether or not pedantic should allow extra data to come into the model or what Behavior to apply when that's the case now if we go back to the pedantic documentation you can see that on many other fields that we can use in pedantic model classes for example any string offer is going to upper case any strings that come into your model all and we also have any string lower for the opposite and you can set things like the minimum and maximum length of strings and many other things too for example here's the one we used called use enum values and I'll link this page below the video if you're interested in reading about these but that's all for this video in this video we've covered a number of different concepts we're going to look at the summary on the blog post we've seen how to use the field function to set default values including how to use the default Factory keyword argument to set a dynamic value in the data we've also seen how you can use the Alias keyword argument to handle when fields on a Model might have a different name than what's coming in from your Source data we've looked at Advanced use cases of the model.dictionary and model.json functions which you can use to export your pedantic models to a python dictionaries or Json strings we've seen how to use the field function to control whether certain fields are excluded from the output and how we can also pass keyword arguments to the model.dictionary and model.json functions to control those when we actually call the function and finally we've learned about the model config class in pedantic and how that can be used to apply model wide configuration to your class and you can use that config to perform cleanup of raw data for example by stripping white space or by uppercasing all strings and we can do some other things such as controlling whether enum values are going to be used and also the extra keyword argument which determines the behavior pedantic applies when extra data is coming to your model so I'm going to do one more video on pedantic which is going to cover model validation in more detail possibly there'll be others after that but if you're interested in anything in particular please let me know in the comments thank you for watching and I hope you've enjoyed the video if you have please like And subscribe to the channel and we'll see you in the next video

Channel: BugBytes

Views: 6,102

Keywords:

Id: Z0a0Vjd992I

Length: 23min 41sec (1421 seconds)

Published: Thu Apr 06 2023