Intro to Pydantic V2

hi and welcome to this video where I'm going to basically give you a quick introduction to pantic V2 I have a full UD Demi course if you want in-depth coverage of pantic V2 the goal of this video is just to get you started with V2 and if you're looking for pantic version one then I also have a video in the channel that covers pantic version one and that link is in the Jupiter notebook that accompanies this video now one thing that you'll need to know of course for pantic is basic python type hinting so you need to know that because pantic both version one and version two essentially use type hinting just like data classes to Define fields in our classes or our pantic models if you don't know type hinting I have a video on that as well in this Channel and if you look at the Jupiter notebook there is a link to that video I'll also provide you a link to the pantic documentation of course you'll need to install piden ptic first and you can just do a pip install pantic I also provide a link to the pantic installation documentation should you need it if you run into problems doing just pip install pantic should work but if it doesn't you have that documentation available now pantic data models are simply python classes with extra functionality provided by pantic via inheritance so let's start by looking at a basic model to do that we need need to from pantic we need to import base model once we've done that we can create a model this way and it is done through inheritance so unlike data classes that are done using a decorator because data classes are code generators nothing more whereas pantic is actually providing the extra functionality that we have the validation and the passing functionality that we have in pantic through inheritance so basically this base model is what has a lot of the functionality that we're looking for so we create classes that inherit from base model so let's go ahead and create a field and we'll create first name to be a string this is the typ painting last name should be a string as well and then say we have age that's going to be an integer and that's it this is a very basic pantic model as you can see we defined the data type of the fields in the model using python type hins and we inherit from base model we can now create instances of this model in a variety of ways we could do something like this we could say p equals person and we'll specify the first name equal to let's say John then we'll say last name equals Smith and let's say age equals 42 like so and if we look at P pantic provides us a default def representation we can always overwrite that and choose which Fields actually show up in that representation by default it includes all the fields and you can see that it basically was able to take this data and create a model and this is basically an object this is a class instance in Python so pantic will also perform validation on your input data in some cases it will attempt to coers the input data to the proper type but when it cannot do so and when validation fails in general pantic raises a validation error exception which is a pantic exception so the idea behind this is that once you actually have an instance of the model you are guaranteed that first name is going to be a string last name is going to be a string and age is going to be an integer and in fact you're also guaranteed because of the way that we Define the things here that those fields will be popular populated they will not be none because these by default are all required we'll take a look at what optional fields are a little bit later so let's go ahead and just see how the validation errors work so from pantic we're going to import validation error and then we're going to do a try and I'm basically going to take this that I have over here and instead of H equal 42 I'm going to pass in a value that is not coercible now I do want to mention that it will try pantic by default will try and coer this so this is a string right but our age should be an integer if we execute that you'll see that our age now is back to an integer so pantic actually validated this but also coerced it to an integer now in this case it can't this cannot be coerced to an integer so it will fail and we'll get a validation error and then we can just print the error and as you can see our message says we have one validation error the age the input should be a valid integer and in fact we passed it an input type of string right with this value over here and in fact let me show you another thing if we go ahead and change this to let's say 100 pantic will not by default serialize objects to Strings because objects always have a string representation so saying like Okay well I'm going to take the string representation of whatever you pass in here and put it into last name is too broad of a coercion it you can easily introduce bugs that you wouldn't even be aware of by doing that so instead pantic says Nope by default if you pass me something that isn't a string here I'm not going to try and coers it to a string so if we try and run this you'll see that now we have two validation errors the last name the input should be a valid string it wasn't it was an integer and of course we still have the age now fields in model instances can be accessed using object do notation so if we go back to uh an example that works I can now access the fields just using notation so we're dealing with a regular python object we can even mutate those field values we could say p false name equals James and if we look at P now you can see that that has changed now the first name was mutated it is now James now one word of caution here by default pantic validates the data being deserialized as we saw here when we try to pass it let's say this it failed and this it failed but it doesn't validate data that's being changed via an assignment not by default we can't force it to do that but by default it doesn't so you have to be a little bit careful here because I could just go say pH equals unknown or if you want junk since we saw that that raised an exception and if we do that you can see that age now has been mutated and it is this string which is not really consistent with what we defined in the model all right so just a word of caution here so I just want to expand a little bit on validation exceptions as we just saw pantic validates all the fields it does not just stop at the first validation error see here we had two fields that basically failed validation pantic gave us the two validation errors it does not let's say reach last name find a validation error and then stop processing the rest of the data it continues it tries to essentially deserialize load this data into the model completely even though by the time this has failed we know that we're not going to get an instance model we're going to get a validation exception it still continues the whole passing to identify all the problems so that you get basically information about everything that's wrong not just the first thing so that's very useful to know now so far we've just been printing the error message but you can also get the list of errors as data using some special methods that are provided by pantic validation error exception so let's go ahead and take a look at that I'm going to just take this over here and instead of printing it I'm going to assign it to a variable called exceptions so I'm going to take the exception and assign it to this variable I now have access to this variable which is this validation error instance and it has a couple of methods on it that are quite useful one is called errors like so and when we do that we get this list it's just a python list and you can see then you have dictionaries for the items and it tells you the message input should be a valid string it was for last name the input was 100 here you also get another error message tells you that it was for the age the input should be a valid integer so basically the information that we saw with the print statement you can recover as data by using the errors and if you're working with rest apis you most likely are going to want to return this data in Json format so pantic has us covered here as well we can say Json exceptions. Json and you'll get essentially the same data but now in Json format and you can if you want modify this or you can just return it as is to the caller of your API to tell them you have a problem in the data that you submitted the next thing I want to look at is des serial izing data in more general terms we saw that we could essentially load a model by providing the field names with their values this way and you have to pass them as named arguments you cannot pass them positionally but there are actually two additional ways of loading data to create a model and of course that is called in general deserializing data here we're deserializing named arguments here we can ALS Al deserialize using dictionaries so if we have data that looks like this I'm just going to copy paste that from The Notebook let's say that our data looks like this as you can see it's a dictionary the keys correspond to the field names in our model and then we have the values so what we can do here we can say p equals person. model validate and we're going to pass it data and you can see that now we end up with this same object that we had before well it's not not the same object but an object with the same state the same data in it but we got that from a dictionary we deserialize the dictionary into a model it also supports deserializing from Json so now this is a string it looks like a python dictionary but it's actually a Json object it's a string so to do that to load that up we have to do a model validate but we have to tell pantic that this is actually Json so we say model validate Json and we pass it data Json and if we do that as you can see again we get the same result we get a person instance with the appropriate data the next thing I want to look at is this required versus optional Fields I already mentioned that by default all fields are required and indeed if we try to deserialize data that is missing any of those fields we'll get a validation error let's try it with the current person model that we have so if we just pass in the age but I omit the first name and the last name then we are going to get a validation error as well and you'll see that we get two validation errors that first name is required last name is required and of course the same thing happens if I pass it a dictionary let's say with age 42 the same thing is going to happen right that data is is missing the first name and last name so if we do that and I'm going to try and we'll say person model validate we'll pass in data and of course we're going to get an exception as well so we'll get a validation error and we can go ahead and print that error and you can see we get the same exceptions essentially so how do we make a field optional we simply provide a default value for it in the model so if we take our model that we had over here right in the beginning we'll take this one and we are now now going to add some defaults to those fields to make them optional so I'm going to make age optional by setting a default equal to zero and you should always make sure that you set your default consistent with whatever the type is that you're specifying so don't go ahead and specify a string here by default pantic is not going to validate your defaults so this means that you could technically put in a string here that that doesn't match this integer type pantic isn't going to comp plane there are ways that you can configure your model so that it will actually validate those defaults and raise an exception but since we're controlling what this is we shouldn't have to do it we just have to be careful that we specify proper defaults so one thing I want to show you is we can inspect what the model Fields look like what their definitions look like and we do that by using this model Fields attribute that is on the pantic model and and you can see that it tells us we have a dictionary the keys are the field names so first name last name and age and then we have this field information it tells us that it's a string and that it is required now you'll notice for age however required is now false and that's because we have a default and the default was zero so now we can create an instance this way we can say p equals person and of course you can deserialize however you want you can use data you can use Json or you can use named arguments I'm going to use named arguments in general because it's much simpler to Define that rather than having to define a dictionary and then model validate but it's doing the same thing so last name equals Smith and you'll notice now I'm not passing an age in before we would have had validation error now we no longer have that and the age was set to the default now if I specify the age it's going to be whatever the age is right so you can see it took that but if I don't specify it it's going to fill in the default value so this brings up the next topic which is the one of nullable fields because we can set defaults to none but you have to be a bit more careful because if you take a look let's say at this class over here let's go back here and let's say that I want to allow first name to be none right now you remember what I told you just now that you have to be careful that the type you're assigning as the default should be consistent with the type that you declare the field to be well none and string are not the same types this is the non type this is the string type so technically this is incorrect now of course as I mentioned pantic doesn't validate your defaults so it will be perfectly happy to put none into first name even though the field says it must be a string so in order for this to be corrected what we really need to specify spe ify in the type hint is that this type string can also be none it can be nullable so we can say it is string or none equals none and that is the correct way of doing it and now if we look at the model Fields let's take a look at what it says here you can see that first name is no longer required and that's because we have a default of none and in a second I'm going to get into this what this Union thing is for now this is how we're specifying a nullable field and this is newer syntax I'll I'll come back to that let me just show you first that we can say person last name equals Smith because now everything else is optional because first name has a default age has a default so the only required field in this model as we can see from the model Fields is just last name so if we do this everything works just fine and of course we have none for the first name and zero for that now let me go back to the this notation here string or none this is just an alternative syntax that's available in more recent versions of python I believe it's python 310 and above for older versions you can use the canonical way of doing this which is using a union so this is just syntax what it what this really means is the following we can say from typing import Union I know I said that you need to know typing you know typ hinting to take this video but let me just show you that very quickly so if I take this class again instead of specifying it this way I can say it is the union of string and none this is the exact same thing as specifying things this way and so in fact if we look at person model Fields you'll see that we exactly have the same thing we have this Union string non type you can see it's identical to here so this is just kind of of you know syntactical sugar essentially that we don't have to have this horrible looking thing that's pretty long instead we can write this much simpler but it's only for more recent versions of python now there is actually a third way of specifying it and that's to use the optional from typing now I do not like this so I'm going to show you how it works and I'm going to show you why I don't like it so what's equivalent to writing this is to say optional string like so so what does this optional string means it just means that this is a nullable field this field can be string or none it happens to have a default of none but this optional has nothing to do with with whether the field is optional or not whether it's required or optional this is why in the context of pantic I do do not like using optional because it is confusing because if you have this model over here right if some if you're just reading this quickly or you're not quite aware of the difference between nullable fields and defaulted Fields you might think that oh well this is an optional field and of course that is not the case if we look at model Fields you'll see that this required now is still true it's back to true because why well because we didn't specify a default for first name so that's why I don't like using optional but go ahead feel free to use whatever you prefer so pantic fully supports Python's type hinting system so for example you could specify a field to be a list of certain type we could do something like this we could say let me go back to our original class person as we had it over here this one so I'll take this model and I'm just going to add to it I'm going to say that person has lucky numbers and that should be a list of integers and I'm going to make it a default of an empty list and I'll come back to this because I know that some of you if you're used to data classes you know that you can't do that if you used to Python and function arguments you know that you shouldn't do that but pantic allows us to do it and it's perfectly fine it's perfectly acceptable to do that in pantic I'll come back to why so if we do that and we look at person model Fields you'll see that now we have lucky numbers if I spell it right and we can see that it is a list of integers it is not required because the default is an empty list and so by doing this the type coercion is going to apply not only does lucky numbers have to be provided as a list or something that can be converted to a list like a tuple for example but the elements of that sequence are going to have to be coercible to integers as well so let's take a look let's say I say p equals person and I'll just specify last name because that's required and then we have to specify lucky numbers as well not because it's required but because I want to show you something so lucky numbers equals and I'll do one then I'll do the string two and then I'll do the float 3.0 now all of these can be CED to integers there is a way that you can configure your model so that you can stop certain coercion from happening that's called strict coercion by default pantic uses something called LAX coercion I'm not going to get into that here but basically it's a it's you you can make the coercion stricter than this in this case however pantic is perfectly happy to coers one to an INT well it's already an INT so it doesn't have to do anything to coers the string two to an integer that's possible and to coers the float 3.0 to3 and these are indeed all integers if we want to make absolutely sure we can say for number in p. lucky numbers and we'll just print the type let's let's just print the type of number like so all right and you can see they're all integers okay the next thing I want to talk about our aliases and this is where I'm going to introduce to you pantic field class sometimes the data we are attempting to deserialize uses names that we simply do not or cannot even use in our model for example consider this data that we would like to model using pantic let's say that you're getting data back from some data source and this is what you're getting and it happens right this isn't far-fetched where you have first name with a space in it last names all caps aging years has spaces in it ID is actually a you know function in Python so we you know we probably don't want to use ID as the field name in our model we certainly cannot use first space name that's not a valid variable you know name in Python we probably don't want to use this because it's all uppercase that's not pythonic and again this one we can't because it's got spaces in it so how do we deal with that well pantic has a way to define an alternative name to our field names and those are called aliases and here's how we would set up a model to handle that data so from pantic I need to import field we're going to need that class let's go ahead and create our class person and we're going to inherit from base model and what do I want well I want this field ID but I don't want to call it ID here so typically what we do in Python is that if we have a reserve word or a keyword or something we just basically tack on a underscore at the end that's how we get around that problem and you'll see that used all the time so this is going to be an integer but that's not how the data is defined in my data dictionary as you saw we had to match between the names in the dictionary or the Json or the argument names to the field name that we have here so here I want to basically provide WR something else and I use this field class and I'm going to create an instance of it and I'm going to set an argument called Alias and this Alias is going to be the name of the field in the data that I'm deserializing so here it's ID then first name I'm going to call it in pythonic terms first uncore name that's a string and that should be a field with an alias equal to well first space name so we have that then we have to Define last name which is again a field and then the Alias equals last name and lastly we have to do the age which is an integer and it's going to have n Alias equal to age in years like so so this is our model and now that we have that we can go ahead and model validate our data that's person model validate and we're going to pass it data and if we did everything correctly it should validate correctly it should deserialize correctly and you can see indeed that we got 100 in ID we got John Smith and 42 so this is why we have aliases that's one of the reasons we have aliases now pantic models also give us the ability to do the reverse process of deserializing which is serializing and so basically we can take this python object right this P here which is an instance of this person class it's just a python object and from that we can generate a python dictionary that's going to contain basically the field that we have the fields that we have here with their values so those field names with the values and then we can do the same thing to generate Json out of that and it's very simple we just say p. model dump like so and you can see we have a python dictionary where we have the field names not the aliases but the field names and the values of each of those fields now if we want Json we can do the same thing but we can just say dump model dump Json like so and we will now get a Json string so as you can see serialization uses the field names not the aliases to serialize now since we have aliases though we could if we wanted to also serialize using the aliases instead of the field names again very commonly used in rest apis how do we do that very simple we do a model dump but now we also specify by Alias equals true and when you do that you can see that now it's using the aliases ID first space name right last name like so and age and if we do the model dump Json by Alias equals true we'll get the same thing but now of course we're getting Json string output instead of a dictionary now when we Ed the field object to Define an alias we lost the ability to set our field to some default value because remember how we were doing the default before we were just saying this n equals z well now I have this field over here so I can't do that anymore so instead what we have to do is we have to move the default inside the field so let's go ahead and do this model person base model and we're going to have first name is going to be a string and I'll make it nullable and that's going to be a field I'm going to give it an alias using the camel case version so first name this is again very common in rest apis where Json follows camel case convention but in Python we follow snake case convention for naming things but here I want to set a default and all I have to do is just specify default and then I'm going to do last name and last name I'm not going to make it optional so it's going to be string it is not going to be nullable and I'm going going to only set the Alias so I'm going to require it and I'm going to make it non-nullable so last name so now if I have a piece of data that looks like this where I have lastore name sorry I need to pass in the Alias because I specified an alias so I'm going to specify last name Smith we'll see that when we deserialize this this data so model validate data you'll see uh P equals my apologies you'll see that first name indeed was set To None it defaulted To None when we specify an alias we must use the Alias when deserializing data as you saw I'm using the Alias here to deserialize in fact if I was to try and deserialize data using the field name right first underscore name we will get a validation error so if I try and say first or last name because that's required so let me do last name if I try and set it this way then I'm going to get a validation exception as you can see we get that last name is required why is it complaining about that we passed it last name well it doesn't know what last name is lastore name and by default when you deserialize data in pantic if you have extra fields in your data it just ignores them there are ways that you can modify that behavior so that you can actually have it raise an exception for you or you can actually have it added to the model on the Fly basically but by default it just ignores it so it ignored last name here lastore name because it doesn't know what that is and so therefore it said well I need last name the camel case version and this would be the same if you had a piece of data for example that had let me take this so if we had last underscore name here it would be the same issue if we were deserializing from a dictionary or a jonon string so model validate and we'll give it data like so so we'll get the same exception so can we change that and the answer is yes and for that we have to do these model level configuration there are many of those I'm going to show you one here which is to allow population by field name not just by Alias so we want to be able to allow this so that we can populate Fields using either the Alias or the field name and to do that we provide a model configuration and we do this by simply creating another attribute in our model so from pantic we're going to import this object called config dict and then we're going to create our model so class person and basically I'm going to take whatever we had before so let me just go ahead and copy that instead of retyping it so I'm going to take this model over here that we had but I'm going to add a configuration on it so here you have to use model config you have to use that name it will not be treated as a field in the model because pantic understands what model config is and this is why the name is important it has to match and we're going to set it equal to an instance of config dict and in config dict we can specify various named arguments in particularly the one that we're interested in here is the populate by name and we set that equal to True by default it's false but here we're overriding that essentially so when we do that I can now go ahead and create a person and normally I would have to use the Alias I would have to use first name equals John and last name equals ALS let's say Smith like so but because I have populate by name set to true I can change that I can say no I want to populate by this field name first uncore name and you can see I'm leaving them mixed here I'm using the Alias and here I am using the field name so if I do that you can see everything worked just fine and if we have a piece of data like this one here then when we do the P equals person model validate data you'll see that everything works fine as well even though I have here the field name and here the Alias now we can populate using either we can deserialize using either okay next I want to talk about mutable defaults a little bit so returning to that one thing that pantic can handle is setting default values to mutable objects something that is usually problematic in Python you don't do that for example with function arguments and it's also by default disallowed in data classes but in pantic defining defaults this ways perfectly acceptable because what pantic does and let me let's let's go back to an example let me just do this model over here so we're going to have a list of integers and that's going to be an empty list so this is what I'm talking about right you don't do that in data classes you don't do that in Python uh default Arguments for functions but pantic it's perfectly fine to do that why because pantic basically looks at the default value that you have on a field and when it creates instances of the model if it sees that the default value is a mutable object it will go ahead and create a deep copy of that mutable object so that's how it kind of gets around that issue right by doing this deep copy so this is actually perfectly fine and perfectly legal in pantic in fact if we do M1 model and M2 model as you well know with python functions if we had that we would now have a shared reference to the same default list now in this case we don't if I go ahead and say M1 do numbers. extend and let's go ahead and extend it with 1 two 3 then you'll see that M1 the numbers has now been extended so now it's the list 1 2 3 but if I look at M two. numers you'll see that it's still the empty list but this does lead to an interesting point and that's default factories because sometimes we want to generate a default not as a static value but rather as a value that should be calculated each time an instance is created when that creation needs the default every time we create that instance if it needs a default we want that default value to be generated at that point in in time it could be for example generating unique identifiers obviously a unique identifier should be different for every instance of your model or it could be a date and let's say it's a date time every time we create an instance of the model we and if we don't specify the date let's say for a particular field we want it to generate the current date time right that kind of thing and of course you cannot do that by assigning a static value here right so instead instead you have to use something called a default Factory so let's take a look at it that will make a lot more sense when we do this example so I'm going to import date time and time zone from date time and now I'm going to create this log class so we're going to have from base model and then we're going to have DT is going to be a date time and I want it to be a field and I want to generate a new date time if I don't provide date time I want the default to be autogenerated as the current date top so for that we have to specify default Factory and default Factory expects a function and it's a function that takes no arguments and it should just return whatever that default is that you want to generate so in this case I want to generate date time. now but I want it in UTC so I'm going to say dat time time zone UTC now the problem of course is that that this over here is not a function that's an actual static value default Factory expects a function so all I need to do is just to turn that into a function make it a Lambda it will get called No values will get passed to it and it's going to return this and every time that the default is required when it's creating an instance of log it's going to call that function and let's do one more here let's do message string like so so now I can say log one equals log and I'm not going to specify the date time which means it will have to use the default and the default will be autogenerated so we'll say message one let's say so I've created that object now I'm going to do log 2 equals log and message equals message two so I've created those two instances and now let's look at log one you can see that I've got a specific date time here and if I look at log two you'll see that I have a different date time right it changed the date time so that's what default factories are for they can also be used very often to generate uu IDs where you want the default uu ID of course to be different for every instance of your model but it could be something else too it could be running let's say a query against a database to go and retrieve some data or it could be you know making an API call to some external API to retrieve some data and using that as your default value so it can be whatever you want it's just an arbitrary function here as I wrote it as a Lambda of course it could be a separate you know function using a def that's a lot more complicated than this Lambda over here so the next thing I want to look at is custom serializers pantic has a default way of serializing data for example serializing floats will result in a certain number of digits after the decimal point being used dependent on the actual ual float of course let's take a look so if we create this model inherit from base model and we're going to create number and that's just going to be a float so now I can go ahead and create an instance and let's say I use 1.0 and if I look at M do model dump so I'm not looking at the representation here I'm looking at the serialization that's very important that's a little bit different so we get this number here this float like so and if we did of course Json we'd get the same thing but as a j on string now let me go ahead and say model m equal model and then numberal 1 / 3 and if we look at what that is with a model dump or or a model dump Json I got to specify number correctly you can see that I have now all these over here right so it's a little bit of a different representation in the serialization and the same thing would happen with the Json dump we get kind of the same result now now similarly date times for example get serialized to Json using the iso format so if we take let's say DT equals date time do now and then time zone. UTC so this is just plain python right now so we've got that if I do DT ISO format you can see that once I fix that you can see that we get this ISO formatted string this is python doing that and we'll see when we use that in a model that's also what it's going to use so let's go ahead and try that so class model base model and we'll have DT which is a date time like so now I'm going to create an instance of that model I'm going to say m equals model DT equals and I'm just going to copy this over here and if we look at M we have that representation if we do a model dump now the model dump is dumping to a dictionary so by default pantic isn't going to try and serialize this date time to a string right that's what ISO format is it's a string so if you do a plain model dump you just get that daytime object as is but if you do the model dump Json then of course Json doesn't know anything about datetime objects it has to be serialized as a string and indeed you can see it was serialized to a string and it basically used an ISO formatted string very similar to what we have here using Python's built-in ISO format so sometimes we want to override this serialization now we have to be a bit careful since we actually have two modes of serialization right we are serializing to either a python dictionary that's the model dump so we're we're essentially dumping to dictionaries and python objects which is why with the model dump we can actually serialize to a datetime object but we can also be serializing to Json in which case it's going to have to be things like for example the string representation of the date time now let's say that we want to customize the float serialization so that all floats are rounded to two decimal places in both the dictionary and the Json serialization so we're going to start there so from pantic I'm going to need to import field serializer and let's go ahead and create a model and we'll do base model and I'm just going to create just the number float like we had before so I'm just going to do that one and now I'm going to create a field serializer field serializer is a decorator and it is going to be used to decorate a function which is going to be essentially an instance method so let's call it serialize float now of course it's an instance method so the first variable is going to be self but pantic also injects the value of the the field being serialized what is the field being serialized here well we have to specify it and we specify it in The Decorator itself we say well this is for the field number and there are ways where you can get the field serializer to apply to multiple fields in your model or all the fields in your model I won't get into that here but it is possible to do that so what do we want to do here well I just want to return round of value comma 2 so when it's going to serialize to dictionaries it is basically going to give me the rounded value as a float in the dictionary but of course when we serialize to Json we're returning a float here and then pantic is going to take that float and serialize it to a string because it needs to serialize you know the floats essentially to a to characters right in a Json string okay so once we have that then we can create a model model let's say we'll take the 13 example that we had just now and if we do a model dump you'll notice that now we get 0.33 not all those threes that we had before and then same thing if we dump to Json it is also 0.33 and that's because in both cases model dump in model Json this function over here was called now for the date time however I do not want to modify the serialization of the date time I only want to modify the serialization to Json because I want override the way in which this is essentially being represented in the Json I just want to customize that but I don't want to affect what's being serialized for the dictionary I still want the date time here I don't want to modify that I just want to modify the Json piece in fact I'm going to go one step further and say I only want to modify the serialization to Json if the number was not none or if the date time because we're going to be doing date time was not none now in this case it can never be none because we don't have you know it's not nullable so it can never be none and so it doesn't really matter but still in general you don't need to override the serialization when the object value is none because the Json serialization of none is going to be null and the dictionary D serialization of none is just going to be the same none object so we don't need to usually worry about those there are cases where you do want to override how things how the non object gets serialized but in general you don't so let me start with this model again and let's add a field let's call it DT date time and I want to create a field serializer to customize how DT gets serialized but I only want it to apply if we're serializing to Json so we're going to start again with field serializer and we're going to be setting it for this field DT and and then we can say def serialize let's say date time to Json and of course it's an instance method and it's going to receive the value being serialized and what I want to do here is I want to return my value which should be a date time I'm guaranteed it's going to be a date time I'm also guaranteed it's not none in this case because of the model so I can just basically call uh string format time and we're going to use this pattern over here we'll say we want the year slash we want the uh one character or two characters but we don't want the leading zero basically for the month then we want the day but again without the leading zero so percentage minus D then I want a space then I want the 20 the uh 12-hour clock and I want the number of minutes and I want the AM PM over here the problem though is that this is going to apply as we saw to both the dictionary and Json serialization in order to stop it and only make it apply to the Json serialization we have to specify an extra argument here in The Decorator called when used and it should be used when serializing to Json but and that's that's valid you can do that but I also want to say I don't want to handle this if it's none so in general I always use Json unless none even though in this case this is never going to be none but maybe it was a nullable field right maybe it was this in which case you'd have to now deal with the fact that value could possibly be none and of course this would fail so by saying Json unless none even if date time is nullable and none it's not going to run our serializer so let's go ahead and try it let's say m equals oh you know what let me go ahead and set it to be a default so I'm going to remove that let's go back to the default that we had so field default Factory equals and we'll say Lambda and it was date time. now time zone. UTC like so okay so now I can create and the reason why because I don't want to have to specify data every time so I'm just saying model number equals 1/3 for example so you can see we have our model if we do a model dump you'll see that we get our objects back so we get this float back and we get this datetime object back but if we do an m. model dump Json you'll see that it actually used our serializer this time and we have this format for the time okay so this was how to write a custom serializer custom field serial utilizer so now let's look at the next thing which is custom validators there are different types of validators available in pantic one type are before validators that run before pantic has a chance to validate and coer the data according to our field definition the second type are after validators that happen after pantic has already processed the raw data validated it and coerced it to the proper type as defined by the field definition we'll see examples of that that will clarify now before validators can be very handy to provide custom passing of data that pantic would otherwise be unable to do for example if we try if we put a date time in our model and if we try and pass a date that looks like this maybe uh 2024 sl11 3:15 p.m. that is not going to work it's going to fail validation pantic python cannot pass this into an actual datetime object if you want to accept that kind of input in your data because it's being provided that way for some reason then you would have to do a before validator where you would basically handle passing this into a datetime object before pantic takes its crack at it because it will fail on that so that's that's one very useful reason for having before validators now I'm not going to cover before validators in this video I'm only going to cover after validators otherwise this video is going to be just way way too long now validators are not just validation functions they are also transformation functions for example pantic validators can modify the type of the data being D serialized to coer it into the proper type we've seen that multiple times we passed it a string that was a valid integer inside the string and the field was an INT it actually modified that string into an integer so they there was a transformation that occurred it's not the validation is not just checking that you have the right type or the right values and then raising a Val validation error if something happens it also transforms data so validators in pantic are validators and Transformers transformation functions and many of pantic predefined special types I'm not going to look at any of them in this video but there are many special types like email and URL and IP address and past date and all kinds of things they do both validation and transformation so an after validator can therefore be used to transform the data as it is being deserialized after pantic has had a chance to basically coer it to the proper type check you know if you if you provided any additional constraints on the field it will basically pass that validation you'll get the data in that precise type and then it will call your C custom validator so let's take a look so from pantic we're going to import field validator and again that's a decorator approach and we're going to create class model and base model and I'm going to create this thing called absolute this field should say to called absolute that's going to be an integer and I want it to always be the absolute value of whatever gets passed in so I don't want to get raised an exception if the absolute value that's passed in is let's say negative you could do that and to do that you would use a field object and then you would use this constraint saying it has to be greater than you know strictly greater than zero or if you want greater than or equal to zero right so if you do something like this then if you can create a model and you can pass in the absolute equal to 10 that will work just fine but if you try to do model absolute let's say -5 you're going to get a validation exception because it failed this constraint over here so what I want though is not that what I want is I want to essentially modify this value right so that if a negative integer was passed in or a negative value was passed in I wanted to become a positive value I just want to take the absolute value every time so to do that I'm going to have to create a validator that's going to be a transformation validator so we're going to do a field validator and this is very much like the serializer you have to just specify which field or Fields because you could actually specify multiple but in this case I only wanted to apply to this one field called absolute now field validated unlike serializers if you think about serializers serializers will run after the model's been created and everything validators on the other hand are going to run before the model instance has been actually created right because pantic is in the process of creating an instance of your model but it's validating the data and so on so that's why serializers are instance methods but field valid datas are actually class methods and here we're going to call it make absolute of course it's a class method so it takes class as its first argument and then value technically you don't have to pass you don't have to set class method if you do field validator with a function pantic will automatically wrap it with class method but if you're using linting things like that or idees they're going to complain saying well you know you're using class here but this is not a class method you need to decorate it with class method so you should do it in this order though you first do the class method decorator and then on top of that you stack the field validator that's important and so all I want to do here and the value is of course the value that's being validated it's going to be this absolute value that's passing through our validation and I just want it to be the absolute of that value now you'll notice I'm not checking to see what the type is of value because it could be a string and which case absolute of a string isn't exactly going to work but this is an after validator so this means that by the time this field validator is called the absolute field has already been validated and coers to an integer so what we get here in value will definitely be an integer and it will definitely not be none since this is not a nullable field and so we can basically without testing or checking or guing for anything we can just say return the absolute value of value so when we do this we can now say say model absolute equals let's say 10 that's going to work correctly but we can also say model absolute equals -10 and you see what happens our data now is still 10 so the absolute value was returned so one thing that's important to note of course because this is an after validator is that it only gets called after pantic has had a chance to basically take this and turn it into an integer and then it calls our validator but if it fails validation then our validator will not even get called let me go ahead and let me put this print function in our validator just so we can see what's being called that it or that it is being called and what type the value is so that's the only change that I made so when I call it with absolute 10 you can see that it was running the custom validator with a value of 10 and it was an integer and then the same thing of course with would happen with -10 but now look what happens when I call it with let's say the integer -10 but as a string so that's not an integer that's a string but look what happens in our validator we still received the integer and we received -10 so this is what I mean by pantic takes FAL crack at it to do whatever it can here so if there was field constraints like the integer you know has to be less than you know a certain value or greater than a certain value whatever the case is so if I say field must be greater than or equal to 00 so let's say I'm only going to allow that right so if I try and do this model absolute equals -200 of course I'm going to get an exception right I'm going to get the validation error input should be greater than 100 and moreover you can see that that print statement that we had was not called and that's because it failed validation at this stage of the game so therefore the after validator here never got called now of course I mentioned that we can use validators for Transformations but of course validators can also be used for validating data so let's take a look at this example where I want to define a field that should be a list of unique integers so this is purely validation now I'm not going to do any Transformations I just want to make sure that all the elements of a particular list are unique so let's go ahead and create this model so class model We'll Inherit from base model and we're going to say numbers and then list int equals and I'm going to use that default that's perfectly acceptable and now I'm going to create a field validator it's going to apply to the numbers field then of course it's a class method and then we're going to just say ensure unique I'm going to call the function ensure unique and of course class and it's going to receive a value you can name that whatever you want it's a positional argument identic will pass it positionally so you can call that whatever you want I'm going to call it numbers since that's the field name and here I'm going to test this way I'm going to say if the length of the set of numbers so I know that I can do that because it's a list of integers so of course sets can only contain hashable elements but integers are hashable so this will work perfectly fine so if the length of the set is not equal to the length of the numbers that means I have some non-unique elements in that list so in that case I'm going to raise an exception now you do not raise a validation error in pantic instead there are a few other errors you can raise one of which is value error and that's probably the one you should raise the other two are assertion errors but don't use those because P python can be run without executing assertions by just setting essentially a command line parameter so you cannot rely on assertions to you know generate your your your exceptions um so raise a value error there is also another custom error that's available in pantic for validation errors that is a bit more flexible but really here value error works most of the time and here we're going to specify an error message so elements must be unique so you raise a value error you do not raise things like type errors and key errors and those kinds of things because those will bubble up as regular errors when you raise a value error pantic is going to transform that error and raise it as a validation error the same validation error we've been seeing it will take that value error and instead raise the validation error which is what people expect when they get a validation error of pantic the expectation is it should be a validation error not a type error or a key error right so you you want to be consistent so to do that just raise a value error and of course if nothing was wrong then we're just going to return the numbers as is remember that your validators have to return whatever the validated value is in this case I'm not doing any transformation on numbers I'm just doing this check so we can go ahead and create numbers equals 1 2 three like so that's going to work just fine and of course I made this lower case so let me change that to upper case let me change that to uppercase let's be consistent in our naming conventions now what I'm going to do here is I'm going to try I'm going to try model and I'm going to pass it a list of numbers that are not unique and you could even you know since remember this is an after validator so even if I pass in elements that are not integers it will coers them to integers if it can so I can pass it one one two and three like so for example that is still going to pass validation penics validation but of course it should fail our custom validator because we have this repeated element so accept that validation error as ex and then just print the exception as you can see we get the validation error the elements must be unique all right so the next thing I want to talk about is nested models and essentially you can Nest pantic models and the deserialization and serialization of the submodels will be handled by pantic automatically so let's say that I have this model here or this data I should say it's not a model yet this is data that I'm getting from somewhere and I want to deserialize that into into a pantic model now you'll notice that we have first name is a string last name is a string but born is a dictionary and inside there we have place which is itself a dictionary and then date which is a date so as you can see we have these nested dictionaries and so the way we're going to model this is by using essentially nested pantic models and to do that we use composition so let's go ahead and actually do this it's way simpler than it sounds and let's go ahead and create the model so the way I do it is I work from the inside out so I started this kind of most nested level which is place so we're going to create a model for that I'm just going to call it place for lack of a better name for it and what does Place have place has country which is a string and it has city which is also going to be a string great now I have this bone thing bone has two fields in it play base and date so I'm going to call it bone base model it's going to have this place field now what is the type for Place field well it's exactly this place model here so we're going to specify that as the type hint this is model composition we're essentially using composition to Define these nested pantic models and then we have date and date is going to be just a date so I'm going to need that date that I imported already so we're fine now I don't want to call it date because that's kind of weird looking me it technically should work but of course we don't do that and what I'm going to do here is just call it DT so this means though that I have to specify an alias so we'll say field Alias equals date because that's what it's specified as in our dictionary that we're going to deserialize okay and then lastly working our way further out now we have this person object over here and it has three Fields first name last name and born so now let's go ahead and model that so class person base model it has first name and I want to use Python conventions so it's going to be first name that's going to be a string I'm going to make it notable so I don't care if the first name is no or not and of course I have to specify an alias so we'll specify the alas to be first name and then also I'm going to specify default so it won't be required so it's nullable and optional and when it is defaulted I will make it none which is why of course I had to make it nullable keep that in mind then last name is going to be a string and I'm not going to make it nullable or opt or uh optional but I do have to define the Alias so that's going to be the Alias here and then finally I have born so born is what well it's an instance of this born model and I'm I'm not going to make it optional and I'm not going to make it nullable if I wanted to make it nullable I would just say born or none and if I wanted to make it optional I would just have to specify a default in this case since it's nullable I'm going to set the default to none so you could do that as well if you want to and now we can go ahead and deserialize our data so let's try that and let's see if I coded everything correctly so we'll do model validate and then we'll pass it data and I just have to type that right and no validation exceptions that's great and then if we look at what the representation is you can see we have first name last name born is an instance of the born place is an instance of the place model country City and then we have our date time and you can access the data of course using notation they're nested so we can look at arthur. born where were they born and maybe I want the place so now I need to look at the place under born and I want to see the uh country for example you can see we get lunar colony of course you can also assign right these are not frozen mut immutable Fields you can assign values to them as well using the same thing and when you do the model dump of course you will get those nested dictionaries back as well you can see we have a nested dictionary and a nested AR and then the same thing will happen if we model dump to Json and of course if we model dump to Json so alther model dump Json it doesn't look very easy to read neither did the dictionary for that matter so a couple of things you can do to get a better um a a better idea of what dictionary looks like when you're printing it you can say from print import PR print and then then you can just print this model dump it will try and format it in a slightly better format you can see it's a little bit better in my mind actually the Json one is the best but of course not this way so what you can do is you can actually say author. model dump Json and then pass an indent for example equals 2 and then when you print you have to print it cuz if you just do the representation I'll show you that then you can see that the indent equals 2 now has resulted in things like new lines and spacing and all that kind of stuff but it's very difficult to actually see what's going on here instead what we want to do is to actually just print it and when we print it now we get this nice looking output over here all right so this was a very quick overview of creating models in pantic with some extra functionality such as custom serializers and custom validators but we've really just scratch the surface there is a whole lot more to pantic than just this and depending on your needs this might be sufficient what I've covered here but more likely you're going to have to dig deeper into the library for example instead of defining these single-use custom serializers and validators that we saw directly in our models we can create custom types using type annotations that attach those validators and serializers to the type itself and then that type can be reused used across Fields different fields in different models and so on so you get a lot of reusability and pantic makes heavy use of annotated types now I also mentioned that we have before and after validators and in fact validation is a pipeline where you can specify multiple before and after validators either directly in the model using the decorators we saw or to types using the annotations that I just mentioned and pantic will execute the validators one after the other to arrive at the final result so you can actually set up these validation pipelines and then pantic also supports lots of other features like how to handle extra Fields fields that are in the data but not in the model how to make fields or the entire model immutable how to autogenerate aliases very often in our models we're going from camel case to snake case because we're dealing with python variables and we're dealing with Json variables or Json data Json on keys so we have to go between camel case and snake case and you can use the Alias you can you know modify that yourself you can specify it yourself but there's also a way to automatically generate let's say the aliases the camel case aliases from the snake case field names very handy to have you also have other things with aliases you can have serialization aliases and validation aliases you can do dependent field validation so if you have let's say a model where you have a start date and end date in the model you may want to add a validation that makes sure that the end date doesn't come before the start date you can do that again with pantic it can handle enumerations it can handle model inheritance where you can have customized base models with these configurations I showed you one which is the populate by name but there's lots of other configurations that you can do at the model level like extra Fields autogenerated aliases and so on instead of having to reach type that information on all your models you create a custom based model with that configuration then you inherit it in your other models so it works that way as well there's a lot of specialized types that pantic provides things like emails and URLs and uu IDs and many more and there's a lot more features far more than I can possibly cover in a video like this but if you do want to go deeper and I know this is a Shameless plug but if you do want to go deeper and explore these different topics and really leverage pantic you can check out my udmi course where I basically Deep dive into pantic I'll even I even show in that course how you can leverage pantic to validate just plain function calls to python functions to your own custom functions and so instead of writing validation code inside your functions you can actually use pantic to define these things with these annotated types and the field validators and so on and to to valid validate the data that's being passed when it when one of your functions is being called so that's very very handy to have as well and I cover that in that course as well as a ton of other things as well all right so thanks for watching and hopefully you enjoyed pantic V2 it's a great library and it's extremely useful for a variety of topics not just for rest apis you can use it to validate data coming in and out of cues you can use it to uh validate and serial ize or deserialize data in let's say reddis let's say you're storing your data you know your values as a Json object well you're going to need to deserialize and serialize that data and possibly validate it as well or maybe you're storing it in a Dynamo table and one of your Fields happens to be Json and all kinds of things so there's tons of applications of pantic other than just fast API all right thanks for watching

Channel: MathByte Academy

Views: 4,614

Keywords: python, mathbyteacademy

Id: ok8bF8M7gjk

Length: 73min 20sec (4400 seconds)

Published: Thu Dec 07 2023