Python & FastAPI - Annotated Type for Data Validation + Metadata!

in this video we're going to continue our exploration of fast API and we're going to look at something in this video that's quite new to Python and new to fast API and that's annotated parameters now annotated parameters can be passed to fast API Handler functions and they can provide additional metadata and also validation constraints to the parameters that are coming into those fast API functions as an example of this imagine a search query parameter in the URL by default a query parameter string can take any number of characters but you can use annotations in order to limit the number of characters as one example to let's say less than 50 characters and we'll see an example of this later in the video but let's get started and dive in and we're going to start by looking at the typing module so we're on the documentation for the typing module here and you can see that this was added in Python 3.5 and this module provides runtime support for type hints very useful when you're using it with a package such as mypie and if you're interested in any mypie videos let us know in the comments now what I want to do here is search for typing do annotated and that's what we're going to use in this video it's commonly used in Fast API and other Frameworks so let's go to that section of the documentation here I'll leave a link to this specific section below the video and what this does is it adds context specific metadata to an annotation so if you want to add metadata X to a type T you can use the annotated type for that and you're annotating the type T and you're passing the metadata X and the metadata that's added can be us used by Static analysis tools or at runtime now this is an important Point here if a library or a tool encounters an annotation and it has no special Logic for that metadata it's going to ignore the metadata and it's simply going to treat The annotation as the type T and there's some other information here that you can read I'm going to scroll down to the code example and here's an example where you have a data class called value range with a low and a high integer and below that we create two variables T1 and T2 and we use the annotated type typ here and we're annotating an integer in this case with the value range now this value range is just additional metadata and the purpose of it in this case you can infer that it's limiting the value of the integer to a range between - 10 and 5 however by itself this is not going to do anything the responsibility of how to interpret the metadata lies with the tool or the library that encounters The annotation now let's see a super simple example I'm going to go back to the code that we had in this fast API project and then this project we had a fast API application what I'm going to do is Define a new file here I'm just going to call it test.py and at the top I'm going to import the annotated type from the typing module now let's define a function here called double that takes a number X which is of type integer and this number should return another integer so what it's going to do is very simple it's going to take X and multiply it by two and return that number now let's create a variable called result and call the double method and print that result to the terminal so let's run Python test.py and we get the number eight at the bottom so we're doubling the number four very simple function it takes an integer and it returns an integer now what we could do is we could use the annotated type that we've imported at the top to provide additional metadata here so what I'm going to do is copy this type and instead of an integer we're going to use the annotated type and we're going to annotate the type of integer with another value here and this is the metadata and this can be anything we want remember the responsibility of interpreting this metadata lies with the particular tool or framework that we're using for example fast API as we're about to see in this video but what I'm going to do here is just simply pass a triple and we're going to go from the number 0o to 100 in this triple so let's infer that this is something similar to the value range we can have an integer only between the number 0 and 100 now what I'm going to do is Rerun the code here and you're going to see that it works exactly the same as it did before we get the number eight back but we're using this annotated type as the type for this parameter now remember that the idea of this annotation that we've just created is that we want to use the numbers on the left and right as the boundaries we want the number to fall somewhere in between those but right now if we just pass a number that's outside that range we are going to still get back the number and you can see it's been returned on the command line here there's no actual checking of what's here there's no interpretation of that at the moment from any of the tools that we're using now later in the video we're going to use this annotation to actually add some validation to the double function we're going to write code that can read this metadata that's passed to the annotated type and then Implement a range lookup based on that now that's more advanced stuff that's going to come at the end of the video but for now we're going to go back to fast API and we're going to focus on the annotations that it offers so we're back on main.py and this is where the application is defined along with some end points here for example this get request to/ bands now what fast API does is it uses annotations to add meod data to request parameters let's see an example of this I'm going to go to the documentation for fast API and it's this page here on query parameters and string validations now let's look at the code in this example here we have a function called read items that takes in a search query parameter q and that parameter is of type string or none now let's say that we want to perform additional validation to this search query parameter for example if we scroll down here we can enforce that even though Q is optional whenever it is provided the length does not exceed 50 characters and we can achieve this by using the query object in fast API along with the annotated type so in the example code what we have here is we have the same function read items that takes that parameter q but rather than just being a string or none we are now using the annotated type and we pass the type of string or none to that but we also pass this annotation here as the second option and this annotation uses an object called query which is imported at the top from Fast API and it passes the keyword argument of max length equals 50 now the purpose of this second option here to the annotated type is to perform the validation of what's coming in if it's a string we're going to check and make sure that the max length is not greater than 50 characters so in this case fast API uses this query object to perform additional data validation on the query parameter q and it uses the annotated type in Python to do that now fast API knows how to interpret this query type Remember by default this annotation will not do anything unless the library knows how to handle what's passed as the metadata and you can use annotations not only with the query parameters in fast API but also with path parameters with headers and with cookies so if you need to perform validation of any of that information you can use the annotated type to do so now let's go back to the code and we're going to look at the endpoints here and if we scroll down we're going to go to this endpoint and it's the app.get endpoint for an individual band and we're finding that band by its ID now the band ID is a path parameter and that's passed to the function as an integer type now what we're going to do at the top is we're going to import the annotated type so let's import that underneath the schemas and from Fast API we're also going to import the path object and the path object represents these path parameters such as the band ID now what we're going to do instead of typing this as an integer we're going to use the annotated type now and the type will remain integer but we're now going to pass some metadata as the second option to annotate and we can use that path object that we imported and one of the things we can do here is pass a title that represents the path parameter so the title here is the band ID we're passing that to the path object and we're using an annotation to do that now what I'm going to do is run the server at the bottom here and we can do that with the uvicorn main app command and we're passing the the reload flag here that's going to start the server on Local Host 8000 and what we're going to do is go to the/ redock endpoint and that will return the redock documentation for all of our endpoints in the fast API and what we're going to do is we're going to go to the band endpoint and let me make this bigger so we can see it better and for the band endpoint you can see the path parameters the one that we need here is the band ID but now because we've added that title you can see that it's now showing up on this documentation so by adding the path here using this annotation and giving it a title we are adding additional metadata that is now showing up in the redock documentation so that's an example of adding metadata that then informs the documentation for your API but what we also can do is we can use this annotated type to add validation to the incoming data on a query parameter so let's do that just now we're going to go back up to the endpoint that returns all of the bands in our application and what I'm going to do for now is remove this code for has albums we don't need that anymore and once we've done that we have our endpoint we're going to add that parameter q and that's going to be another annotated type now we expect a string here but it could also be null so we're passing a union here of string and none and the second option that we're going to pass to annotated is that query object that we saw earlier on and we're going to give it a max length of 10 now I need to import query at the top along with path from Fast API and if we scroll down here we've got an error what I need to do as well is give it a default value so the band's endpoint that will return a list of bands but it now takes a query parameter q and this will either be a string or it will be none and we're giving this additional validation here by using the query object from Fast API and the validation says that we cannot have a query parameter q that has a length greater than 10 and this query parameter is going to allow us to search through the bands and only find the bands that match the expression that's passed in as that query parameter so let's go to the code here in our fast API fun function and we're going to implement this now if we have a parameter Q we're going to filter down the band list based on that parameter so we're going to return each band B for B in the band list but we're going to add a conditional expression here and we're going to check the query parameter q and we're going to lowercase that with the lower function and if the text that's passed in in this query parameter let's say that text here is ABC if that is in the band's name if the band's name contains that text we want to return it in this list comprehension so we're going to take that query string q and we're going to check if it's in the band's name and we also want to lowercase that band's name and that's all the code we need to write to perform this very simple contains lookup using the search query parameter Q Let's test this out and go to the browser now I'm on the bands endpoint here this returns all of the Bands but if we want to search for let's say a band that has the word black in it we could go to the URL and we have the query parameter Q now and I'm going to pass black in there and we get back Black Sabbath in this case and if we pass something even more simple such as just the letter T we get back all of the Bands whose name contains the letter T and let's do one more if we pass wo we get back the wuang clan now the important point for this example is that we now have this annotation and it should restrict the parameter Q's length to less than or equal to 10 so let's see if that works if we go back to our endpoint here and we type something that has greater than 10 characters we get back this message and the message says that the string should have at most 10 characters and the error type here is string too long so this validation error is occurring because we've added The annotation and we've added the query object with the max length so that's an example of how we can use annotations in fast API to perform data validation and what I'm going to do is go back to the redock documentation and we're going to go to the band list endpoint and we can see the query parameters for the genre and also for the search term Q if we expand that you can see here at the bottom the string Q must have less than or equal to 10 characters so that annotation is also represented in the documentation here for this endpoint so that's a very quick example of how fast API uses the annotated type to perform data validation on incoming query parameters and other data and also to add metadata to our endpoints and to the parameters coming into the endpoints for example by giving them a title that can appear in documentation and that extra metadata can be useful when you're sharing that documentation with other developers now I want to move on at the end of this video and dive a bit deeper into how annotated works and we're going to write some code that's going to take this annotation that we wrote earlier in the video and actually implement the range lookup based on The annotation so we're going to add that validation now based on this Two element tple and we're going to write this in the function but later on we're going to extract it to a decorator and the code for this is going to be heavily based off this answer on stack Overflow and it's the top answer here and when we write this code it's going to demonstrate some features of Python's typing module so let's get started what I'm going to do is go back to the python typing module and this is the documentation for that we're going to search for a function called get type hints now you can call get type hints and it's going to return a dictionary containing the type hints for a function a method a module or a class object so this returns add AR what we're going to do is go back to the code and we're going to import that function from the typing module it's called get type hints now it's our responsibility to interpret this annotation and write the logic in the function for dealing with this metadata so let's start by getting the type hints here and what we're going to do is we're going to call that get type hints function and we're going to pass our function name double into that function and then we're going to print out what we get back when we call that so let's run the program with python test. pi and what we get back when we print the type hints is the parameter name X which is of type integer and we also get the return type which is also of type integer but we don't get any information about The annotation now there's a keyword argument we can pass to this function and that's the include extras keyword argument if we set that to true and we rerun this code at the bottom this time the type for X we can see that that is an annotated type so what we want to do in the logic of this function is get access to this annotation and there are a couple of extra functions that we're going to import at the top in order to do this so let me import these now one of them is called get origin and the other one is called get arcs these are both from the typing module now what we're going to do is we're going to extract this x value from the type hints that we got back and that was a dictionary so we'll create a variable here called hint and we are going to index into the dictionary and we're going to get the key with the value X here so let's print that again to the terminal and rerun this so we get the annotated type and we can actually check if this is an annotated type by using this get origin function so let's go back quickly to the python documentation and I'm going to search for get origin if we call this and pass a type into it what this does is it gets the unsubscription version of a type so if you have a typing object that has the form of this here which is exactly the form that we have for the annotated type where we have a Type X in this case annotated and then we have some additional data here in the square brackets the get origin function is going to return the unsubscription so we're checking if the parameter X is an annotated type and if that's true what we're going to do is we're going to get the hint type and also the hint arguments by calling that other function that we imported called get args and this function gets the type arguments with all substitutions performed so let's see an example of what that means we're going to pass the hint that we extracted into that for the parameter X and then underneath that I'm going to print these two values to the terminal so let's see what happens now with these values and I'm going to remove the final print statement when we run the code we get back the class of integer for the type here and what that means is that the raw type for X is integer and when we look at the arguments we get back The annotation here as you can see it's a list with that tiple that we passed as the metadata to the annotated type so we can use get arcs to extract that information from the type that's passed into the function now what we're going to do is we're going to take this triple here and we're going to destructure that into two values low and high so how do we do that let's go down here and we're going to create two values low and high and in order to get them we're going to look at the hint arguments we're going to get the zero element which which is the first element in this list and it's the only element and that's the tiple there so we unpack the tiple into two values low and high I'm going to clear this print statement and we're going to print out low and high to the terminal and again let's clear this and run this and we get back 0o and 100 now and at this stage we can actually perform the range lookup and we're going to use an if statement for that now the condition that we need to abide by here the low value must be less than or equal to the value of x that's passed into the function and that X Val must also be less than or equal to the high value that's passed in in this tiple so what this code here is going to do is make sure that everything is okay and if that's the case we can progress in the function but what if it's not okay what if something is not right here and the value X is for example less than the low value or greater than the high value we're going to invert this if statement we're going to check if this condition is not true and if it's not true we're going to raise a value error and we're going to see that x Falls outside of the boundary between low and high so this logic here actually implements some code based on this annotated type by introspecting what The annotation was and checking if the value passed in meets the expectations set out in that annotation so let's test this out just now we have the double function being called with this very high value and that should no longer work when we run this code and if we expand the terminal here you can see that we get the value error and the number we've passed in it's outside of the boundary between 0 and 100 and this is now flexible we can change this code and let's say change 100 to something like 10 million I think that's 10 million but I haven't double checked that but when we run the code it's no longer going to return a value error instead we get the result and we now have the ability to control the boundaries of what's passed into this function by using The annotation now this is nowhere near as sophisticated as What fast API is doing but these functions such as get type hints and get origin they are commonly used in Frameworks like Fast API and and pantic in order to check if we have annotated parameters and if there's any data validation defined on those parameters so this is now working but what we want to do now is Outsource these validation checks and that's because this double method here should only really worry about doubling the function all of this code here should be moved out into some kind of object or decorator and that's what we're going to do now we're going to create a decorator so what I'm going to do at the top here is I'm going to import the wraps function from Funk tools and at the top here I'm going to Define a function called check value range now because this is a decorator we're going to take the function that we're decorating as an argument and then we Define an inner function I'm going to call wrapped and this is going to take any arguments passed into the decorated function and in our case that's the argument X now what I'm going to do here is copy all of the logic from the double function going to copy that and we're going to paste that into this wrapped function here and I'm going to fix the indentation so again we're just getting the tyght pints from the parameters past to the function we're checking if these types are annotated types and then we are extracting the arguments and we're getting the low and the high values from what's passed in and then we're performing this validation check here now there's one last step that we need to perform inside of this function here if the validation here is fine if we don't raise that value error what we need to do is we need to actually execute our function so what we're going to do here is we're going to return the original function which is called Funk and we're going to pass the value X the param into that function so Funk here is the function that's passed in as a parameter to our decorator and the wrapped function here defines the logic for checking The annotation and Performing that data validation now from our decorator we need to just return the wrapped function here and the last thing to do is decorate this function with the wraps decorator from Funk tools so this is a bit complicated but we're just creating a decorator that's going to implement this logic to check the annotations and we're returning that function when we use this decorator now let's copy the name of this function and go back down to the double function and I'm going to remove all of the code here for the type checking and we can use the decorator now and the name of the function is check value range and the boundary here is between 0 and 10 and we're passing in a number that's much higher than that so let's expand the terminal and run this script and we get the value error now that that number is outside of the boundary but the double function looks a lot cleaner because we've extracted the logic for looking at The annotation into a decorator and if we pass a value such as five that is within the boundary we're going to get back the result 10 so that's useful we have a decorator now that looks at our annotation and performs the data validation based on what's passed into the function and it uses these utilities from the typing module such as the get type hints function in order to do that now you might be asking why would we do this now one benefit is the flexibility that's provided by the anit type we can very easily change what's passed in here and we can easily change the lookups and the boundaries that we Define in order to validate the incoming data and that's quite powerful and it's used to a good effect in fast API and if you write a library that uses these annotations other people can then use the annotated types and they can do that in a flexible way that's very easy to implement and it's very convenient and modern way of doing things like data validation if you have packages such as fast API and pantic that Implement a lot of this functionality now the important point is that annotations will rely on contextual logic that you will write or that another package will write on their own the annotated types do nothing so the metadata that's passed in here in order to do something useful with that there has to be code that looks at this and understands how to interpret what's passed in and then knows what to do based on those values and to sum up what fast API does it uses annotations to add logic validation and metadata to Dynamic incoming data from path parameters query parameters headers and cookies so that's all for this video it's been a bit of a deep dive into the typing module and how we can use some of the constructs in that module to create validation around annotated types and we've also seen how fast API uses the annotated type in order to perform these data validations and add metadata and information to your API documentation so thank you for watching if you've learned anything or enjoyed the video give it a thumbs up on YouTube And subscribe to the channel if you've not already done so if you have any suggestions for similar content let us know in the comments and we'll see you in the next video

Channel: BugBytes

Views: 3,196

Keywords:

Id: 9Hc-mql6Gv4

Length: 23min 41sec (1421 seconds)

Published: Wed Jan 31 2024