HUT 8 Techtorial - DevNet Associate DEVASC 200-901: Parsing JSON Data with Python 3 - Part I

[Music] [Applause] [Music] bienvenidos who sharm d and welcome to this hut a networking tectorial on the new cisco dev net associate certification 200 - 901 in this tutorial we're going to be focusing on how to parse JSON data using the Python programming language now there is a little bit of foundational work that we have to cover to fill in some of the gaps to get from programming with python to actually parsing JSON data so I'm gonna be breaking this down into multiple parts and I think it's gonna end up being three parts in total so what are you going to learn as part of this tectorial well we're gonna be talking about Python data structures specifically dictionaries because those play an integral part in the application that we're going to be building and we're actually going to be using the MapQuest developer application programming interface or API and we're gonna be making a request and we're gonna get data back that's going to be in JSON or JavaScript object notation format and I'm gonna talk a little bit about the other types of formats that are out there as well so let's go ahead and clarify what we're going to be doing here so we're going to introduce learners to the art of parsing JavaScript object notation or JSON data using the Python programming language by way of the MapQuest API now we have some prerequisites that are going to be very very important first and foremost you're gonna need access to Python 3 now everything I'm going to be doing is going to be on an imac Pro and on this iMac Pro I have the anaconda distribution installed and the reason for that is the anaconda distribution comes with hundreds of different Python libraries or modules that you don't in the Mac OS X release of Python 3 but we're going to be talking about pip and we're gonna look at how we would install some of the additional modules that we're going to need access to so don't panic if you don't have anaconda again as long as you've got python 3 you're gonna be able to do exactly what we're going to be doing here now let me transition here real briefly to this dev net associate exam topics web page on the Cisco Learning Network the highlight in the exam description in terms of the topics that we're going to be covering where this information that we're going to be covering Falls and you can see right here right out of the gate compare data formats XML JSON and yam all describe parsing of common data formats XML JSON and Y amel and you'll notice two Python data structures and those are the Python data structures I was making reference to earlier now the dictionary in our application plays a critical role that's why I highlighted the dictionary but again we'll be talking about other Python data structures like the list so don't panic if you're not familiar with those and you can see down here understanding and using api's construct a REST API request to accomplish a task given API documentation and that is actually what we're going to be doing as part of this activity we're going to be using that MapQuest developer API in order to get back data in the form of directions in JSON format write that common data format which is going to be JSON and we're going to parse that JSON data to pull out or extract the data that were looking for all right well let me clear the screen here now the reason I said Python 3 is because Python 2 has been sunsetted and I've got a quick webpage here at python.org and you can see that as of January 1st 2020 so over a month ago there will be no more improvement or development done on Python 2 so python 2 is end of life end of support end of everything now another prerequisite is going to be this JSON view and I'm going to be using the Chrome web browser here Google Chrome and you would just simply add it to Chrome and it's gonna tell you it can read everything and I'll just say add extension and then it's gonna go through and now it has been added so JSON view is what we added and we'll come back to this a little bit later on when we go to take a look at what some of the JSON data is going to look like when we make our request so again this video is going to be tightly coupled with the dev net Associate exam blueprint in terms of comparing data formats describing the parsing of the common data and constructing rest api's or REST API requests I should say all right so let's take a step back here the MapQuest developer API key now you're gonna need an API key but you get 15,000 requests if you have a key so I'm more than happy to share my key here with you and this is the key value right here and again remember you only get 15,000 requests I highly recommend it takes not more than two minutes to sign up for your own MapQuest developer account and you get your own key and that key value is going to be critical because that is what we're using as a token to authenticate our REST API request that restful request that we're going to be sending to the MapQuest API and so we need that key and you can use my key there or you can go ahead and register for your own API key and then we're going to need access to the Python requests module now this mam does not come by default with the OS installation of Python 3 but I'm gonna talk a little bit about PIP here in this video and again in this video we're gonna take this we're gonna start the script and we're gonna get it all the way to the point where we're making the JSON or I should say we're making that restful request to get the JSON data and we're actually gonna see what that data looks like and that's where we're going to break and I'm gonna split this up again you'll see that it there'll be a number of modules here that we want to break this up into to make it a little easier to understand so we've got some key terminology that we need to talk about so JSON is JavaScript object notation well what does that look like if I was to pull my web browser back up and come over here to the MapQuest developer API for thee get route API you can see that this is how I'm going to interact and here is what the JSON data looks like now it looks like this in this box nice and clean and again if you're familiar with Python you'll notice that we've got what appear to be dictionaries here now the curly braces in JSON represent objects right so JSON delineates objects with the curly braces but it looks like a Python dictionary so when you see these curly braces think it's a JSON object now JSON also has the square brackets and when you look at that you may be thinking oh it's a Python list well JSON calls these arrays right so similar but not exactly like Python so again when we see the curly braces with the JSON we're thinking a JSON object and there's many JSON objects in here as you can see when you see the brackets all right so the square bracket that is a JSON array right similar to the Python list and it looks nice and clean here when we see what our application returns it's be all jumbled together and a little more difficult to understand but that is it and here's what I want you to think about when we looked at the exam topics write data formats right compare these common data formats and so that's all XML JSON and llamó are is they're simply different ways to provide you the data to your request the answer or response to your restful request and it's gonna come back in one of these formats right now you can sometimes pick the format sometimes you can't pick but the three common formats that the exam is gonna be covering our XML JSON and Y amel we're working with JSON but the takeaway here is think of XML JSON and Hamel is simply three different ways of responding to your request and it's just a different way to format the data and if I were to go back over here to the directions API for the get route you can see that that's what XML looks like it looks very much like HTML and that is a format that you can get a response back in this format this common data format can be given to you when you make your request to the MapQuest API and again we are focused on JSON so that is what JSON is and again it's just a data format Gamal XML the same thing a data format but those formats are all different so now let's talk about one of the very confusing terms that learners get kind of confused with here and that's representational state transfer or rest now rest is not a programming language rest is not a common data format like JSON XML or Y Amal rest is nothing more than a framework and you can think of it in loose terms think of it as like a standard and for something to be a part of the rest standard it needs to meet certain criteria certain constraints certain characteristics or traits and so rest in order for something to be considered a part of the rest architecture or the rest framework it needs to meet three standards and we're talking about web services API is here right just like the MapQuest API we're going to be using right it follows and meets the rest constraints or the rest characteristics and traits and so what are those three things so a base you are I and you can see we've got that term down there our uniform resource identifier and you might be saying well wait a second how is that different from a URL right a Uniform Resource locator well just think of it like this URI URL and web address they're commonly used as synonymous terms but technically a URL is simply a type of you are I and it's one of the most common right because everybody's familiar with HTTP colon forward slash forward slash it's and then you put in your your URL right so a URL is just a type of URI and you'll hear me use web address or URL but you need to have a base URI so that's criteria number one criteria number two is it needs to have standard HTTP methods like post or get or delete or put or head right and then the third criteria is it needs to have a media type there needs to be some sort of a media type involved so again a base URI a standard HTTP support for standard HTTP HTTP methods like get post delete etc and then immediate i p-- and so now let me transition back here to the browser because you may be saying we'll wait immediate type that doesn't make any how I've never heard JSON referred to as a media-type well lo and behold RFC 46:27 is the application JSON media type for draw JavaScript object notation right and so here is your media type so again a web service API that wants to adhere to the rest framework right to meet the criteria to be considered restful and that's all restful is restful is some sort of web service API that meets the rest criteria and we call that a restful api and so in order to meet those three in order to meet the three criteria if you've got a a base URI standard HTTP methods support an immediate type like JSON right here right you are considered or will could be considered restful and so those terms are all very very important terms all right so let's go ahead and let's get on to the CLI here and let's talk a little bit about sorry about that talk a little bit about what it is we're going to be doing now remember the first criteria I said is that we need to have the requests module but what I want to do first is let me run the application and this is gonna be what our application looks like when this application is set up and fully configured so I'm gonna enter my starting location I'll put down Baltimore my destination location let me put Miami I'm gonna hit enter and you can see we're gonna get a set of directions here and there you go line by line direction by direction and if I scoot up a little further this is a little more interesting what we want to look at so here is the URL right and we can see we're using this restful api and there it is it's the route api right the MapQuest API and it's the route there is my key which is that MapQuest developer key I show you earlier and I'm gonna break down these lines we're gonna talk about other characters you may see in here this is just simply a separator there's my from Baltimore a little field separator and then to Miami and this is the get request this is the restful request that's being made to the MapQuest route API and I send in my key I sent in my from city and my to city and I get back my directions right I see my trip duration an hour 16 hours and 31 minutes I see how many miles it would be to go from Baltimore to Miami I can see how many gallons of fuel are going to be used and again these are all JSON objects that we are simply grabbing from the JSON data format response that we get back from the MapQuest API and again I'm gonna break these down and what you might say is gonna be excruciating detail but again we're gonna take a very very inductive approach here to make sure that you've got a really good understanding as to how all this is working and again this is just a for each loop that goes through and prints out turn-by-turn directions and that's what these are they're just simply turns and then you can see the program continues to run and I can even put in addresses let me put in my 36 20 or II organ drive my home address is a child in Santa Rosa California whoops and let's go back here let's just say 36 20 Oregon Drive and Santa Rosa California 9 5 405 and then let's go to San Francisco and I'll just say San Francisco California and you can see here it keeps everything on one screen trip duration an hour 56 miles to gallons of fuel and that's pretty good pretty good average there on fuel and you can see you go Oregon Drive to Carly Road out to Sonoma Avenue to farmers Lane twelve out to 101 and again you get the idea and the program is just gonna run and run and run and what's doing this again is a restful request to the MapQuest API and we're getting the data back in JSON format and I'm doing all of this with a Python script so this is the end game here so now let's take a step back and talk about the requests module that we're going to need so the first thing we're gonna talk about is the Python package installer pip right so if I want to see the version of pip that I'm using I can simply type in pip - - version and you can see I'm using version 18.1 now we're altima talega be upgrading this because that is actually a little bit old now what if I were to say pip list well this is gonna show me a list of the packages that I have here whoops and as you can see it shows me the package and the version and we can clearly see that I don't have the requests package here it's not listed here at all in fact if I was to go into Python 3 and I was to say import requests you'll notice that we get this no module found right because we haven't imported or added it to our package library and so that's why we receive that error let's say control D here and you'll notice and let's go ahead and do this right now because chances are you're going to see this if you're running these commands on a Mac and you can see here it says we're running version 18.1 however version 20.0 dot 2 is available you should consider upgrading so let's make that happen let me go ahead and say pip install - - upgrade pip and what will happen is we'll need internet connectivity here it's gonna go out and it's interesting what you're seeing happen here is it's actually loading pip into memory and running it in memory because it's deleting the existing package you'll notice you can see here it's uninstalling that existing package and successfully installing pip 20 0.2 so now if I was to come back and say pip - - version we should see that we're running 2002 now this is the Python install again that is the basic iMac or Mac OS X installation right I've got anaconda on here and I'm gonna move to that soon but chances are you're running if you're running on a Mac this is what you're going to see now what other things can we do well I can go ahead let me clear the screen here I can say pip I'm sorry pip and let's go ahead and install requests so I would say pip install requests now you can see here we've got some requirements right and you may not see this it may ask you do you want to install these dependencies and you're gonna obviously say yes but now requests should be installed so if I was to say pip list now we have requests and that's how simple it is and this is one of the great things about the Python programming language is it's that simple to add in additional packages modules right that you're going to be using as part of your work alright so if we wanted to see in fact we can even say and let me go ahead and pip and I believe it's Pip show yeah pip show requests and this is gonna show me information specific to the requests package that I just installed in fact we can see Kenneth writes is the author you can contact him here and it's licensed with the Apache 2.0 now you can see that we're doing this under the Python 3 library right for all of the packages that were installing so now we have requests so if I said Python 3 and I said import requests we don't see that error in fact will say Durer and it'll show me what's loaded into my current namespace and lo and behold there is requests so this is going to be your first step the first thing you're going to want to do is probably upgrade pip and then you're gonna go ahead and install requests now now that we've got that installed the next thing that we're going to do is we're going to go ahead and there's another module that we're going to be using but that module should be included in your base installation right and that's called URL Lib dot parse all right well let's go ahead and let's create our script right now I'm going to do a ctrl D here and I'm sorry let's pull that back all right I'm just gonna call this VI directions dot P Y is gonna be the name of my program and I'm doing this in VI you could do this in another editor that you're more comfortable with it doesn't have to be VI and let me give you some recommendations you could be using idle the integrated development learning environment you could use adobe brackets right and here's some notes I had from coaching course I'm taking right so you could use adobe brackets and then come on to the CLI and run it again it depends on your level of comfort and familiarity with the editor you're using but again there's multiple things you can do idle might be the easiest way to get this done alright so the first thing we're going to do is I'm gonna go ahead and say import URL Lib whoops Lib dot parse now we need this and you're going to see what this gets used for we're going to give this method our this module and there's a specific method we're going to be using called URL in code we're going to pass it a dictionary a Python dictionary with some values it's gonna take those values it's gonna mash them together and it's going to create that URL that we saw earlier right it's gonna create this URL and actually I'm sorry let's go and come right back over here it's gonna create this URL for us right and I'm gonna show you how it breaks it down we're gonna take a look at that module and we're gonna see how we can all of this information you can see there's quite a bit of information here right and that is what URL Lib dot parse dot URL encode is going to be doing for us it's going to put together our restful api get request to the MapQuest API for a route all right let's go ahead and take a step back here over to this window so we're gonna import URL Lib parse and then we're also going to import requests write that module that we just used pip to install now we've got these two modules so what's next well now we need to create some variables for that API request for the restful request that we're going to be making so let me put a little comment in here create restful api call to MapQuest and I'll come back up here and we'll say import needed modules all right so let me come down here and we'll try to keep it as clean as we can here so the first thing that we need to do is we need to create a variable and I'm sorry I was moving back here but I want to just come over that screen there so we need the base URL which is going to be a we're gonna create a variable for the base main API URL here the HTTP and in fact I'm gonna copy that right now let's come back over to this window and I'm going to say base API URL and it's a string right so here and again this is one of the great things I really like about taking Python and creating an application something useful because it gets you out of that mode where you're simply thinking about these abstract Python data types like yeah it's a string okay I know it's a string well what am i using strings for well here is an example of what you're going to be using a string for so we've just created a variable called base API URL of the underscores are in there and that is the value and we're gonna be concatenated that with some other values so the next value is I'm going to put down my start underscore location and I'm gonna hard-coded in here right now we're again we're gonna crawl before we walk before we run so I'm gonna hard code things in here so that we can get something working and then we'll look at adding some spice to it right spicing things up a little bit and we'll get it to clear the screen and some other things like that so my starting location I'm just gonna put in here Baltimore right and what about my destination location well my destination location let's try to pick something a little bit closer I'll say and well actually Richmond spots put Miami and we'll stick with Miami right because those are two locations that aren't going to be confused with any other like there's a California Maryland and so I don't want to put California in there because maybe it ends up picking California Maryland which might not be what I want and then the final variable is we're gonna create something called API key and what do you think that value is you've got it it's the MapQuest API key right here and it only runs up to that little D again I could go back out to the MapQuest screen and take a look at my key if I needed to but that is the value and there's the key and so right now we've got the base API URL for MapQuest we've got our starting location our destination location and we have the API key defined so what we're going to do now is we're going to use that URL Lib dot parse module and specifically a method that's in there called you l encode so how are we going to do that because again remember the endgame here is and again this was the San Francisco example but the endgame is I need this URL I need to create this so that I can request via a get operation from the MapQuest API and request it in JSON format the response to come to me in JSON format I need that right there well how do I get all of that because you can see sorry about that it's shifting on me here sorry about that so you can see that we need to get all of this and we do have a lot of it right now we've got the base API right there we have the key right there we have the from we have the two now we just need something that's going to put it all together and create that restful api call so let me go ahead and we're gonna simply say URL right so this is and again URI URL web address I'm just gonna use URL so what am I going to be concatenating together well the first thing is gonna be the base underscore API underscore URL and I'm gonna be concatenated that to the rest of this information here now remember there were special characters in that HTTP string here right that's part of that restful call now remember the question mark we've accounted for but what about key equal right what about the key what about that ampersand that separates things here what about this % to C and the plus signs where are those coming from and we're going to talk about each of those in turn here so we come back over the base URL I want to concatenate that and here's where we're gonna put in this URL Lib dot parse dot URL in that's the method we're using there is that URL encode method so what am i encoding well I'm going to be sending in a dictionary write a Python dictionary I'm gonna say key as and again a dictionary is nothing more than key value pairs right and that's what we're going to give to this URL encode method which is part of the URL Lib parse module right I'm gonna pass this data in and it is auto magically going to create for me the from and the to and the key format that I need and it's going to concatenate it to that base API URL and then we're gonna end up with this restful call the this information this URL that we're passing in as a restful call to the MapQuest API so what is my key value going to be here well it's simply going to be the key right or I should say sorry the API key the API underscore key now let's go ahead and get the from and the - so my and actually let's say start location so start location and actually know I can use from there sorry it's gonna be from so my from is going to be not the it's gonna be the start location and sorry I'm doing this from memory here it's gonna be the start location and then the - right because that's what we see in that URL that's being passed we see from and we see - and we see key so the from our I'm sorry the - is simply going to be my destination underscore location and you can see we just wrapped around the screen there but that's okay and there you go right there and in fact let me do this let's let's trim down destination location here I'm gonna try to keep everything on one line and let's go to the end of that line and let's pull back and I'll just use destination that way we can keep everything on a single line and so we now have everything we need here in order to make that restful call to the MapQuest API we've got the base API URL we're concatenating that to what will be pulled together from this URL encode Python method and it will give me write or I should say it's going to give the MapQuest API this information right here again our information I'm pointing to this and this is the San Francisco to Santa Rosa but we know we're talking about Baltimore to Miami right now so there's the URL now the question is okay that's great you've got the URL we've used this URL encode to create that base API URL with all of the other information that's going to be needed to send a restful request to the MapQuest API so how do I make that request well we're going to use the requests module that we imported earlier and this is exactly why we imported this module so here's what I'm going to say JSON underscore response because I'm going to request right that the information come back so I'm going to say my JSON response and that's a variable name so I'm going to be storing whatever's returned to me and it's going to be all of that information we saw earlier in JSON format and I'm gonna store it in that JSON underscore response variable I'm using an HTTP GET right remember the three criteria that we need to meet that restful standard one of the criteria is a base URI and a Staunton at support for standard HTTP methods so yes the MapQuest API is restful because it supports this method this get this HTTP GET method okay so what am I going to send in well we saved off our URL that we created with all of that information into the variable named URL and so here I'm basically saying I'm sending an HTTP request it is a get request here is the URL which we created here that I'm passing as the argument right to this get request and how do I want the information the response to come back what is the common data format that I would like to get back and I want it in JSON format because that is what we're talking about here and finally and again this is a little crude right now we'll clean things up a little bit later I'm gonna print out the JSON response and this is gonna show me the JSON data and this is actually our stopping point here after we run this this is gonna be where we break as for part one so let's go ahead let's save this file let's clear the screen and let's come in here and let's say Python 3 directions dot py now we may get a I'm not sure my permission settings the defaults here but let's see what happens and it looks like we're going to be good and there is all that JSON data now you'll notice that that is not very pretty right and it's not nice to look at and this is one of the intimidating things about parsing JSON data is when you see it like this it is not very attractive so again we've hard-coded that information into our script let's go ahead and do this let me go ahead and VI that file again and it's easy enough to come in here and change us let me change this from Baltimore to Santa Rosa my hometown and I'll put in here califor and then let's come down here and change this to San Francisco now I'm showing you this because I showed you earlier we have it set up to prompt us in to run again and again and again here's what we're doing now right before we start to run this is us walking and here is us clearing the screen and then re running this and you'll notice we don't get as much information but that is JSON format right there that's exactly what we're looking at now you might be asking yourself well how would I see the URL well to see that we just have to print out that URL and let's do that down here at the very very bottom let's add in a print and I'm gonna print our URL and now we're gonna look at the JSON view plug-in that is part of Google Chrome that I just added to Google Chrome so let's run that again and the last thing that we should see is our MapQuest URL and there it is right there so let me copy that let's go over here to Google Chrome let's open up a new window and let's add in that URL and I'm gonna hit enter now the plug-in that we have is what's facilitating us seen the information like this and there's all the turns right and you can see there is the route object that we're looking for in fact I could close these up and there are the two objects right they look like Python dictionaries and here are some of the other objects that are in here right we've got our longitude latitude come down to some more the real time right how long is it going to take you the amount of fuel used has country cross what are some other things session ID here's my session ID there is the time right and that's that value we saw earlier so here's where all of this information is being returned to you and here's the different lait ie the different turns that you're going to be making right go south on US 101 and so you can see that's what the JSON format looks like and again it looks like Python data structures it looks like dictionaries which in JSON are objects and it looks like Python lists which in JSON excuse me I'm getting over a pretty bad sore throat which in JSON the list of I should say what appears to be a Python list in JSON is actually a JSON array that'll have data in the array and so that's what that JSON view plug-in allows you to do in Google Chrome so we've talked about quite a bit in this video we have a working Python application that is making restful calls to the MapQuest restful api right so we've got all of this set up right now let's take a look one last time here and let's simply cat out that file directions dot py and that's it that's all that's required right and so this is a great stopping point right here you've got something up and running you can experiment with this change the start location change the destination and see what happens and when we come back in subsequent videos we're gonna be taking a look at how to clean that up and how to get it so that it runs uh it runs until we quit it's going to run and it gives us the directions in a nice laid out format and we pull out some of those objects those JSON objects like the trip time like the amount of fuel that's going to be used like the number of miles that are going to be traveled all right well that is going to do it for this video again I'm really excited about this these are there's nothing better than taking things that you've been learning about right like I teach the Python programming language all the time but it's not until we get an have an application an actual use case of all of that Python programming information that we've learned that it starts to become exciting and you can really start to see some of the really incredible things that you can do by using Python alright thanks so much for watching and I hope to see you in the next video

Channel: Travis Bonfigli

Views: 2,687

Rating: 5 out of 5

Keywords: AACC, CCNA, CCNP, CCIE, Cisco, Networking

Id: B8ERfAbxiUM

Length: 42min 52sec (2572 seconds)

Published: Wed Feb 05 2020