JSON data parsing in Python [Python Programming Basics to Advanced]

in this lesson we will study about the json files and passing those in python json is not a data type but it is a data format it stands for javascript object notation don't get confused with the javascript because it is not something supported in just javascript language but almost every programming language has some library to read and pass the json file and at present it is probably the most widely used data format what makes json one of the most widely used data format it is a well structured and easy readable data format and it is mostly used for web applications for data transfer and also for the data transfer between other software applications it is being used to exchange information between web clients and web servers today it is the format of choice for most of the publicly available web services the information between client and server is transmitted through an api an api is a set of programming codes that creates an interface which enables data transmission between one software application and another a typical scenario of web application is shown here when we use a web application basically we send a request to the web server over the internet and this is sent to the web server through the api interface of the web server for example when you click on add to cart on some online shopping web application it is the api which will tell the web server to add that product into the customer card and web server will update that in its database similarly when we are requesting different web pages the web server will send the data in the form of response to the client again through the api and the data from most of the web servers is transmitted as json files there is also a format known as xml but json is better readable and lightweight and hence a widely used format finally not just the web applications json data format is also used in other software applications for example in this visual studio code we set different settings like the color theme the font size and any other settings we specify and that is available in a json file and visual studio code configures different settings by reading that json file you can go to this setting icon and click settings search for setting.json and click this setting.json it will open the setting json file and you can see different settings of the video studio code for example editor font size is 22 workbench color theme is view studio dark and many more so what we are going to cover in this lesson we will first see how the json format looks like we had a look at setting.json file but we will explore the format in detail then we will see how we can read and pass a json file in our python program we will see the example of periodic table and we'll see how json format is much better format for such cases we will also see if we have data in python in the form of a dictionary how we can convert that to a json file it means we will see both conversions from json file to python data and from python data to json file then finally we will see how we can get the json data from some web server through its api directly into our program without downloading that json file into our system our python program will directly communicate with the web server and will bring in the json data directly into the program you can see the timestamps of different things we will be doing in the description so let's start the lesson here inside the working folder i have a simple json file with name example underscore1.json a json file basically contains a json object and that is indicated as enclosed between the curly braces then the data inside the json object is identified as key value pairs for example the key is string name and value is faro vocabulary then is the second key value pair i hope you can very easily see that this is exactly how a python dictionary looks like that is also specified by curly braces and has key value pairs so json format is pretty much same as python dictionary there are a couple of differences but we will see that handling those differences is also very simple now let's write a program to read this quite simple json file and bring in the data into our program for that we will import the module json and it will do most of the job for us in the json module we have a method load and as input argument we should open the json file using the open function the json file i want to load is inside the working directory so i just need to pass the name of the file otherwise i will have to pass the complete path to the file let's assign the output to some variable for example data this load method will load the json file and will read that and convert the data into python dictionary let's print the data and also its data type see that it is a dictionary object and the value is the content of the json file with the same key value pairs so it's pretty much simple process and data in json file is converted to python dictionary and then we can process that in any dictionary supported operation as we process dictionary previously this way of opening a file to read the content is not a good practice and we should use context manager to open the file as we did in case of text file and csv file you can find the link to those lessons in the description for further detail the context manager is initiated using the with keyword then we specify the file name and then the mode in which we want to open the file to read the file the mode is read mode and we specify that with the letter r if we don't provide the mode its default value is also read then we can bind the open file to some variable for example f so file will be opened and we can access that through the variable f and the context manager will close the file automatically when the interpreter will come out of the context manager so inside the context manager we will load the file into the python program see that we also have a method loads other than load we will see its use in a while we have the same result now let's see what kind of different data types we can have inside a json file you can see here both key and value are of string data type and these braces specify the json object let's see another file that has more data see that the whole data is inside the curly braces as the previous json file which represents the json object then it has a key with the name persons in most of the cases we see the json file having just one key and that is also known as the root key and the value can be some nested data like nested list or even nested json object over here the value of the key persons is a list i am saying it is a list but in fact in javascript this is known as array and not the list we will see these differences in a while but just see that the value of the person's key is an array or list and this is the first element in that array what is this element this element is another json object so it is a nested json object inside the main or the outer json object this inner json object has some keys and values some value is again another inner json object there is a key with the value as inner array and that inner array has inner json objects and like that basically this inner structure of the json object is one important reason for its popularity and better management of data note that this value is null in python we have none in javascript it is null then also note that this boolean value is false but with lowercase letter f in python we have uppercase f for false but don't worry these are the things that will be managed by the json module or library we have in python let's see these differences indicated in the python documentation in json format the json object is translated to python dictionary and we saw that a while ago then an array in the json format is translated to python list the string in json is translated to string in python in json or javascript a string is specified only by double quotation marks and cannot be specified with single quotation marks as in python then the numbers are translated to into or float in python the boolean true and false in json starts with lowercase letters while they start with uppercase letter in python and finally the null in json gets translated to none in python so let's read this json file now see that the json data is converted to python dictionary which we can process further as we process any other dictionary here in dictionary we have the key persons and the value is a list with two persons this is the first person detail which is in the form of dictionary or you can say inner dictionary and then is the detail of second person as second element inside the list as another inner dictionary this is the beauty of nested information that we can access the inner information easily for example data here is a dictionary having just one key persons so this will return the value which is a list of two persons you can see the length of the list is two let's just print the detail of the first person which is at index 0 and see that it is a dictionary so we have inner dictionary inside the dictionary data it will be better if we print the key value pairs of the percent detail in a new line to see that clearly we can apply the dictionary method items to get the key value pairs as tuples these are all key value pairs of the inner dictionary of one person the address key further has another dictionary as its value phone number key has a list as its value and that list has further dictionary objects inside it see this value against the key salary which is none if we see the json file it was null there and the json library of the python has converted there to none similarly this is false in json format with low ref and it is converted to false in python with upper case f this json module will do these conversions for us so in short json format is quite close to dictionary format and the few differences which they have are handled by the json module we have discussed the differences and the documentation link is given in the description a couple of more differences which is not mentioned in the document should also be noted the first is that in json the string are specified with double quotation marks while in python it can be either the double quotation or the single quotation marks then in json only the strings can be the keys the values can be strings or numbers or arrays or null or boolean value or some other json object but keys must be strings while in python dictionary we can have any immutable data type as keys that can be string number or none or boolean value or even a tuple we will see in a while if we will convert a python dictionary to json format how those keys will be handled anyways at present we have converted a json data into a python dictionary and there is so many nested information in there for example in one person json object we have a key phone number and the value is an array having two json objects so i can access this key of the first person and see that the value is a list with the two dictionaries note carefully how i am using the term list when it is the python dictionary and i use the term array when it is the json data now to get the second phone number information i will specify the index 1 of the list and this is the second phone number inside the phone number list and it is a dictionary from this dictionary i can access the key number to get the number value so you can see there is five levels of data in the json file and the corresponding python dictionary now see another json file here it is having simply two key value pairs but there is one non-ascii character in here let's see what happens if we load this json file see that the non-rc character has been changed here this might vary in different ides depending upon the settings how they handle the non-ascii characters but there is a way we can specify the character encoding and that will handle the non-ascii characters so while opening the json file we can specify the optional input argument encoding and set that to utf-8 and now see that we get the correct output with that non-ascii character i forgot to mention one thing that instead of json file if we have text file but the text inside it is in the json format still we can load this just like a json file so here i will specify the text file and we get the same result it's not just about the text file but even if we have a string in the json format still we can load that in case you are thinking that why we will define a string as json format and then load that to get a dictionary the reason is that when we send a request to some web application the web server sends the response and we can get that in the text format and the json data will be in the text form or string so we would need to convert that to a dictionary to work on that so here we have a string a and the string is in the json format you can verify that a actually is a string now to load the string as json object we have the method loads where the last s is for the string let's print the data and the data type you can see it is a dictionary with key value pairs as described in the string so we have seen the ways we can load the json file or the json text these are loaded as python dictionary now we will see the opposite meaning that if we have a python dictionary how we can convert that watch.json file i will copy a dictionary from other file here we have 4 key value pairs the key courses has a value as a tuple but let's first convert that to a list then we have a boolean value and a none as another value to create a json file we will open a file name it something like student1.json and the mode we will specify as w4rite write mode means python will create a new file with the name student1.json and will write data on it or if the file already exists it will overwrite the old content to write on the json file we can use the jsonmodule method dump the first input argument will be the disney which we want to convert to json and second will be the file name let's run the code there is no output since there was no print statement but see that a new file student1.json has been created if we open this we will see the content of the dictionary written here all key value pairs are on one line we will see how we can better format this but first see that strings here are enclosed in double quotation marks although in the python program we had those in the single quotation mark as i said earlier in javascript we must have double quotation marks for the string and this conversion is done by the json module moreover see that this false has been changed to false with lowercase f and none has been changed to null however the non-ascii character is written with the unicode instead of the letter itself to correct that we will specify the encoding here as utf-8 and while using the dump method we need to set the optional input argument in short underscore ascii to false which is true by default now you can see the correct letter written on the file for better formatting of the file we can set the optional input argument indent to 2 4 or any value you like it will add that many indentation in each nested data now you can see it is well formatted json file now let's change this list to a tuple and see the effect there is no error and that tuple has been converted to array so going from python to json list and tuple both are converted to json arrays which is a good sign since we will not have to worry if our dictionary has some tuple as some value in json the keys can only be the strings but we know in python dictionary we can have keys of other data types so let's change this key to none although it doesn't make any sense but let's just see what happens there is no error and in the json file see that none has been converted to null but as a string because in json only string data type keys are allowed let's see if the key in dictionary is some number see that the number has been changed to a string in json key if we have some boolean value here that is also converted to json boolean format but as a string finally let's see if we have a tuple as key in dictionary and this time we get an error it is type error that keys must be string in float boolean or none and not the tuple it's not an error on dictionary the dictionary is valid but the json module cannot convert a tuple into a string key for the json file we have another optional input argument skip keys that we can set to true and it will skip the dictionary items having the keys that cannot be converted to string keys for the json see that there is no error and in the generated json file that key value pair has been skipped there is yet another optional input argument that can be helpful and it is sort underscore keys and if we set that to true it will write the json file in the sorted order of the keys see that in json file courses is at the top because the key courses come first alphabetically and so on the other case we also have a method dumps just as we have loads and it will not create a json file but will create a string in the json format again for some web application or some other software application we might need to send the json data in the string format and for that we can use this method we just need to pass the disney which we want to convert to a string it looks like the same dictionary is printed so let's print the data type and it is a string we see that non-ascii issue here as well and we can resolve that by setting the ensure underscore ascii to false we can also set the indent value this is the output string we have seen these four methods to translate the json file to python dictionary and vice versa now let's get some online json data and pass that information here we have a json data for the periodic table the json data has one root key elements and the value is an array the array contains one json object for each element inside the periodic table with a lot of details about the element the file is quite lengthy it is more than 5000 lines let's click this raw and we have just the file content i will copy the content and will create a json file in my directory after a couple of minutes we will see how we can grab in this json data directly into the program without downloading or copying that data like this but first let's just explore this data and apply some query on it by creating a new file name it like pt.json and i will paste the copy data here and now we can read this json file as we did earlier we should not print the data for over 5000 lines see that the root key of the json is elements and the value is an array of all elements detail so here the json data is converted to a dictionary i can specify the key elements and the value will be the list having the elements as inner dictionaries let's just see the length of the list and it is 119 meaning that there are 119 elements in the periodic table let's just see the detail of the first element you can see there is a lot of information in the form of a dictionary for just one element it will be better to print each key value pair on a new line as a tuple for better readability instead of the first element let's print the detail of 11th element which is sodium and i should specify the index as 10. these are different key value pairs of one element like name appearance and so many others the key source has value as wikipedia link there is also the summary key then symbol and xy position in the periodic table the key shells has the value as a list it means the sodium has three shells and these are the number of electrons in each of the three shells in shells we have subshells and that is specified here this is just a brief representation of the electronic configuration it indicates that the electronic configuration of sodium is same as of neon and then there is one additional part as subshell s in the third shell having one electron then is different ionization energies and a couple of more properties you can also view this in the json file let's do a couple of tasks on this data like generating the list of symbols of all solid elements we have the data as dictionary and it should not be a difficult task i will do it with the list comprehension i will iterate over the all elements inside the value list of the key elements we need to consider just the solid elements in element dictionary we have one key as phase and the value is gas solid or liquid so i will apply the filtering condition on the face as solid and we have the key name and also the key symbol over here we need the symbol of the elements so in the expression part i will specify that let's print it and this is the list of the symbols of all solid elements of the periodic table you can do the same to find all gas or liquid phase elements now the task is to get the list of the names of all elements which are gas and have the boiling points less than the room temperature again i will use the list comprehension filtering condition will be the phase of the element as guess and for the boiling point condition we have this key boil with the value as boiling point in kelvin so we should add the condition as boiling point is less than the room temperature which is 295 kelvin and we need the names of the elements let's print it and these are the names of the guesses having the boiling points less than 295 kelvin one final task i will do from this data is finding the symbol of elements which have incomplete third shell in third shell there can be maximum of 18 electrons and incomplete means it has electrons less than 18 but of course not zero we need a symbol of the elements and for the filtering condition as incomplete third shell we have a key as shells and the value is array or the list in python which represents the number of electrons in each shell one list element means there is just one shell and if we see some other element for example here we have five elements in the list meaning that there are five shells and the third value 18 is the number of electrons in the third shell and here we can apply the condition that from the shells key check the third element if that is less than 18. that's fine but it will generate error for the elements which are having less than three shells in total because in that case there is no index 2 in that list and it will generate the list index out of range error so we should check this condition if there are at least three shells for the element for that we can use this period information which is same as the number of shells in an element or maybe we can check the length of the shells list that it is greater than 2. and recall the short circuiting technique if the first condition is false for and operator it means the overall condition is false and interpreter will not go for checking the second or more condition and hence the index error will not be generated these are the elements having the incomplete third shell you can also use the filter and map function instead of list comprehension you can watch the details of filter and map from the other lesson now there will be one review question from this data which you will do and answer in comments the task is to find the list of symbols of all alkali metals having one valence electron the electron in the outermost shell are known as the valence electrons now is the time to see how we can request some web server api and get the json data directly into our program without copying or downloading that let me clean up the workspace a little bit let's first see some example json file available online and i just have one opened here in the google chrome it is as one single line first let me show you that if you go to chrome web store and go to extensions search for some json viewer extension let's try this first one and add to chrome it has been added this was the json file all in single line and after adding the extension if i will refresh the browser you can see the json data in well formatted way this was just an extra tip and as one more tip you can search different json viewer online tools there is one code beautify it offers many good code viewing services and one is json beautifier here you can load the json file from some url and it will show you the json data in different views you can validate if it is a correction format you can download the file and few other things you can explore by yourself all right that was not something we were intended to do we were actually trying to access some online json data from our program for that we will use the request module it is not a built-in module and you can see the error indication we need to first install this module for that we can go to the command prompt by just typing in cmd in the windows search bar and it will open the command prompt here we have to type the command pip install request instead of command prompt we can access the terminal from the vl studio code as well go to view and click terminal and you can type the command here it will download the package and install in your system if you see some error like pip is not recognized as internal or external command it means there is some pass setting issue and for that you should watch the very first video in this playlist in which all path settings were discussed we got a warning but it is just that a new pip version is available i am fine with this version and see that successfully installed requests i will press ctrl s and the error is gone now i have one example json file in my github repository see the content of this whole page we have many objects other than the json data and everything will be returned if i will send request by this url but if i click this raw and now i have just the json data and only that will be returned when i will send a request on this url so let's copy this url and save that in our program in some variable as string in the request module we have a method named as get as input argument we pass the url this method will send a request to the specified url and in return the url will send the data and we call that as response let's save the response in some variable let's print what we get as response here we see the output as response object and this value 200 is the status code of the response 200 status code means ok meaning that response has been received correctly there are many status codes we can get in response but at the moment we can't discuss those but i hope you must have encountered 404 not found response while browsing and 404 is the status code anyways 200 means the status is ok meaning everything is fine from the response object we can retrieve many informations like it has one attribute as status underscore code and it returns the status code what can be the use of this status score well as good programmer you should take care of different possibilities we will get in response and should handle those intelligently the best can be handling those with the try accept clauses there can be very lengthy discussion on the request module and the response results but that's not the topic of this lesson here we will assume that url is working fine and we will get the ok status now the important thing is how to get the json data from the response object for that we have a method json in the request module that we can apply on the response object and it will get the json object from the response and will convert that to a dictionary let's assign it to some variable like data and print that see this is the json object converted to python dictionary and we got this dictionary directly from the url you see we have not used the json module here and have used the json method which is available in the request module but there is another way to get the json data from the resmos object the response object has one attribute as text which converts the response object to a text and in case there is just the json data in the url the text will be the json object in the text or the string form it means we can apply the loads method of the json module on that string to get the dictionary just a quick recap of getting the json data from a web server api we need to have the valid url for the json data then we apply the get method of the request module it will send http request to the url the api of the web server will send the response from the response we will get the string using the text attribute and that we can use with the loads method of the json module to get the python dictionary now here is the periodic table json file that we used earlier this time i will simply copy the url and will paste that here in this variable and i will get the periodic table json data into the program i can delete the periodic table json file from the working directory i have two such files and both are deleted now i will copy one task we did on periodic table json file let's run it and we get the output i hope you can see the advantage here that we don't need to have the lengthy json file and the program will load that directly from the web server secondly if there is any change in future in the json data for example a new element is discovered and added into the json data we need not to worry because our program will load the updated data from the url now as the last part let's see that instead of github if we have the json data on some other api how we can get there actually nothing special is needed for that i have one repository link of the github indicating different public api from where we can get the data and use that in our program or our own website or some web application anywhere we like there are many categories and there are many links against each category let's go to this books category for different api links provided here you can see this authentication column many of the api don't even need some authentication some need that where you have to create an account and they provide some api key to use there google cloud api is one such example let's see this quran api and this does not need any authentication you can see link to different data here let's see the chapters here we have the option of try it out and if i click this send request you can see the response this is detail of different chapters in quran see this response is 200 okay anyways all we need is to copy this url and use this in our program just keep in mind that the json object has the root key as chapters and the value is an array having one json object for each chapter in quran so we can access that using the chapters key and in python the value will be a list let's print just one element of the list say the second element at index one and we get one dictionary with different key value pairs let's view the key value pairs on separate lines we have id which is a chapter number revelation plays with the value medina revelation order is 87. bismillah pre is true meaning we need to recite bismillah before this chapter al-baqarah is the name this is arabic name but is not printed very well actually this is a problem with visual studio code and not with the json data wheel studio code has issues printing the words of right to left language like the arabic or urdu but if you will run this on some other ide or on python built in ideally you will see this correctly then in some other detail as well there can be many queries you can perform on this data say finding the chapter with the maximum number of verses see that we have versus underscore count as one key so we can apply the max function on the list of chapters and should provide the parameter key to decide the maximum element it will be a lambda function that for chapter c the output is versus underscore count of that chapter the output will be one chapter as dictionary so let's just print the name of the chapter and it is al-baqara let's also see the one with the minimum number of verses it's a lesser if you are thinking that it should be alkosar the reason is that both al-assad and al kosar have three verses each and allah comes first in the list enhances the output you can do many other queries as well like getting the list of maki and madhani chapters in chapter dictionary we have the key as revelation place and we can use that for filtering now will be two more review questions from this data the first is to order the name of the chapters based on the revelation order of course you need to generate the list of the names of the chapters in the revelation order then in the next task you are supposed to find the total number of verses in whole quran and you should do this using the reduce function so that's all from this lesson thanks for watching

Channel: Learning Orbis

Views: 447

Rating: 5 out of 5

Keywords: python json, json in python, python json tutorial, python json dictionary, python json parsing, parse json python, python json api, json python 3, python 3 json, python api, json and python, python json parser, json python tutorial, use json in python, python json module, python create json, python json to dict, python json example, python requests json, convert json to python, install json in python, json response in python, json api in python, periodic table in python

Id: tSjKEThlVfg

Length: 37min 51sec (2271 seconds)

Published: Thu Jun 24 2021