Bash Script: Using jq (command-line JSON parser) to parse and use data from a JSON file

in this video i'd like to look at a bash script that uses jq to read and parse a json file so let me show jq as it says is a lightweight and flexible command line json processor so i've installed that so i can use jq to read and understand json and then here's the json i'm going to be pulling down i have it on my website it the json starts with an object it has a property painters which is an array square bracket means array so this is a property then a colon and the value of the property starting with a square bracket is an array and then it's an array of objects uh it's the painters yeah they have properties like last name first name country icon country name birthday death date and then they have another property not going to be dealt with in this script but but a later one i will you do something with this but there's another property of painters which is paintings which is then another array an array within an array but not here maybe later okay so that's what we will be we'll we'll get this file then we will uh be reading this file in and parsing it with jq okay so here is i broke the getting the i broke the file down into two parts this is more set up for later but i had a base url which was the location up to the file and then the name of the file and then i put them together concatenate them together and use w get and i've suppressed the standard output and the error output so i won't see any result of w get other than if i look into the file system i will have the file and i know that there was uh and then i can when when it's brought down by w get the name of the file will be uh dollar sign file i could use if i had used full url then i could use base name to get the name of the file but since i was separated the base from the file from the beginning than i already had um this is a line of code that gets me how many painters there are so again painters was a property of the object in the file and i'm finding using this uh vertical bar length i'm piping it in and using this length i'm finding out how many there are and then um i do it again with this minus r there's a raw option a lot of the things jq gives me like without down here without the r when i'm getting the last name it will put quotes around it and if i don't want the quotes then then the minus r okay so i'm getting the number of painters and then i'm setting up a sort of c style for loop starting at zero going up two but not including the number going up by ones and then i'm doing cat to sort of spell out the contents of the file and pipe it into the jq the the json uh parser and i want the painters property remember that's an array with a particular index i dollar sign i so it'll probably be the zero one and then the one and the two and then i'm pulling out the last name property and this minus r this raw option will uh forego the the quotes around it um so i'm getting in this version the last name and then the first name and i was echoing them out but then i turned that into a comment then i'm creating a full name but i'm doing the last name first so it's not the standard full name um but i want it by the artist's last name they're more well known by their last name so i want the last name first um [Music] then in a later version not in this version or in version two but in version three if you will i want to eventually turn these uh artist names into the into folders and names a folder then i hate spaces and the names of my folders so i wanted to get rid of my spaces and so this here i'm doing sort of a sub bash substitution the slash slash means global so i'm going to uh replace all the spaces one of the artists see if i can find that is james abbott mcneil whistler so he's got many names and spaces in there so when i create his name there'll be many many spaces and i want them all to be replaced by underscores so i need the global so this is slash slash is a global replace and replace it with an underscore and so then i'll have the full name uh without any with the spaces replaced by underscores and then i will use the tr trace um to replace a capital letters with small letters so i will get the names out all in small letters so let me i mean the the name of the file was painting data painting painting data.json and this is just if i had it i would get rid of it i didn't have it but okay now i am running this script art.sh and five was that there were five artists and then here is uh edward monet monet underscore edward and then in small letters so just my five artists edward bernay george sora rousseau gustav klimt and james abbott mcneil whistler okay so that was version one and what i didn't like here was this uh sort of catting the whole file and then cutting it for the last name and then cutting it all again for the first name and so i wanted to try and combine this so that's what i did over here in version two so same start url base and file getting the file with the suppression um finding out how many there are setting up a loop i didn't need to do this twice but it was just a vestige left over from my debugging and working through it here's my loop uh over again the old-fashioned c style loop over the index up to the uh up to num not including and then here is me getting the full name [Music] uh and once so cat the file spell out all the contents of the file send it to jq do the raw thing getting the ice painter and this time getting the last name and the first name property um and then i'm echoing it out so we see what we have and then i am doing this replacement of space with an underscore and printing it out and then what we'll see is that didn't have the uh full effect that i wanted um so we know that this is we knew in james added neil whistler that this took care of multiple spaces but it won't eliminate all of uh the spaces and the names and so it must not be a simple space i never did figure out exactly what it was but we'll see that then there are still spaces in the name after this and so we'll have some underscores and james have a neil but we won't have it in in this new whatever this was in giving us the last name in the first name was not just a simple space so this didn't work so then it came along this time and used some regular expression here and use this sort of set and in when you start dealing with square brackets and you're dealing with the set in regular expressions the carrot within within the square brackets means not and so i'm replacing anything that's not a capital letter a small letter or a number with an underscore and then i finally have what i want in terms of getting rid of the spaces and then i put it all into lowercase so let me delete that painting file because i have it from the previous running of the script now i'm going to run bash art 2 and all i want to point out here is i just got the name from from the jq with the two things and echoed it out so that is here edward monet and then i tried to do the replacement of a space with an underscore and it did not succeed for edward monet it was doing we see here under james abbott mcneil that it was effective within within this long first name with spaces it worked but with with whatever this is between the last name and the first name coming from this jq that's getting us two properties um [Music] this this wasn't working and i needed this now i could have then eliminate 13 i think 15 will do the job of 13. it's it's a more robust thing so i could eliminate 13 but i wanted to show you that this wasn't working okay so then i get rid of some spaces for underscores now this time i got rid of all spaces for underscores and lowercase and so that's what we see here okay that was what i wanted to show you sort of using bash to read and parse a json file thank you for your attention

Channel: Thomas Blum

Views: 1,287

Keywords: bash, script, cat, jq, raw option, array, object, property, name-value pair, regex, regular expression, replace, global, tr command, loop

Id: 7d_EZrOJMM4

Length: 11min 24sec (684 seconds)

Published: Thu Mar 03 2022