The Ultimate JQ Tutorial: Everything You Need to Know to Parse JSON Like a Pro

hey in this video I am going to show you how JQ Works you've maybe used it before it's just a command line program uh for dealing with Json files and usually people just use it to pretty print a Json document but it can do so much more and if you kind of learn the basics of it then you'll be set like time spent learning the basics of JQ will pay off very quickly okay here we are right the first thing to know about JQ this is just my vs code by the way is what everybody knows right if I if I have this um you know it's just a simple little Json document and I pass it to JQ with the dot like that it formats it it does a little pretty printing same thing happens if you don't even add the dots but this is just the beginning right the way that I tend to use JQ is right I have some sort of rest API that returns some data and I need to play around with it figure out what's going on right and if I pass it the dot it formats a nicely big deal this is by the way the the JQ endpoints in GitHub so yeah if I pass it the dot it formats it which is nice but that's just the beginning of the object indexing system so you can do Dot and then a field right so right here is my document I can just pick any field in it and go into it so that's topics right and now we're just returning that part of the document now some other examples here right here's name so now we get the name is JQ here's owner right and so you can see the owner has a whole bunch of information and we can further move down from there right here's owner.login right so this is the simplest thing to understand the object identifier index I guess they call it basically JQ lets you treat a Json document like it's a JavaScript object which I mean it sort of is and you can just you know dot into it like you can do like dot key and then dot sub key and then dot Sub sub key right you just it's just an object and you're just exploring the properties going into it easy enough what uh you may say happens then if you have something that's an array right so this is JQ and we're looking at issues this is the JQ GitHub project and we're looking at you know the first page of five issues and you can see here it returns an array right and so this works the same way right if I want to get the fourth item in that right I just dot into that array and then get the fourth item and you can do a lot besides that right like here I have the array one two three or five I can get from the second to the fourth element counting from zero to get three four right I can also just do two to the end you can also do I think negative indexes to start from the other side so that's negative starting from that side you can do negative this way to start at the beginning bam that's simple right it's just an array right you're just walking into that array just like it's something in JavaScript and then you can combine this with your object Index right so we do this and right now right we're going into the fourth element and we're getting the title very simple yeah what if we want to go across like we want to get all the titles right then we don't apply a number we just say into the array and this is sort of like uh I like to think of like functional programming terms it's sort of like a map like you're saying go into this array get all of these properties and return those back right so here we go these are all the titles another thing to note is there is this Dash R right so right now when I'm returning all these values they all have quotes wrapped around them which makes sense because they're strings but what if my process is different right what if I'm getting a bunch of IDs and then I want to pass the IDS you know onto some further curl command I need to unwrap them and that's where the dash R comes in handy right without the dash R it's returning strings and I add this and then it unwraps them okay so that's a race right arrays are simple in JQ you just use these brackets to get inside of an array and you can provide certain offsets or ranges of offsets and you can combine it with the object indexing right so this is we're going into a key which contains an array which has a sub key which has another array where we're getting the second element very simple it does exactly what you would expect so the next thing to learn after that so we have object indexing array indexing is array Constructors right so if I pass an empty string 2jq and then I tell JQ to return one comma two you can see it returns one and two on a separate line the kind of an interesting program to think about in a way JQ is taking none of the input and just returning two elements which it returns I'm by line but that's not actually valid Json right like to return a object like that we actually need two wrap it in an array right and so this is an array Constructor now this is valid Json so we're just wrapping it in Array that may not seem useful but it is useful because earlier we grabbed the title right we grabbed all these titles but this is not a Json document this is just a list of strings so if we take that same query and we wrap it in the array Constructor oops and we execute that right now well we have makes a lot more sense now we have an array so that's an array Constructor that's pretty simple you just wrap things in an array just like you would expect here's uh some interesting examples right so here I have made an array and in that array I have an object right two objects they both have a property named a and they have different values for it right so I go into that with my DOT right and then I'm going to go into the array which is this part and then I'm going to get the a right and so I get that back B and C but that's not a valid Json document and usually with JQ I'm just like transforming or extracting some Json into some other Json right so the way to fix that obviously is you just wrap it in an array usually you put a space there because it just looks nicer like that right now we're talking then we can also use the dot R if we want which actually does the same thing in this case yeah I'm not sure why I added that example there to be honest so once we have array Constructors the next thing to know is object Constructors so using the same repo right we're going to the JQ issues API and we're getting back issues and we're going to do this so I want to I want to go to the issues right and I want to get back just the title and the number of the issues right so there's an array of items and for each one I want to get the title in the array so it's easy right I go into the return results then I go into the array then I get the title hinge the results into the array get the number but when I when I look at what's returned it's not it's not really what I want it's just a list yeah so we can do this instead right so now we're wrapping in an array still not what we want right we need to actually wrap them in objects what we want is an array with objects in it and then we want each to have a title and a number it's easy to see with maybe a smaller example right so this is my name Gordon Bell if I select the zero with element in the second element I can get my name without the middle name right but if I want to make a new object I just do this right it's basically the Json document structure so I'm gonna start with the Paran I'm gonna say first name then I have my JQ selector then you do last name and then I have the selector for the last name right so I'm just I'm making a new object using these selectors like this right and so now I get first name Adam last name Bell so it follows from this that we use the same thing to produce our example right so we want to get back an array of results we want to put them in an object and the object is going to have a field called title and then we use the same title selector where we go into the object into the array get the title and same for number go into the object and see where I get the number right so we're taking this document apart putting back the document that's much smaller that has just the title and the ID actually this looks like it is incorrect fixed manual selection title interesting we're going to get to how to fix this in a minute another way to do this is you can see in this example it's basically we have our selector here right and then inside of it we go into the array into the title and then here we go into the arrange title twice right so one way we can simplify that is to extract it so we're going to we have our array Constructor and then we're going to go into the array and we're going to use a pipe which I'm going to explain later and then from there we'll get the title and the number let's try that there we go now we're getting here's our two separate titles and our two separate issues we're gonna I'm gonna explain how the pipe Works later on the important thing to learn from this section though is that JQ object Constructors just work like this right you you start a paren you give a key you have a JQ filter and then you give another key and you have a JQ filter so that's how you can take a big document with lots of fields and pull out specific elements and put them together into something that's the shape that you want it which is like one of the more common things that I do with JQ is just like reshaping things and so here's a very simple example that we had before right we have title and then we do dot title we have number and then we do DOT number so if I run this I mean it does exactly what you think it's going to do right and the great thing about that is I can change the labels right I can say instead of title I can say what's my name again instead of number I can say thing and that works now we get new labels but most of the time you just want to use the same label that it has right in that case there's a shorthand available you can just give it the name if you're using the name that exists if you like the selector name is equal to the label name you can just omit it and do things like this and that is how this example up here worked this is just a shortcut you don't actually have to remember it but if you're doing this title colon dot title you can just simplify that okay next up sorting and Counting so if I run this which is the JQ issues API and I get title and number and labels right we're just going to get one item back right but what if we wanted to sort them right what if we had a whole bunch then we wanted to sort them well JQ has a sort function I can just call it like this so here is an array I'm passing in and I call sort on it and it's just going to sort those you know based on the type that they are if you give it an object to sort like a whole bunch of if you give it an array of objects what it's going to do is pick the first element and those objects and sort based on those and if there's a tie go to the second one and so forth right reverse is another function right so reverse the output looks exactly like sort so maybe I should have picked a better example but it's just reversing the order that we have so if we change this to Z what's the best way to make this make sense we need reverse to work differently than sort and what if we put in duplicates right so then we do that there you can see it's not sorted it's just reversed so now those that is at the beginning there's a whole bunch of JQ functions probably the thing you need there's a function already for it so length is just going to give you the length of an array and then you can start chaining these things together coffee sit break so I have a document title is jeq Select so I want to get the title and then get the length of that and so the way I'm doing that here right this whole time I've been piping my results to JQ but here I'm using a second JQ so this title will return one thing the title JQ select and then I'll run length on it which will just take the length right it's kind of a strange way to do this but it totally works right we get the length is nine now there's a simpler way to do this which is that actually can you even notice the difference between these two lines it's the quotes right and the JQ because here what I'm doing is I'm running the pipe inside JQ so JQ has its own pipe it's different than the shell like terminal pipe that I have down here but conceptually it works exactly the same this selector will be run and then its results sent to whatever the next selector or filter is and so this means I don't have to run pipe JQ pipe JQ I can combine things and gets better than that because I can combine things in interesting places within my JQ query so using it all together things can get a little bit more interesting right so I'm gonna go I'm gonna go to this issues API right same as before I'm going to get the title I'm going to get the number but for labels I'm going to return the labels and then I'm going to sort them right so this selector here is using a pipe in the middle of it and using one of these functions so let's run that see what we get right oh here's our issue here's our labels there only seems to be one label in the when I tried this previously there was more than one label um examples going sideways but rest assured if there were multiple examples it would be sorting them right and how would it be sorting them they'll buy this first key right and if they were exactly the same then by the second key then we can do more from there right like we can do in our second example here we're going to return the labels but then we're going to sort by name sort by we're passing into sort by labels right and then sort by can sort by a specific key so now we're just using a different function to sort in a different way the specific examples of like sort and sort by don't matter I'm just trying to show you how you can combine the things that JQ offers to get some neat results right oh look at this we got two labels and we're sorting them by name fixed and master it's sorted to first right here so now we've got back our results we're sorting them by label but yeah that is how you use pipes and functions and if you need a specific function like you can just Google it JQ has a bunch of them built in and you can write your own which I'm not going to get to today and you probably will never need to do anyways but here's another example right now I am counting the number of labels using length right and now I can see that title number and labels by count is two so this is like a big step up the first things we covered were pretty simple right an array you know we can go into an array we can go into an object we can wrap things in the right we can wrap things in an object but now we're saying composition right we can combine things together with pipes and we have functions and we can chain things together you can take a value and then use a pipe and then send it to a filter or do some other transformation on it and you can do these things all in the various spots where you could select things so that's pretty cool right we're getting close to bringing it all together where you're going to be able to slice and dice various like rest a points or you know various Json documents or files that you have kind of ad hoc by just chaining these things together we're getting close to having the full picture and then you can compose things as you need but first I want to tell you about Earthly Earthly is a build tool nobody likes item with their Ci or with their build or you run some Suite of tests and then something occasionally fails because it's flaky it's just not fun or like you have some problem in CI and so you push it and then wait for it to run and then it fails then you need to make a change and push it again it's horrible Earthly is like Docker but for builds for CI it lets you run the same build exactly in CI on your local machine on your friend's machine on your co-workers machine seems like that shouldn't be a big deal but it makes a huge difference in productivity once you're dealing with you know like a team once you have a couple different tools and a little bit of variety this can really lock that down and remove that problem and also it's fast because it uses some of the same caching stuff that Docker does a year-based caching which means a lot of times it can build your build faster than you built it before and you can do it locally or on some beefy machine in the sky yeah so that's Earthly check out Earthly I work for Earthly and it's awesome okay last thing to cover before we bring this all together is maps and selects we're gonna make some more complicated things here so um similar query as before right I have the issues I'm getting 100 of them I'm gonna get the title the number the labels count right and I'm going to pipe that into length then I'm gonna get the title count which I'm getting by getting the title and piping that to length and you may notice that there is a little bit of duplication here right first everywhere I've been using the array Constructor I'm first I'm mapping into the array and then doing all this stuff and then I'm wrapping it back in the array so map is actually just a way to do that it's basically exactly equivalent to to this right so I just take that out and I put in map and now map just says okay go into the array apply this stuff and then re-wrap in an array that lets me clean things up a little bit the next thing that's super cool is Select select is how I filter select takes a predicate right basically something that evaluates to true or false I can so I'm putting together here I made title length right which is just title piped into length and now I'm saying I only want to select the elements where title length is greater than 30. right and the way I'm running that is I'm mapping into my list again I'm piping this whole thing so once I get my results from up here then I'm mapping into that and returning it right so now I'm kind of doing like a query like I mean I guess I was always doing query I'm doing like kind of a SQL lead query like checking for certain values and only returning them right so now I'm only returning things that have a title longer than 50 characters or 30 characters right and I can just keep going from there so like map lets me go into things selects lets me filter and at this point you basically have all the knowledge you need to combine all these various elements there's Map There's select there's pipes you can start putting things together right so so GitHub has a great API you can do all kinds of queries and stuff but I don't know how it works and I don't remember all the syntax but I don't have to know because if I want to find some very particular thing about issues like I can just turn the paging up real high to get back like 5 000 rows and then do this filtering client side right so here what I'm doing is right I'm same as before I'm getting back the title the number I'm building a URL of the issue that I want based on the number I'm getting the title length and then I'm sorting by title length because I'm saying oh what about getting all the issues with the longest titles I just have this weird idea that if you put together an issue and the title is like fix this or like broken it's not a good issue right but if it's like a very descriptive title and it's like oh found issue in version 3.5 but not 3.2 CD deals within more likely to be good right so I don't know if to how to query that using the API or the UI but I can do it right here in JQ so I'm going to sort by title length title length sorts it with lowest numbers first so to get that back I'm going to reverse it and then let's just get the top two so this is just like a random exploratory query I made up right and now we get the two longest issues back tidy up no ma'am hand invitations it'll yeah and this one can't use search bar and documentation because it's too low I feel like this is contrary to my theory right like this issue that the poor JQ guy has to deal with is just a complaint about the JQ documentation I mean docs are important but it feels like this isn't the most important issue ever and yeah if I want to like this is the nice thing when you understand just some of these components you can just mess around with this right like what if we look the other way so we're going to get rid of the reverse now we're looking at the the two smallest and look at this I assume that this is not a good issue let's look good day yeah exactly this is totally what I was thinking his his ticket starts with good day and then can somebody tell me how to build this project that's awesome oh yeah once you master these things a small amount of pieces with JQ then it's easy to explore just Json documents like this right so let's just review let's bring it all together this is the most important thing by the way this is you know next time you need to do something with JS document you could go to chat GPT and just ask it how to do it but if you understand these three lines I think that next time you have a problem you'll just be able to think it through maybe glance back at these three lines and you print them out or bookmark this or whatever and then you'll have the skill to solve the problem yourself so first thing right is just how to get into the elements that you want right so for objects you can just do like dot key for raise you can do you know element zero then sub key you can grab ranges of the array and you can just keep going right that's how you select in to get elements out into the shape that you want if you want something in an array wrap it an object wrap it in Array Constructor want something in an object you wrapped in parens which is the object Constructor and then you just build it up right key one and then my selector one key two my selector two where these can be anything I'm up here that gets you a lot of the way and then the last thing is just there's all these built-in functions and there's ways to pipe and combine things right so if I want to count all the sub elements of something right there is like length for counting an array right if I want to turn a number into a string because I want to concatenate it there's two string I don't and then yeah I can just give it my own labels and I can use map to more concisely get into arrays and I can use select click up here to filter out elements that don't meet certain parameters right so that's it that's these three lines are sort of the whole thing I mean you can go way further with JQ you can write your own functions it's sort of a full programming language a functional programming language that's kind of neat but you don't need that this is what you need these three lines I hope that was useful if you have more questions about JQ just hit me up I'll keep making these I'm trying to make a video a week we'll see if I can keep that up hopefully this was useful right yeah so that's the video right if you want to learn more about building software subscribe to the channel if you want to build software in a better way check out Earthly you know for a little tiny project if you're just using JQ and some shell scripts maybe it doesn't matter but once you get beyond that you know like having a solid way to build things that's no nonsense really can make a big difference and let me know your JQ questions in the comments leave them below and uh yeah might Inspire another video anyways until next time thank you for watching my videos

Channel: Earthly

Views: 12,024

Keywords: json, jq, json tutorial, what is json

Id: m9dhrq9iRHA

Length: 25min 26sec (1526 seconds)

Published: Tue Jul 18 2023