Jetson Xavier NX Lesson 4: Understanding Gstreamer for Absolute Beginners

[Music] [Music] [Music] just kidding guys we are not going to be rumbling because really engineers and computer scientists tend to be pretty mild-mannered people but i fear if they ever begin to rumble there might not be a way to put the lid back on it so we will not be rumbling but we will be learning some cool new stuff today because we are here with lesson number four in our incredible tutorial series where you are learning artificial intelligence on the jetson xavier nx now i'm going to need you to pour yourself a nice big enormous mug of iced coffee and i am going to need you to get ready to learn some cool new stuff so let's get out our jetson xavier nx gear and let's get ready to learn some cool new stuff hey if you don't have your gear yet check out the description down below i have links over to amazon so you can pick up the identical hardware that i will be using in this series and it will make your life easier in my life easier if we are working on the same set of hardware also as always i want to give a shout out to you guys who are helping me out over at patreon it's your help and encouragement that keeps this great content coming you guys that are not helping out yet look in the description down below there is a link over to my patreon account think about hopping on over there and hooking a brother up but enough of this shameless self-promotion let's jump in and let's talk about what we are going to learn today we are going to learn g streamer yes and i have to say this is one of the hardest things that i have ever tried to figure out because all the tutorials out there assume that you already have a tenured position in the computer science department of mit and it was very hard to kind of break it down into how can i use this or how can i actually take advantage of this tool and if you were here for lesson number three you have already done your first g streamer commands and i will just pop that window open and that was this long and tedious command here which we put in this long and tedious string which we put in last week to allow us to fire up the raspberry pi camera so if we run this program it launches the raspberry pi camera using g streamer boom okay and what it seems like is somewhere out there at some point along the way some guy figured out this huge long string that if you launch g streamer with that stream on uh the jetson nano or the jetson xavier nx then magically the raspberry pi camera would pop up but the problem is it is really really confusing looking at that and the problem is if we just go and run someone else's command how do we know that they really know what they're doing how do we know that we are running as fast as we possibly can how do we know if we're running at as high of a resolution as we can how do we know if we are getting the highest image quality that we can answer is we don't we just copy and paste what someone else does what we're going to do today is we are actually going to learn how to use g streamer now you're not going to be the world's leading expert when this is over but you are going to know enough that you can get uh you can squeeze a lot more performance out of your work when you are using g streamer in opencv on the jets and nano or jets and xavier nx now this work that i'm doing i am doing it in the context of the jetson xavier nx it would work exactly the same on the jetson nano and really it would work the same on any system that has g streamer installed if you have the uh jetson xavier nx or the jetson nano it is already installed on your system if you have some other linux machine just look it up it'll be just a simple command that you'll give to install it you'll have to kind of figure that out based on what system you're on if you were on a windows machine well i can't help you you're hopeless if you're using windows i'm sorry you're on your own but i will show you how to do it also this this ought to be able to run a lot of this stuff ought to be able to run on the raspberry pi for you guys that are playing along at home with a raspberry pi so what we're going to do is we are going to learn g streamer today and let me start by uh just kind of giving you an introduction to say what is g streamer well in uh the real world we have cameras and we have displays and then we got to do something to go from a camera to a display and usually it's not as simple as just plugging the thing in because there are a million different cameras which have a million different formats and a million different resolutions and frame rates and and encoding and decoding and all this sort of stuff and then a million different displays and so it would be like if you just had a hardware store full of millions and millions and millions of screws in a hardware store full of millions and millions and millions of bolts and you're trying to get them to work together it can be kind of tedious so what is g streamer g streamer is creating a pipeline that will go from a source like a camera to a sink like a display okay and now there are more than one sources the most obvious source would be a camera but what could another source be another source could be a file or another source could be a stream like a like a udp stream or something where you have data streaming in it could come from a file it could come from a camera and then similarly you could stream out or you could go to a display so there's different syncs and there's different sources and gstream is all about creating a pipeline from the source to the sync and we've got to start thinking in terms of these these terms they mean something source is something like a camera sync is something like a display okay so this is the way to look at it you're creating a pipeline but it would be like and you've got to get it all to match and all to fit together but it would be sort of like this imagine that if you're starting at one end with a pipe and the pipe was huge like three feet and square and then imagine at the other end you wanted something that was round and small so you got to go from huge and square to round and small well you might say okay we'll use an adapter yeah but the problem is you can't find an adapter that goes from huge and square to small and round so the way this works is you would maybe go from huge and square to medium size and square and then from medium size and square you would go to small and square and then small and square you could go to small and round so you would have adapter adapter adapter adapter and you usually can't make that trip with just one adapter you've got to stack them until you get something that takes what you have from your source and get something that will work with your uh with your sink okay now i'd mentioned displays or an obvious sink but in the context of the work that we are doing in opencv the opencv application itself is the sync okay so let's go back and let's look at that command that we did last week it ended up with what i hate that it ended up with app sync so that's the application opencv as a sync so you can figure this pipeline and it dumps the video stream into opencv in a format that opencv was happy with okay and so it's about building these pipelines and this is a little bit different because as programmers and engineers we think in terms of linearly giving commands like you would write a program to do this do this do this do this and you give a sequence of steps this isn't that way you were constructing a pipeline in one line of code and then the data goes down that pipeline and it goes in in one way and comes out in a different way it goes in and what you're getting from the camera and it comes out in what you need for your application or for your display okay enough of this pontification and let's jump in and let's learn g streamer okay i will need you to open up a nice fresh terminal and there's a quick install that i'm going to have you guys do and that's just because i'm going to be working today with both the raspberry pi camera and the webcam and when we're using a webcam you need to kind of know what the parameters are possible with your webcam and so there's a quick little program you need to install and so you need to do a sudo apt get install and it is v 4 l dash utils v 4 l utils ok that is looking pretty good and since i already installed it of course my install probably went a little bit faster than yours did but nonetheless you need to go ahead and get that installed okay and let me clear that i am going to do a tiny bit of windows management here to make sure that you can see this good okay so we talk about these pipelines you go from a source to a sink and then you do things in between if the source doesn't match the sync then you have to do conversions all right but let's just do and this got to tell you this will work for audio and video and actually it will work for just about any type of data any type of streaming data you can build g streamer pipelines on so it's a very very powerful but very daunting piece of piece of software to learn so let's look at the very simplest thing let's have a source an audio tone and let's have a sync a speaker well i will do a gst dash launch gst dash launch and then a dash 1.0 so we're using gstreamer we're going to launch it in version 1.0 well what i want is an audio test audio test source okay and then i'm going to pipe it to exclamation point is what connects one module to another module so my first module my source is an audio test source if you install g streamer i'll give you a heads up you already got this okay so don't worry about it it's just kind of like for demos it's a source so we don't have to go out and worry about what kind of microphone you have or something like that we'll just use the audio test source and then where are we going to send it we are going to send it to a l all s a also sync so this would be like audio something sync so we go from our source the test tone to our sink and then we click enter boom do you hear that we have a source and we've connected it to our sync and we have sound coming over our speaker okay so that is about as simple of a thing as you can do but now how do we really start learning because this is again back to the example i gave you before i know what i'm doing you don't you just copy what i do and you're never going to be able to do anything on your own what i'm going to show you here is the secret to starting to build these pipelines instead of copying other people's you've got to know how to inspect the element and then see what parameters you can set on it like what formats it knows and what things you can set and so i'm going to give you a new command i gave you the gst launch i'm now going to give you a command gst.inspect okay and we're going to do audio test src audio test source so we're going to go in and we're going to look at that module okay whoa audio inspect i'm sorry gst inspect dash 1.0 okay gst dash inspect dash 1.0 audio test source and look at all this stuff here man look at all this stuff so i'll kind of take you through this this uh very top stuff is just kind of like names and and stuff like that who who developed the plugin and some different things like that and then what really sort of matters is where you get down to the source template okay so remember you have two things you have sources and you have syncs now a converter module would have on one end a source and on the other end a sink okay but something like a a tone or a microphone or a camera it is just a source but the source of the camera feeds into the sink of the next block and then that so the sink of a block is kind of like the input and then on the other end of that block will be another source okay and then that source will go to the sink of the next block and then it will have a source that will go to the sink of the next block so what you've got to see is is that a sink is like an input to a module and a source is like an output it kind of seemed backwards to me took me a little while to really kind of get comfortable with that notion and i need to okay you know what i am going to do today yes i think i am going to uh give me just a second here ah give me a second i know this this these lines are like really long so i'm going to turn that off like that okay and now i think i will also probably get further out of your way and i'm gonna make this window bigger because those those lines are kind of long there this window kind of doesn't want to resize today okay and once again i have the sticky window problem where it has decided that in fact i re i hate that that i am really wanting to resize this full screen which i do not want to do okay that looks pretty good okay so all this is just background information on your on your source which is your module which is audio test source but this is what you want to see is you want to come down and look at the pads because the pads kind of tell you what the capabilities are so the first pad is the source all right now you can imagine for a tone you only have a source you only have an output you don't have an input so this has an output pad but it doesn't have an input pad and what you can see is is that you can now look and see the formats that it can put out it can put out audio x raw okay as the kind of upper level format and then within that there are all these different formats that it can use okay so that's kind of telling you what it can do all right this is telling you what it can do now what you really care about are the properties and the properties are the things that you can set okay the properties are the things that you can set okay and so you can see here that the things that we could set on this module through the command line is we could set the name or the parent block size uh number of buffers type find due time stamp wave okay so now here's all of a sudden something we might be interested in we can set what type of wave is generating the tone you could set a sine wave a square wave a saw wave we could do different things like that and so that would allow us to uh adjust and you don't really audibly notice a lot of difference but you can tell a little bit of difference depend depending on the shape of the waveform so how would we do that well we say let's see what a square wave is a square wave is the number one now default is zero so if we don't put a parameter we're getting a sine wave all right so let's come here and let me remind myself what that was it's wave and so if we just come and run it like this if we just come and run it like this we are going to get a sine wave okay it's pretty smooth sounding okay but if we come here we can put the parameter in after audio test source what was the variable wave is equal to let's set it to one which was the square wave and now let's run it okay wow that's a lot harsher sound the square wave is in the same way so you really could tell a difference there all right now i have to show you something very very important when we are setting these parameters when we are setting these parameters i am going to try to adjust this a little bit on the fly here okay that will put me a little bit further out of your way all right so when we are adjusting these parameters here is the module audio test source and now we're going to set a parameter we put a space not a comma okay so audio test stores space wave equal one i didn't like that i'm going back to wave equals zero let's see what else we can set oh look we can set the frequency okay the frequency will control the tone so let's come down and let's set the frequency again between parameters you put a space we'll put space frequency and let's say is equal to 800. that's very high let's put 200 that's very low i don't know if you can hear it i can hear it let's go to 300. okay so we can control the frequency uh let's see if there was anything else interesting there i thought i saw something else volume okay so uh the volume the volume parameter uh will go from zero to one and the default is point eight okay it will go from zero to one and the default is point eight so if we want it louder we will say volume is equal to 1. all right notice we have spaces between all of our commands you know what i think i'm going to do all right yeah so i can detect i can detect that that is louder i'm going to move this up like this and that way you will be able to see the whole line like that how you like that i hope you like that okay we've given our first g streamer command okay so the key thing is when you're setting parameters you put spaces between what you're doing also very important you have to have a space between the exclamation point and the next module if you put the next module without a space it doesn't work and so this is really kind of very tricky you've got to get this formatting very very carefully so number one when you're listing parameters you put a space between them like here and here and here and here and here in spaces in a minute we're going to be setting those caps that i showed you a while ago darn it when you're setting the caps you use commas and so you've got to keep this straight commas when you're doing caps and spaces when you are doing uh when you are doing parameters and let's go back and look at that program that we did last week okay so you see here is the module all right and then here is the parameter we use a space and now this exclamation is going to the next module no this exclamation what this is actually doing is this is setting the caps and so this is the other strange thing sometimes the exclamation point is taking you to the next model and i have frozen up i will fix that post haste okay there we go i'll try to be mindful to watch that it's just i've got so much usb stuff going on here my usb line freezes sometimes okay so you can see that sometimes this exclamation is taking you to the next module and sometimes the exclamation is setting the caps or the filter for the existing module and the way you look at it like when we looked up there on those caps it listed like a hundred different formats that it puts out well on the on the caps you put the exclamation and of the hundred it can put out you tell it which one you're going to use okay so you're you're sort of setting those capabilities but that happens after the exclamation point and so we are telling it here that we wanted video x raw memory for the format and then a comma in the size and a comma and the height and a comma and the frame rate so if you're setting caps you've got to put commas between the commands or the parameters if you're setting parameters you use spaces and just understand this is really really important and as i go through the examples it's going to be more clear all right so we look here audio test source had a parameter of wave equals of wave and frequency and volume and all of those are the space here we're not setting caps we're just going directly to the next module which is a l s a sync okay i hope that makes sense to you okay we changed the frequency a few times we looked at the parameters for the audio test source okay so let's uh let's go back and let's look at those parameters again okay so i can uh go back up to that gst inspect all right and we're going to look at audio test source and one of the things that you see here is you see under caps okay that's uh it's under what they're calling pads here or or the element properties okay this is what we call the capabilities the caps or the properties and i am misspeaking here here it is caps are capabilities i misspoke caps is short for capabilities which would make sense okay so you can have audio x raw out and that's kind of the only option but then for format you can do s s16le b e u 16 l e you can do all of these different formats but what you got to ask yourself is what format does a l a a l s a sync want so we better look at that one okay so we are going to look at a lsa sync a lsa sync and then we can see that this is going to be a sync so it is not going to have source caps it's going to have sync caps and so let's look at this you can see the pad template is sync okay and then here we are with capabilities and what does it like well it likes s8 again good news is it takes audio x raw so we're not going to need a we're not going to need a a converter in there we're not going to need an extra module we can feed directly from the source to the sink but what we have to make sure is is that we're outputting from our source something that can be read by the sink and we're looking at the sink now and we can see that the sink will take u8 okay there's probably others that will match but i see that it takes u8 and then i also see that if i go back to the source under capabilities it has u8 as an option as well okay so we can get a little bit more sophisticated in that we can do the gst launch we can use wave is equal zero frequency is 300 volume is one but now let's set the caps for the video source how do we do that well we put the exclamation and so now in here we're going to set the caps now there's two things we could do in here we could put a whole new converter or we could set the caps for the previous module we're going to set the caps for the previous module and what do we want well the format that it would do was audio over x dash raw now what do i do here do i do a space or do i do a comma this is a cap so i do a comma all right and now what do we want we said for matt equal u8 okay format equal u8 does that make sense it really should okay so we're taking that audio test source and we're putting this format on it and then we are dumping it to the alsa sink ah that should have been an equal not dash u8 hopefully you guys caught that okay all right now we're getting the same tone and we don't detect any difference in the tone but what we've done here is and it's ctrl c to kill that in case i didn't say that ctrl c to kill what we did there was we did it in a format of our choosing and that's important because for other sinks it might need a certain format so we're giving the format of our choosing there okay so that is pretty uh that is pretty simple okay let's try another one let's uh come in and let's try u 18 e okay let's do format u18e oops what did i get i got an error and i got an error okay i'm sorry because that's not one at all it's u18le okay and here i got an error and that is because uh let's go back and look look at those capabilities for the audio test source okay inspect audio test source and let's look of all the ones that it will do up here it it does u 18 l e okay so it can produce that so that did not generate the error because that's one of that's good as far as it can produce it so let's look at our cap of our let's look at our uh inspect our sink okay like that and now let's see what formats it can use up here okay i see u8 i see u16 i don't see i don't see the i see l s18le but i don't see u 18 l e okay so that is where the problem is that i chose something that wasn't going to work all right and so that was uh you let's go back where was that u18le i get an error it can't make that the output and the input are not matching together correctly all right so watch this though i've got an output or a source coming from the audio source and the sync doesn't like it so what do i need to do in order to make that work i need to do an audio convert okay and here i'm going to add so this is the module audio test source these are the caps that i'm setting on it and then before i go to the sync i need to what i need to put a converter in here and so i'm just going to do space audio convert and sometimes these audio or video converters they'll figure out this guy's giving me this this guy wants this and they'll figure it out and kind of get something to match and you don't have to put any parameters boom okay so i took something that wasn't working and i made it working i made it work by putting in this additional module of audio convert now this one is kind of smart and it saw what it had and it saw what it needed and it made a match but usually what you would do is you would come in and you would you would inspect audio convert and you would look at its capabilities these are its sync capabilities these are its inputs okay and if you look at it at its inputs it will take in here somewhere u 18 l e okay it will take that in and then it will put out whatever it was that that uh that that uh audio sync wanted okay but normally you would look at the inputs and look at the outputs and you would set those as as capabilities and therefore you would get the conversion that you needed okay well let's keep moving right along here and i don't mean to belabor the point but it's just an audio source and an audio sync it's so simple and if i can get you to understand how it works with something really simple then when we go back and we look at that mess that we had in our python program it'll begin to make sense if you don't understand something figure out what the module name is go and do a gst dash inspect dash 1.0 on it look at the caps see what it wants as an input see what the caps are for the sink see what the caps are for the source and then you know do your magic okay so let's uh let's move right along here and let's see what else we can do so we did an audio convert there okay and actually what we can do is we can there we just sort of let we just kind of let the audio converter figure it out but here i'm going to show you how i can set the caps and so we've got coming in as the input u 18 l e and then what we knew is we knew that a l sync what it liked was u8 and so what i can do is i can come in and set the caps or set the filter for the audio convert module and remember mindful to put your space in there and we are going to still stay in audio x raw okay we're going to stay in audio x raw and now we're setting caps so what do i need to put we're setting cap so i put a comma not a space and then format equal u8 like that so i'm taking u18le which alsa does not like and i am audio converting it to u8 boom look at that okay so you're converting your inputs to your outputs that's kind of neat isn't it okay enough of this audio nonsense if i have an audio test source what do you want to bet that i have you can bet that i have a video test src video test source and i'm going to pipe that directly over to x image sync okay let's see if we can get lucky like we did with audio we're just taking the test source and we are piping it to the image sync x image sync okay boom look at this we got the happy little test bars and you can see it's live video because you kind of have snow down here i think that is pretty darn neat and so we got lucky that one of those formats that video test source outputs is one of the formats that x image sync can read and so sometimes when you're doing simple little demos this works but never in the real world in the real world you got to do a lot more than that but the first thing let's do is let's go in and do a gst inspect on 1.0 and let's inspect video test src all right let's see what we can do there hey there's a flip method there's a motion there's an animation mode there's a lot of stuff that we can set here okay and these are all properties or parameters right these are all properties that we can set so we would do that without the exclamation in just using the spaces we could set quite a bit of interesting things and i think one of the things that we might want to do is it looked like pattern was interesting you see we can set all these different patterns okay look at that there's a circular pattern let's see what that looks like all right let's see what a circular pattern looks like so we're going to come here and it is a property it is a property right so how do we set that we set the parameters here space p-a-t-t-e-r-n is equal to i do believe i said 11. okay look at that we got that circular test pattern by setting our parameters okay by setting our parameters let's go back now and let's set our pattern back to zero and let's go in and let's set the capability all right so i guess we probably are going to need to inspect that again or we just inspected it right this video test source let's go up and look at what the capabilities are or yeah the caps so we'll come up higher and this is just a source okay go up higher so this is only a source and here are the caps here are the capabilities all right so we can get a video x raw out a video ex bayer out and then we have all these different things and like you'll probably recognize bgr is blue green red alpha uh xrgb is sort of just a four channel but it gives you rgb i think the x means nothing is really being used there and so you can have gray 16 bit you can have gray 8 bit you can have nv 21 nv nv12 is one of the ones that nvidia uses a lot on some of their commands you can see that there is a whole lot of things that you can do with this source but we we could choose between video x raw and we could choose between that and i thought i saw another one video ex bear right so those are the two choices of what we could set but let's try the following let's set those caps because you know like i say in the real world the output never matches the inputs you're going to have to learn how to do conversion so we're going to do that we don't have to do a video converter we're just going to set our caps we're going to say video x raw and these are caps so what do i use a comma and then format equals b g r for blue green red so let's see if that works okay it did not image sync x image sync did not like bgr so either that or i made a typo video x raw that looks pretty good so let's inspect x image sync okay so we're going to do a gst dot inspect dash 1.0 and what are we going to inspect x image sync okay so let's come up and look at its source capabilities okay so here's its capabilities i'm sorry it's sync capabilities it's an input so it's a sync it likes video x raw okay but it doesn't look like that it is going to recognize the bgr okay it doesn't list bgr as one of the things it only likes video x raw well what if we what if all we had was vgr i mean bgr what if what was coming was bgr and we got to figure out how to get our video x raw bgr we got to figure out how to get that over to x image sync which just wants the video x raw how are we going to do that okay well what we can do is we can use one of these smart ones you can do a video convert or a lot of times if you really want it to figure it out okay what we know is this output does not play nice with this input so there is a really cool converter that is called so we're going to go between the two exclamation points this is not caps this is a new module and it is auto video convert and it's kind of like going to see what am i getting from this guy what does this guy want and i'm gonna make the magic happen okay there you see that worked so the auto video convert kind of took care of the hard thing for you so i didn't have to go look at the format coming out of the video test source and i didn't have to go look at what was needed by x image sync this auto video convert took care of things for you my experience has been in the real world that these auto converters usually don't work and you really do have to kind of go in and look at the output of one look at the source of one that's the output look at the source coming from one and look at the sink wanted by the others and you got to kind of set those to something that matches now yeah you want to set it to something you want but it's kind of got to be two things it's got to be like what you want and what you can make work i hope this is making sense okay so let's uh let's go again uh so what do we like about this what do we like about this we like that it works i've got an input or a source and i have an out an output or a sink and i've got them working together i kind of like that but i don't like this little small window right so so let's start making things more like what we want so i'm going to come up here and that auto video convert really doesn't give you any handles and so what i'm going to do is after doing that that's going to get things in the right format for me but after doing that i'm going to do kind of a manual video convert okay i'm going to do a manual video convert video convert i'm going to do a manual video convert and now i'm going to set the capabilities of that so i put an exclamation and now comes the caps i am going to keep it in video x raw i'm going to set the width equal 1280 i'm going to set the height equal 960. all right boom look at that giddy up so i made a big window with the test pattern all right now what you might ask is how did i know to use video convert well you go in and you look at these modules and you could see that if you looked at the inspection of auto auto video convert it doesn't let you change the size but video convert does and so i let auto video convert take care of the problem that x image sync doesn't like bgr but then i use video convert because it gives me the caps then that i can resize that window so again keep in mind that you've got the source you've got the sync and you're putting all these little adapters in there until you get something where all the outputs all the sources play nicely with all the sinks and you just kind of make the changes until you get these two things that don't match on the ends to begin to match so i think that is pretty neat okay another thing that we could do is another parameter that we can set with the video convert or maybe i should just inspect that video convert just to all right so if you look at things that you can set with video convert gamma mode matrix mode alpha value alpha mode chroma resampler you see there's a lot of things quality of service parent there are a lot of things that can be set there and one of them that can be set is the frame rate okay now i ask you is the frame rate something that you would do with the command with spaces or is it something you would do after the exclamation well if we look at frame rate it is a capability it is not a property properties go with the command go with the module with spaces capabilities come afterwards all right so if we're going to do the frame rate we come up like this okay and where we uh where we just looked at video convert so we're in the caps on video convert and we're going to put comma frame rate equal and now i will tell you this is the goofy thing that gstreamer wants frame rates as an integer over an integer so like an integer over a one so i'm going to say it is 30 over 1 for my frame rate and that looks strange you would say why not just use 30 because it wants to see the frame rate divided by 1. and i don't know if that's a way that they could get a floating number like maybe it has to have an integer and maybe if you wanted a 33 frame rate you would put you know 100 over 3 or whatever but let's see if this works okay boom there it is and look at that you see we've got a nice 30 frames per second frame rate now you guys know of ah sticky window problem again okay what you guys know right is is that on a camera you can only set the frame rate to frame rates that the camera will support and also you couldn't like run the camera at 10 frames per second and then downstream uh up the frame rate to 30 frames per second now you could slow the frame rate down but you can't speed the frame rate up because it's actually the camera that limits that so let's kill this so let's just look so let's come in and say that we give a frame rate of one frame a second okay so now i want you to look at this do you see this gray one two three so you see that is changing once every uh once every second now i just can't resist i wonder if you could make it uh two seconds per frame by saying the frame rate is one divided by two let's see if that works that's kind of crazy crazy thing to try to do one thousand one one thousand two yeah okay so you can make very slow frame rates by making the denominator and probably they wanted to work with integers and that's why they made you do it in this goofy way okay i think that was pretty neat okay what are we gonna do here all right we we play around with different frame rates we've already done that okay so now instead of using these goofy test patterns let's do a gst launch 1.0 and i want to launch my raspberry pi camera again i am on the jetson xavier nx it will be exactly the same the jetson nano it will be the same on these different nvidia products that use jetpack if you're on the raspberry pi i'm not sure exactly whether it would be the same or not but what you could do is you could look that up pretty quickly but what it is to use the camera what is the name of that camera as far as a source it is envy like nvidia it is an argus arg us camera src that is the name of the raspberry pi camera probably more accurately it is the name of the little port that the raspberry pi camera is hooked to on the jets and it is the envy it's an nvidia argus camera src src is in source okay now i have no idea so i'm just going to do an auto video convert and then i'm going out to x image sync all right so i've got a source and a sync and i don't know how they what says what what's listening for what so i'm going to do an auto video convert because i haven't gone in and i haven't looked at the caps in the properties auto misspelled it auto video convert boom that frightening that is larger than what you want to see me it's like too huge and it's upside down so we're going to ctrl c out of that hopefully ctrl c out of that i'm going to have to click over here at least it didn't close up my terminal and i will control c here all right what's the good news so the bad news is we are frozen up again good news is i can fix that good what is the good news the good news is that i did get the in vargas camera source talking to the x image sync now let's go over here and now all of a sudden on this cam set that we did last week we understand what that is that is the module that represents the raspberry pi camera there are two little slots on the jetson xavier nx and there's two little slots on the new jets and nano boards and so we have to tell it which one of those we're using so i set the property okay i set the property to sensor id which is zero which is the one when the when the slots are facing towards you which is on the right and then exclamation i come in and i start setting the caps of that all right i come in and start setting the caps of that so you see already that is beginning to make sense i do believe so what do we like we've got the camera talking to the display we've got the source talking to the sync that's good but it's kind of like when we do that audio convert we don't get any input at all on what is going on so let's try to do a couple of things here the first thing that i want to do is i want to after yeah after doing the the camera source after launching that i want to put an nv vid convert that's niv v that's like nvidia vid conv like video convert nv v-i-d-c-o-n-v and then it's a parameter so i'm gonna put flip method equal to okay and then i'm going to go ahead and hand it off to the audio convert no i'm gonna on this in video convert i'm gonna go ahead and i'm gonna set the caps there on that and the caps that i'm going to set are i am going to tell it i want video x raw and i want a width equal to 1280 and i want a height equal to 840. and you guys probably don't know that this bothers me way more than it bothers you that this thing keeps freezing up it really bugs me but i just have so many cameras hooked up i just overwhelmed the bandwidth and i could reduce down and make things a lot simpler or we could just live with this glitching out every once in a while so we are going to do an nvidia convert we're going to set the parameters or the properties to flip method equal 2 which should flip the camera upside down and then we're going to set the caps and as we're setting the caps we're going to change the width and the height to something more reasonable boom look at that alright you guys see that look at that that is the raspberry pi camera there and you see now we are starting to exercise some control over it we are starting to be the ones in control i wonder okay let's uh let's look here i wonder if i could do 640 by 480 i'm just curious 640 by 480 like that yeah look at that that worked okay hello all right so we are starting to take control we are starting to take control of our camera all right now let's go ahead and let's do an inspect on gst inspect 1.0 because i'm kind of just throwing these parameters at you and they work and the reason they work is because i'm in there doing these uh inspections in the vid conv let's look at that and then if we come up and we're going to look at what this thing can put out so since we're looking at what it puts out we're going to look at its capabilities and we don't want it sync capabilities we want its source capabilities it will put out video x raw with the memory nvm or it will put out video x raw so we chose to put out video x raw and then the parameters that it will take are width height and frame rate those are the capabilities that you can set we set width we set height and we did not set frame rate now again you can only set frame rates that the camera knows or that the camera is capable of but let's just try that just for fun so on that convert we are going to comma frame rate equals and we're just going to have to kind of guess because you can go in and look at it but right now i'm just going to guess i'm going to say can we do 20 over 1 like that did not like that maybe a 21 over one did not like that it's kind of not good to just sit here and guess you know but and no okay i just want to look okay so really probably the better place to set the frame rate why do i go off on these tangents like this but let's see if we can set it as a cap here set it as a cap of the camera itself and so we would set this as video video x video x raw and frame rate equal 21 over one let's see if that will make it happy i didn't put a space i warned you guys about that where was that i'll just have to type it in again nvidia convert okay so we are gonna put the exclamation there and then here we are gonna say it is video and over x raw and then frame rate equal 21 over one let's see if that will make it happy still not happy okay well the frame rate it's got to be one that's offered by the camera and i really thought that that was one but maybe it is not but i won't waste any more time on that what you can see is you'd have to go in and do the inspections and do the inspections but i don't want to slow us down with that but let me see if we can continue here so we did that okay so really probably what we need to do here is we need to come in and i'm just going to kind of start this whole thing over and and really we would be better off to come in and at this point where we have our source okay let me think about what we want to do here okay probably right at the source is where is the best place to set these things because if if when you have a source if you don't set these things you get further down the pipeline and you're really not sure what you have and so right here i'm going to set the caps of the source and what i'm going to do is i'm going to say now this time is a little different we're going to have to make this a string and the reason we got to make it a string is because it has print it has parentheses and because of that i've got to put all of this inside of a string so i have to put a string because g streamer doesn't want to see a parenthesis so i've got to force it to be a string okay and so we're going to have video over x raw and remember that this wanted memory colon in vmm like that that's what it wanted okay then it has the width is equal to now let me give you a little pro tip on this raspberry pi camera don't set the width and the height you want as a capability of the camera because if you do all it does is crop it is putting the video out at 3264 pixels wide and 2464 pixels high okay and if you change it here it doesn't really change it it just crops off the part that you didn't keep what you want to do is you want to grab the whole frame and then you want to convert it and you might say well there's a lot of overhead with that no not with this nvidia not with this xavier nx it has a huge pipe between the camera and the gpus and so it is no problem at all grabbing this whole frame so we're going to say the width is 32 64 and then the height is equal to 24.64 and that's just what it is that is the limit of the camera so with this 3264 height is 2464. these are caps so we're using commas okay and now uh we'll put the frame rate is equal to 21 over one okay now i'm going to end that quote all right because it was this thing that you've got to force to be a string from here to here and any time one of your caps has these parentheses in it you've got to kind of force it to be a string because that confuses the parser if you don't understand it just take my word for it you've got to do it if you want it to work okay and so now what i'm going to do is i'm going to go to my next module which is nvvid conv why because that's the module that has the flip if this other one had the flip then i could have done it there but it's not a capability and it's not a parameter so i'm going to flip it here nvidia convert and then the parameter or the property is flip method equal to okay now i'm going to set the caps of the nvidia converter i want to go to video x raw not with the memory nvm because i know that my sync doesn't want that and so i'm doing the conversion here to get the format right and now i'm going to do the for the the conversion of width equal 640 comma height equal 480. so you see what you do we're grabbing the full frame and then in the parameter nvidia convert in that per in that uh module nvidia convert we are changing the dimension as part of the cap the capability all right and then that's all we really need to do and then just to clean up any formats that aren't working we will say auto video format and then x image sync like that auto video format oh auto video convert i meant auto video convert and then x image sync boom all right so you see how we're generating this thing okay and also i'm just curious i'm wondering if you really need that auto video convert what if we tried it without that we are in a we are in a video x raw and i'm not sure about some of the other parameters okay we can do it without that audio video convert we were close enough that we didn't need that step so we could take that out all right guys do you see how this is starting to look a whole lot like this do you see how that's starting to look a whole lot like this i'll give you a heads up why was this cam set so difficult the cam set was so difficult because appsync that is opencv that's the app appsync wants bgr and so what i had to do was nvidia convert the nvidia converter can take you from nv12 ii bgrx but it cannot take you to bgr one of its caps one of its capabilities was not bgr so i did the next best thing which which was bgrx and then i used our old friend video convert and he took us all the way to bgr which then goes to opencv which is the app and so that's why you have like all of this from here to here would be if they had made nvidia convert where it could do bgr we could have made this a lot simpler but hopefully this this this black magic here is starting to to make sense that we're trying to get from what the in var envy argus camera produces which it really likes to output the video extra memory nvm and we take that all the way to opencv which wants bgr all right what do we want to do now okay let's have a little fun here you can also do some special effects so we know now that everything is working right we know now that everything is working i wonder if i can make that like 800 by 600 yeah i say it's working then i go in and monkey with it it's less dangerous to change this one than it is that at first width and height let's try it like that that's pretty good okay let me get this out of the way a little bit all right now there you guys just go in and play with this but there's all types of cool things like i can come now i know that the thing is lined up the way that it needs to be so i could add a module that like does a special effect so i could say like aging tv and i could add that module in and boom do you see how it makes it look like an old i don't think that's really an old tv that's more like an old uh super 8 movie right it's putting the noise and the dirt from the super 8 movie on it i think that's pretty neat and then let me just inspect that i'm wondering what type of parameters you could put on that aging tv ah i did a launch on aging tv and citizens uninspect and inspect okay dust is true pits is true color aging is true scratch lines you can go from zero to twenty that it's set at seven are there any uh dust is either true or false pits is true or false color aging is true or false the scratch lines you could put a lot more scratch lines in there so let's try that let's come down here and let's put in scratch lines scratch lines equal 20 just because we can i thought i saw scratch lines with a hyphen okay i guess there's a lot more scratch lines there i'm not noticing a huge difference there let's make sure i put that in right that looks good okay and i didn't see anything else too interesting as far as parameters that we could set because most of them are either do you want it or you do you not let's try zero scratch lines and see i just want to make sure that we're actually changing something i should get rid of the scratch lines okay indeed it did get rid of the scratch line so that command does actually do something okay so let's uh let's kill that and let's see one other thing that i think is kind of interesting [Music] what is kind of interesting is we could also do we've done aging tv we could also do another special effect here we could come in and we could put in another converter and what we could put is the converter color e effects and the preset is equal to sepia okay like that all right now we're really looking like an old like an old uh old-time movie we got the dust we got the sepia we've got all of that and so that is kind of neat i really like that i like that a lot and so you guys can see i mean really what you've learned from me is you've learned that you have sources and you have syncs and then you have these modules and the camera modules are just sources the things like a display it's just a sink but the things in between it will have a source which is uh it will have a sink which is the stuff coming in it'll have a source which is the stuff going out and then you can either set properties with the module or you can set the caps after the exclamation point if you're setting the caps you got to use commas if you're setting the properties you use spaces all right and then you you go sync to source sync to source sync to source and you string these things together and so you and then the most important thing you learn from me is just that gst inspect to see what it can take coming in and what it can give put you know on the output and then i'm just stringing enough of these things together and i've just shown you a few of the modules there's a huge number of modules and so you just got to kind of play around with it but i do believe that i've given you enough modules now that you can kind of understand uh what this long string was that we did over here we go all the way to appsync from the camera so let's go ahead and look at this and let me set this larger let's see i'm going to run this and then see what some of the numbers are and then i'll run it again okay so now i'm going to quit out of that and then what i want to see here are what some of these 1280 by 720 so i'm going to set this one at 1280 and guys i hope you have your program from last week if you don't go back and look at this from last week so some of you guys just came in and you learned g streamer some of you guys are going through the whole series of lessons if you're going through the whole series of lessons i need you to go back to the lesson we did last week and remember here we set width and we set height and then that i'm going to use this one and i know i'm going to use i'm going to use the raspberry pi camera which is this one and then remember how we take these three numbers and we sort of insert them into this long string like this okay but 1280 by 720 should be pretty nice size so i'm actually changing that g streamer command okay boom all right now if we look at this and let me try one more time i'm sorry i just i'm such a perfectionist that it really slows us down sometimes let me try 1920 by 1080 let's see if we can even manage that 1920 by 1080. it's just i want you to see a big view of what i'm doing here okay let's run this program i hope we can run the program well that was that was the wrong size by a long way wasn't it 1920 by 1080. i warned you that if you put the wrong dimension then you can crash the program okay boom there it is okay i don't know how well you can see this but what makes me happy is we're controlling the camera and gstreamer what makes me happy is we're finding our way with these video converters between inputs and outputs let's see on latency there is maybe a hundred millisecond latency but that is not bad at all so we're in control of the camera we're getting the size we want we are able to go from our inputs to our outputs we're able to build these streams we're learning lots of parameters we are learning lots of cool things okay all this stuff is working really well what is it that i don't like okay i don't like that if you look at this do you see all the noise like uh where is a good place like if you look kind of back at the background here like at the soap dispenser and at the wall do you see all that white snowing noise i don't like that okay i also don't like that the contrast of the image is very low i don't like that the saturation is low so yeah i've got an image and i could do image recognition and i could do machine learning and i could do all that stuff like that but what really bugs me is this is an 8 megapixel camera and we ought to be getting a more visually stunning image out of it than what we are so your homework for next week is to go in and play with g streamer okay it's to play with g streamer and uh i got a cue out of that one okay uh play with g string and i want you to come up with a g stream string that will launch that raspberry pi camera and what i want to see is i want to see no noise i want you to get rid of all that snowy noise out of it i want it to have higher contrast i want it to be darker and i want it to have a little bit more saturation okay i want you to create something that is a lot more visually stunning image because if you come and look over here do you see how clear this is how good the contrast is how good everything is that's because this is really doing things professionally here and over here we are just sort of dumbly launching a camera and and in today's lesson what we really worried about was just getting from our source which was the raspberry pi camera to our sync which was opencv and get all of that thing set up but now we want to make it look good and so that's what your homework is for next week is to go in get that string to where you got it looking really good and i'll try to make this a premiere maybe you guys could share on the premiere what the uh cam set string is that you create to get the best uh to get the best stream and maybe then we could all play around with them and sort of see who came up with the best one okay guys i've had a lot of fun with this i hope this lesson hasn't been too tedious but what i really absolutely hate is i hate to just run out and copy someone's line of code and then launch it and use that and not have a clue where it came from i think even though today's lesson was tedious i think you know and understand now what that cam set string is and you have an ability to go in and make it do things for you if you didn't see the converter or you didn't see the effect that you wanted today at least you need to you know how to go in and find those modules that are available and then find the caps and find the properties of those uh of those modules where you can make g streamer do what you want okay you can make it sing and you can make it dance all the way that you want okay guys this is paul mcquarter from boy.com i will talk to you guys later

Channel: Paul McWhorter

Views: 41,349

Keywords: Jetson Xavier NX, Gstreamer, OpenCV, Python, AI, Machine Learning

Id: _yU1kfcC6rY

Length: 82min 42sec (4962 seconds)

Published: Wed Jun 10 2020