Part 9 - Unix/Linux for Testers | Piping and Usage of tee and xargs

hello everyone welcome you all in the today's video we are going to learn one of the very important concept from unix or linux which is piping concept so first of all let us try to understand what is piping and what we can achieve by using piping concept so let me put some points here regarding the piping so the piping means sometimes we can pass the output of one command as an input for the next command which is called as a pipe for example let me execute one command like this let's say i have executed one command and this will produce some output for us so once you produce some output we can pass the same output as an input to the next command let's say command 2 and then we will get the final output then we'll get the final output what does that mean is whenever we execute a command which will produce some output and that output we can pass as an input to the next command then we can get the final output so this is basically called as piping so we can pass output of one command as an input for the next command is called piping so by using piping concept we can execute multiple commands together we can execute multiple commands together so that means one command will give you some output and that output we are going to pass an input to the next command that means we are able to execute multiple commands together by using piping concept so we can use piping using vertical bar so this is a symbol which you have to use to achieve the piping concept now i'll show you a few examples how we can use piping let me just write one example here so let me just print something let's say els iphone l slash etc so what this command will do is ls iphone l command will display all the files and directories long listing of files and directories and then i have provided some directory here so basically this particular command will display all the files and directories from this particular directory let me just execute this command once let me connect to my ubuntu vm right so this is my terminal so this is my terminal let me execute this command ls iphone l slash etc so this is the command i'm executing ls iphone l slash etc so which will display all the files and directories from the edc directly let me execute it now we got some output here so where i got the output here standard output this is a standard output means we got the output in the terminal itself terminal itself right so whatever output i am getting from this command right i just want to find out how many lines are displayed here so once i got the output here i want to find out how many lines of output we got here as a result let me execute one second so when i execute this command which will display the some number of files and directories after that i want to find how many lines are displayed here okay so then what i can do is i can use one symbol called pipe and how to find number of lines we use wz command and lines words and characters also we can find if you want to get only lines we can say w iphone l is a command now let us try to understand this command ls iphone l slash etc this command will display all the files and directories in the standard output device that is terminal and that will be passed as an input to the next command so whatever output you are getting from this command we are passing that output as an input to the next command so what this command will do is wc iphone l will count the number of lines so whatever output you are getting here it will just count the number of lines now let me execute this command so clear it i say ls iphone l slash etc pipe and then here i'll say wc just x0 wc what it will show so it will show something like this so this is number of lines and the second number which is representing number of words and number of characters and if you want to get only number of lines we can simply say iphone el now we got only number of lines now if i try to understand this command ls iphone l slash etc which will display all the files and directories and that output we are going to send as an input to the next command by using pipe so what the next command will do which will take the output of the first command and apply this command wc iphone l then which will find the number of lines in this particular output okay so the first command will display the output in the standard output device which is a terminal okay and that will be taking as an input to the next command then finally we got the result here so let me just execute one more command for example let's say example two ls iphone l slash etc okay now when i execute this command normally what happens is when execute only ls iphone ls slash etc let me first clear this so ls iphone l iphone es slash etc now it is printed all the files and directories list here right but at a time i cannot see so i want to display all of them page by page so i want to display the output page by page so then what i can do is i can just apply the pipe here so ls iphone l slash edc which will display all the files and directories and the output of this command the output of this command i'm going to send to one more command called more more or less whatever command you want you can use so basically the more command will display the output in the page by page so for example this particular command will send the all the files and directories list and the more command will show that output page by page so what is the input of this more command is the output of the first command will become the input of the next command so that can be achieved by using piping concept let me execute this when i say enter now it is displayed the first page now when i say spacebar then it will display the second page now third page fourth page fifth page sixth page seven eight nine ten eleven twelve so totally twelve pages of data which is displayed so what this command will do is ls iphone edc which will display all the files and directories and that output i am passing to another command called more which will display the data in page format page by page data okay this is one more example let me show you one more example let's say example three so ls iphone el slash etc so which will display all the files and directories from edc directory then i want to display top 5 lines for example let me just clear this i am executing the same command when i execute this how many lines which is displayed totally so many lines are there but i want to display only top five lines top five lines i want to display from this output so how we can write the command top five i want to display so i can write one more command pipe we can use head command right head hyphen five now what this command will do is the first command will display all the files and directories and that output is going as an input to the next command so in the what is the next command hit command we'll take the input from the first command and find out the first five lines and we'll display as an output so let me execute this command clear ls iphone l slash etc pipe i say head iphone 5 so now we can see only the file ends are displayed including the first line one two three four five lines are displayed so the first command will display all the files and directories and that is an output we are getting from the first command and the same output we are going to send as an input to the next command that is head minus five so this will display the first file ends of the output so we can achieve this piping so the input of the first command will be sending as a the output of the first command will be sending as a input of the next command so the output of the first command will become the input of the second command okay so these are all few examples so now let me show you one example let's say i have example four okay in example four so in this what i'm going to do is say as a ls iphone this is a command i'm executing ls iphone l again last edc and which will display all the files and directories i just want to i just want to count number of lines number of words and number of characters simply what i can do we can use wc command so what wc will do is whatever output you are getting from this command wc will just count the number of lines number of charac number of words and number of characters so let us try to execute this so let's try to execute this command so which is display this output so ls iphone lash edc displayed some output all the files and directories and that output is going to going as an input to the next command which is wc and what wc will do which will find the number of lines number of words number of characters we got the output now again i want to pass this output to the next command let me write like this so i'll say wc pipe and i'll say wc iphone l now what is an output of this command so let us try to understand the first command will display all the files and directories and that will go as a input as a and put to the second command wc which will return the output like this so number of lines number of words number of characters so only one line which is written which contains a number of lines number of words number of characters and above on top of this output we again applied one more command wc iphone l so whatever output we are getting here on top of this output we applied wc iphone l so what is w iphone l will do which will find the number of lines so how many lines we have here only one line so the final output will be one so the first command will return all the files and directories which will send us a input to the next command which will find number of lines number of words and number of characters and that is again going as an input to the next command which will find the number of lines which is one okay so this is how we can just use piping concept so what is piping means the output of one command we can pass as an input for the next command and in the piping we can use multiple commands together and we can achieve piping by using vertical bar okay so these are few examples so now let me show you one more example and if i just look at these uh examples in every example the first command whatever executed basically this uh this particular command will return a standard output right in the terminal itself and that output we are using as an input to the next command we are using that output as a input for the next command okay remember this now let me just give you one more example let's say example four okay so this is example five so in example five my requirement is let me write my requirement here so this is my requirement so the output of ls command should be saved to output.txt file this is my first requirement so the output of ls command should be saved to output.txt file and should be provided as input to the wc command so let us try to understand this requirement once again the output of ls command should be saved to output.txt file this is one requirement so the output of ls command should be saved to output.txt file and the same output of ls command should be provided as an input to the wc command okay so what i'm going to uh tell here is there is ls command so whatever ls command we will execute this is my ls command so once ls command executed we will get some output so we'll get the some output right so once you get the output i am going to use this output i am going to write this output into dot txt file okay i am going to write the output into output dot txt file let's say output dot txt file i want to write the same output and the same output i want to send as an input to the wc command so this is my this is my requirement so whatever ls command i'll execute whatever output we are getting from the ls command the same output will be written in output.txt so the output of ls command should be saved in output.txt the same output come here output.txt and whatever output you are getting from ls command should be provided as an input to the wc command input to the wc command so this is something a different type of requirement now we will see how we can achieve this okay so normally when i write like this let me write some example here and then we will understand so let me write like this first requirement is what the output of ls command should be saved in output.txt so what is basically it is doing it what is basically talking here is redirection so the output of ls command should be saved to output.txt so normally what you will do ls iphone l this will give you or lsifn slash edc so whatever the files and directories we have in edc will be displayed so the output of this command should be saved to output.txt that means we can use redirection symbol right so i'll say output.txt so this command achieved the first requirement so the output of the ls command should be saved to output or takes you this is the first requirement right and after that what i'm saying and the same output whatever output you are getting from this command should be provided as an input to the wc command input to the wc command okay input to the wc command so we have to write another command by using pipe let me write like this all right so now let us try to understand this command what exactly we are doing here is whatever output we are getting from this command i am writing the output into output.xd file i am saving the data in output.txt file by using redirection after saving the data in output.txt do we get any output for this there is no output right so after executing this butler command there is no output because whatever output we are getting from the first command we are writing or we are saving the data into output.txt that's it but we are not getting any output there so wc will not have any input now because this command pipe command or direction command what this exactly doing is it is basically uh breaking the piping concept that means whatever output you are getting here we just saved that output in output.txt that's it we are not getting any output out of this whole command because once you get the output from this whole command then we can pass that output as an input to the next wc command but here we are not getting any output so that wc doesn't have any input for that so that is the reason this command cannot achieve this requirement so the output of ls command should be saved in output.xd this will work but wc doesn't have any input for that so which will not give you wc command will not execute because wc command doesn't have any input for that okay let me try to execute and see what is an error we will get let me just copy it and connect to ubuntu just a moment just a moment let me connect to my ubuntu okay [Music] now it is connected so now let me execute this command so i say paste so ls iphone l slash etc these are for my first command and whatever output you are getting from this command will save into output.txt that's it so here it is get stopped so there is no output of this command right so if there is no output of this command we do not have an input for the next command right so let me execute this what happens it is 0 0 0 why it is 0 0 0 because after executing this command there is no input for the wc if there is no input means what no lines no words and no character so 0 0 0 is given so this is not fulfilled our requirement now okay so this is not fulfill our requirement now so let me just modify the case now just a moment okay so now let me just modify the command slightly so instead of doing that let me write something like this so here the problem is in the middle of piping okay so the here the problem is in the middle of piping if i use redirection then it breaks the piping because we are not getting any output from this command because once you get the output we can pass that as an input to the next command but here we do not get any output for this command so the piping the middle of the piping if i use redirection then it breaks the piping concept so then how we can achieve this so what is our requirement we whatever the output we are getting from this command we have to save the output in output.txt and also on the same output i want to execute wc command okay this is something called uh we have to pass the input for the two commands we have to pass the input to the two commands now let me show you again so here we are getting the output so ls output we are getting first we have to say the output in output.txt file output.txt file there is a first requirement and the same output we have to send to the wc command and if i just look at here there is something called it seems like a t all right it's a con something called t junction that means what whatever output we are getting from the ls we are sending the output to the output.txt and also same output we have to send to the another command so how we can achieve this so to achieve this kind of requirement we have to use something called t command because it seems t junction right so we have to use a t command so what the t command will do is whatever output you are getting from the ls we will save the output in the output.txt after that the same output we can send to the another command that can be achieved by using t command okay now let me show you how we can use a t command so the same command i can write like this by using a t t command which is very very useful command we can use along with the piping concept so we can use a d command so here we have to write command like this ls ls iphone l slash etc which will give you all the files and directories and what what's the first task we have to do we have to save that into output.txt okay so let's say use pipe here and here i'm using t command t output dot txt and then wc so let us try to understand this command ls iphone l slash etc this command will display all the files and directories and that output is become the input of the output.txt and wc so how t command works here is whatever output you are getting from this command the output first saved in output.txt okay so what t command will do is whatever output you are getting from the first command will be saved into output.txt after that the same t command will send the output of the first command or which will forward the output of the first command to the next command that is wc so that means what the first thing is which will the output of this command will be saved into output.txt file that is done by the t command now after that the same output of the first command will forward to the next command that is also done by the t command itself okay now whenever you find this kind of requirement we can use t command along with the pipe symbol so what the t command will do is whatever output you are getting from this command we can pass the same output as an input to the next commands or multiple commands and this basically t command will save the data in file after that it will forward the same output to the next command okay t command will save the data in first file save the data in a file that's the first activity after that it will forward the same data to the next command so that is a use of t command now let me just show you a few examples how we can use t command let me just execute the same command now go to the terminal clear clear i am just executing this command ls iphone l slash edc the output of this command will be saved in output.ext and the key command will forward the same output to the wc now execute now we got the output so 222 lines are there in the output and 2 0 0 5 number of words and this many number of characters we have in this in this output and the same data will be saved in output.txt also right this is output which is producing by wc command but what about output.exe the output of the first command will be saved in output.txt and then forward the same output to the wc so wc command will give some output here but what about output.txt let me print that so cat output.txt so now we got the data so this is the data which have copied into output.txt so this is a data which is copied in output.txt okay so this is how we can use t command okay along with pipe along with pipe let me show you one more example okay let me show you one more example here the same command i'm writing like this ls iphone slash edc pipe symbol t output.txt i'm just removing wc then what happens so this command will print all the files and directories and what t command will do which will write or save the result into output.txt after that do we have any command no command so just it will not return anything for us it will just write or save the data of the first command into this output.ht that's it let me just execute this command clear and before that i'm just removing output.txt okay i just removed now execute this command paste so ls iphone ls slash edc whatever output you are getting from this command will be saved into output.txt so t command will do that after saving it the t command will forward the output to the next command but do we have any next command here we don't have any next command so simply it is display the output here let me execute one more time so here we don't have any next command just a second let me write one second ls iphone el slash edc pipe t and output.txt okay so what exactly it will do is whatever output you are getting from this will be saved into output.ht after saving do we have any other command so after saving the command will forward the output to the next command but we don't have any next command here so whatever what happened to the forwarded output which will just display the so let me repeat once again the first command will repeat the files and directories list of files and directories and those files and directories will be saved in output.txt by using t command after saving it then it will forward the same output to the next command but next next command is not there so whatever what happened to the forwarded output then what will happen to the forwarded output there is no command here there is no next command so simply whatever output is forwarded by the t command will just display in the terminal window like this so now two things it has done one thing is it is just copy the output in output.txt and also display the output in the terminal window okay it is just copy the output to the output.txt and also display the output in the terminal window so two things we have achieved this so this is a one more example by using uh t command okay this is let me just say this is example six so the output of ls command should be saved to output.txt file and should be displayed so we don't have any further command so t command will just display the output in output.txt but there is no other command so it just display the output in the terminal so this is how we can use t command along with piping now let me show you one more requirement let us say example seven example seven so let us have this kind of requirement so tesla dot txt contains some file names and we have to read all the file names from this particular file and remove each file okay what is my requirement here is let me just draw something here so i have one small file let us say this is my test.txt file which contains let's say test1 and test 2 test 3 and so on so multiple file names are there multiple file names and also i have these files physically ok test 1 tester 2 test 3 i have created multiple files in my directory and i also have another file called let's say test.txt which contains multiple file names whatever files i already created in my directory the same file names are available in this particular file my requirement is i just want to read these file names from this particular file and whatever file name i'm reading from this file i just go and remove the particular file from the directory okay so let us say i have hundred files like this test one test two test three let's say test hundred but in this particular file i'll mention test one test two test three test four and test five then i want to read whatever file names i mentioned in this file i want to just read those file names accordingly i just remove the files physically from the drawing okay this is my requirement so let's see how we can achieve this requirement by using piping concept so let me repeat again the same requirement so i have a single file which contains only file names and these files also available in the same drive so whenever i execute one command single command i have to do all these things so whenever i execute a command which will read all the file names from the file accordingly which will remove the files from the same directory and whatever file names are mentioned in the file all the files will be removed like let's have 100 files now here mentioned only five file names and only those files should be removed from the directory okay this is my requirement now let us see how we can achieve this very very important so test.txt contains some file name let me just create some files here first of all clear let me just clear it so let me just create some files so how to create multiple files we can have a command called touch command by using this we can create multiple files at a time so let's say test our touch command test one test two test three and test four and test five so let's create five files so touch command will create multiple files at the same time and let me just list out here okay let's say ls iphone ls iphone el which will display all the files test one test to test the dashboard h5 so five files have created and apart from this i will also create one more file called uh let me just create test.txt file okay so let me just create i use get a recommend g edit test or i can say test.txt and inside this i'm just specifying all the files whatever i created so totally five files are created test one to test five so here i'll say test one test two and test three and then save the file now if i just look at this clear when i say ls i have test one file test two and test three four and five so totally five files are created now if i just look at this tests dot txt which contains a three file now my requirement is i want to read this data from the test.txt file and whatever file names i mentioned here i just want to remove those files from the directory let's say test123 i mentioned so 1 2 3 should be removed from the directory okay this is my requirement which we have to achieve by using piping concept so how we can do that let me just write a few commands here let me just first i want to get the files from this particular file so how we will get by using cad command so cat test dot x let me write commands like this so let me write like this so first command i'm writing like this cat test.txt so what this will return basically which will return standard output right so written standard output in the terminal so what is the standard output which will basically display the files so whatever content we have in the file will be displayed okay this display the standard output now if you want to remove all the files so read file names from the test.txt and remove each file so how many files are mentioned in the test.txt three files test one test to test three i want to remove them so to remove the files what is the command we will write rm test1 to test three so like this we can write so what this command will do which will remove this particular command will remove three files right so these are the two commands we can use now i have to combine this the first command is just will display the content of the file so just display the whatever file names we have in this test.txt so what it contains now it contains a file one which contains a test1 file test one test two test three right these are the file names we mentioned in the test.ext and after reading these file names i just use remove command and remove command by using remove command i just want to delete these files i want to delete these files so how we can achieve this and when i use piping concept observe this very carefully when i use piping concept i'll say command one pipeline command two like this we use right normally what happens the output of the first command will become the input for the next command then we will get the final output then we'll get the final output so this is the concept of piping right now when the second command will take the input from the first command so the first command will return some output and that output will become the input of the next command that is command one then finally we will get the output so in the piping concept if the first command if the first command returns a stream stream is nothing but it is not it is not standard output it should not be any standard output so but here if i just look at here test.txt what this command will do which will return a standard output standard output in the terminal as a test one test two test three but what about the rm rm needs some arguments like this rm command will need some arguments so when i write the command like this i'll say cad tester tester.txt pipe rm when i like when i write like this what happens is the first command will display the file names which are present in this particular file but what is our intention here rm command should remove the files whatever file names we mentioned in the first file should be removed by rm but if i just look at this command the first command will display the output in the standard device that is terminal let's say file one or test one test to test three but rm requires parameters or arguments so in this context in this context rm cannot read the data from this output from the first command output why because the first command will provide the output in the form of standard output so test one test two test three but rm requires parameters rm requires parameters okay if i just look at our previous examples the first command display some data and that data will be this command will be applied on that particular data but the wc iphone l doesn't require any arguments here so whatever output you are getting from this the second command will execute on top of the output okay but wc command doesn't require any parameter here but whatever output is retaining from the first command will become the input of the second second command it doesn't require any arguments only the output of the first command will become an input of the second command but in this case in this particular case the output of the first command cannot become the input of the second command because the output of the first command is just print the standard output but rm needs some arguments if i just look at this command rm requires some parameters or arguments but here from where rm get the parameters but it is just giving the standard output let's say test one test two test three but i want to pass these three as an arguments to the rm and then only it will work then only it will work okay so suppose when i say rm test one test two test three suppose if i have a three files i mentioned like this if you have more number of names in the test.txt there are so many names are there how we can achieve this so rm required the parameters but here the first command just display the output from that output test1 test2test3 we have to pass these three file names as an input as an argument to the rf so to achieve this kind of requirement we have a special command called x so we have a special command called x args let me just explain this so by using this command we can achieve this so whatever output you are getting from the first command if you want to pass that output as an arguments of the second command then we use xr command so let me just write this command like this i say cat test.txt okay and here i'll say x args xrs rm okay exalts rm so now how this command works is so the first command will display the standard output test one test three test three but rm cannot rm command cannot take the standard output it reads it needs some kind of a stream so whatever standard output we are getting from the first command will be converted into stream will be converted as an arguments by using x arcs command so x asks command will convert the output of the first command as an argument of the second command okay the output of the first command will become the arguments of the second command that is rf because rm is required arguments rm requires some arguments and those arguments are the basically input of the first command so when i execute this command this will fulfill the requirement so test.txt contains some file names and read file names from test.txt and remove the corresponding files let me just execute this command let me just execute this command let me open my ubuntu yes now clear this i'm executing this command so cat test dot txt which contains only three files i mentioned test one test two test three okay let me just print once again so i'll say cat test.txt we have only test one test to test three but in ls we have five files test one test two test three four and five i want to just remove test one test to test three because i have only three file names in the test.txt so i can write command like this i can say first we need to get the file name from the file i'll say cat tests dot txt so this will give you a file names after that i use pipe symbol then i'm using x args command so what this command will do whatever output you are getting from the first command will pass that output as an arguments of the second command what i am saying why i am saying arguments here because rm require the arguments here and whichever file name is specified that file will be removed so whatever output we are getting from the first file whatever file names we mentioned in the test.txt will be sending as an arguments to the second command that is rm now when i execute this command which is done now when i execute ls command you don't see a test one test to test three so you can see test four and test five but you cannot find test one test two test three because those three files have already removed by this command okay so this is a very very important scenario so x are also a command which we can use along with the piping concept along with the piping concept so xr these are another command which we can use along with the piping concept okay so whatever output you are getting from the first command if you want to pass that output as arguments to the second command then we can use xrs basically this will convert the standard output to the screen because the rm command requires only stream so pipe works only with the streams not the standard output or standard input okay now let me just show you one more example then you can understand this so i want to example seven so i'm just going to show you one more example let's say example eight so display the output of the date command by using ego command with the piping concept so normally how we will display the date we say date command so we'll say date command will display the date but i want to display the date by using ego command okay what echo will do is when i say echo in the double quotation whatever we see whatever print okay that will printed in the terminal window so this is a purpose of equal command but my requirement is i want to display the output of the date command in terminal by using equal command so what i have to do we have to send the output of the date command as an input to the ego command okay as an input to the eq command so what does it mean when execute the date which will just display the standard output but eco command is required argument so eco command is required some arguments here the message itself is an argument so how we can pass the output of the first command as an argument of the second command we can write like this a date pipe right xargs eco okay now what happens the date command will display the date that is a standard output and if you want to pass that output as an argument for the next command that is eco then we can use a rx because ego required argument equal required argument just like a remove so rm is also required argument here okay so that's the reason we are using ar so what our output we are getting from the first command we are passing that output as arguments of the second command similarly here whatever output you are getting from the date we are passing that output as a arguments of the second command which is eco so when i execute this command which will display the current date when i paste here which will display so whatever output we are getting from the date we are passing that output as an argument of eco command okay argument of equal command so this will display right so this is how we can use x arc t and xor so three things we have learned so what is piping so basically piping means the output of the one command become input of the next command which is called as a piping and in the piping we can execute multiple commands together and we use vertical bar to achieve piping concept and sometimes if you want to pass the output or if you want to forward the output to the next commands then we can use a t option or t command and what t command will do is first the output will be saved in the file then it will forward the output to the next commands in that case we use t command along with the pipe there is one more command called xrs sometimes few number of commands will display the output but i want to use that output as an argument of the second command arguments of the second command and that case we use xrs option this is a command not an option so whatever output we are getting from the first command we are passing that output as a argument of the second command okay so in that case we use x arcs so these are our very very important options which we have in the piping concept okay so that's all for this video guys thanks for watching

Channel: SDET- QA Automation Techie

Views: 6,170

Keywords: Python, Selenium with Python, Selenium webdriver, Automation, Testing tools, Selenium tutorials, Python tutorials, Manual testing, SDET, Java, selenium with java, big data, hadoop, web services, postman, soapui, rest assured, big data testing, hadoop testing, ETL testing, SQL, Programming, Software Testing material, selenium videos, python videos, automation vidios, jira, jmeter, performance testing, functional testing, QA, QC, Agile testing, SDLC, Automation Step by Step, restapi

Id: Z1C4Qo0Q1G4

Length: 46min 30sec (2790 seconds)

Published: Thu Dec 10 2020