Using `sed` and Regular Expressions (Unix/Linux command line)

in this video we're gonna talk about how to do some basic operations on the command line with said and using regular expressions this is sort of at the request of a lot of people recently just because you know I do all these scripts using all these I guess UNIX command line stuff that I guess people have sometimes you know a piecemeal understanding of but it's very useful to at least get into real-life implementations things that I've used to understand how to use things like said and grep and aaalac and stuff like that but in this video I'm gonna talk about some of the main uses that you can use said for replacing stuff deleting stuff printing stuff and using again a real life implementation that I've had and also how to I guess get a lot of magic out of it but anyway so we'll go ahead and get into that and as I said this will involve some regular expressions if you don't know anything about regular expressions you'll be fine though you'll figure it out you might even learn something so anyway at a basic level I'm gonna open up a file and this the file origin is not particularly important but if you watch my channel you might know that I auto generate shortcuts from a file that looks like this where I have a directory or some herb here it's a different kind of file on the right and then some shortcut to it on the left and I have a little script that reads a file that looks like this and generates aliases for bash or something like that or Ranger and other stuff but when people started using my dot files they started saying hey it'd be really nice if I could you know say like you know put in a comment or something like this so I'm putting in a comment or have some you know blank lines you know blank lines and comments because originally when I made the script that parsed this file they it only really just assumed that it was already in the correct format and there were no comments for it to ignore or not you know pay attention to or something like that so I want in this video to you said what I want to do is as I said the script that I have is going to read a file that looks like but I want to give it an input that looks like this that is with comments and blank lines and I want to have a set said script actually not a said script one single said command one line of said that will change a commented file with blank lines and all this stuff into an uncommented file that my script can use okay and it's gonna be very easy but it involves learning some of the basics of said which again are very easy now let's go ahead and get into this so at a basic level I am going to run set on this file be m-files now you'll notice here I just gave said in the quotation marks those are the commands that you're running with said in this case I have nothing and I ran it on be m-files so this is actually just gonna print out the file it's gonna show you what it looks like that's all that's all now you can put in the quotation marks one of SEDs many commands and one of the most common ones that we can use is s S stands for substitute it's one of the most commonly used of all of the different you know of all the different set of commands but we'll go into the syntax of it it's very simple so as an example you know let's say so what substitute does as you can probably guess is you find some pattern let's say they're a bunch of C's in this document lowercase C's let's say I want to replace lowercase C's with capital C's the syntax of this is gonna look something like this so you have s for substitute then a slash then the pattern you want to replace then another slash then the pattern you're replacing it with then another slash and at the end I'm gonna put a G here this is for other options about the command I'm just gonna run this and I'll tell you what G does in a second so if I run this you'll see that all of our lowercase C's have now been replaced with uppercase C's so that's what we want in our hypothetical said replace command and this is the basic syntax for it it's pretty simple of course you don't just have to replace single characters you can replace lots of stuff you can replace lowercase e with LMAO or you can replace CF with LMAO or anything like that now let's let me now what is the G for no I should be clear let's say let's say I'm just replacing see here now at a basic if we run it with G as an option you'll notice that for example on this line it replaced cfb bash RC with LM a two instances of a LeMay oh if I run it without G you'll see that what that does is by default that is if you don't have G said is only gonna run the substitute command once per line so our first see on this line was replaced but not our second one if you want all of the instances to be replaced you gotta run it with G so I pretty much always run my said replace commands with G I mean it's probably minutely fat I mean if you know that you're only going to be making one modification per line of course don't use G and in fact it might be faster not to but I find that most of my said commands end up using G but or replace substitute commands so that's that's how we can replace a sequence so what does this have to do with getting rid of comments well it's very easy what we can do is we can replace the pattern of a comment with nothing you can very easily you know just have nothing as your replacement pattern you'll see all the C's have now been deleted from the file so what we want to do is instead of replacing C we're gonna replace what a comment looks like so comments pretty much always look like this they're gonna start with a pound sign and then we're gonna use a little regular expression magic and that is we're gonna type in period and then cleany star so if you don't know what this means it's very simple period just stands for any character and the star stands for whatever the last thing you just typed just say any number of those okay so what this is going to do is actually gonna replace a pound sign and then basically every character that comes after it in the line that's what it's gonna do so if we run that you'll now see that our comments have disappeared we have no comments anymore this one is totally gone it's been replaced with nothing all the ones down here are totally gone you can see that all the comments have now disappeared from our five now I should be clear which said is a stream editor by default we are not modify the file we are just taking the file as an input and then printing out some output if hypothetically we want to you know change the original file we could run said with the I option but we're not going to do this in this video that because that's not you know it's not what we're doing but just know that's how you do it so don't be afraid if you're not running you know something are you exceed if you're running said without I it's pretty much safe to run whatever don't worry about it but um okay so I do want to say there's one additional thing we want to add here and that is there are a couple lines if you notice so we've gotten rid of this comment but technically speaking you can't see it but there is a space right before this pound sign now what we can do to ensure that we get rid of all those spaces is we can do the following I'm gonna add to this replacement expression slash s and then another clean e star now what slash s means is any whitespace character and again the star means any number of those of those characters so that means it could mean zero spaces it'll match for zero spaces it'll also match for 100 spaces anyway if we run this it's gonna look exactly the same as it was before but technically it's a little better to do this because if we run it without the this there are technically some trailing whitespace now that might not matter for a lot of the script you're making but I guess it soothes my autism or you know it's more into it it it just makes more sense I know what I'm dealing with now okay so that that's half of the way there because I wanted to get rid of comments but I also want to get rid of these nasty blank lines in fact let's let's add some more blank lines in because why not make it more visible okay so we're gonna so now we have some eight extra blank lines so what can we do to get rid of these blank lines now I will say in addition to this little command here we can also concatenate different said commands so I can do if I want to get rid of comments but I also want to replace lowercase C with uppercase C I can do the following after this said command but still in the quotation marks I can type in city : and then another replace command to replace lowercase C with uppercase C right and I can run that and you'll see that all of the them have now been replaced okay now I don't wanna to get rid of the blank spaces I'm not gonna be running a replace command or a substitute command instead said has another so said has a way that says general syntax is give me a pattern and I will look for that pattern in a line and then I'll perform an action on it so let me give you an example let's say how that looks like is so our new command I'm going to have a command that looks like this and then put a D at the end now what this means is I want you to find wherever C and F appear so CNF appear you know in multiple on multiple lines go to all the lines where see an effort there together and then perform this operation D you can probably guess D means delete so if I run this you'll see all of the lines with CF had now been deleted okay now we can do that as well we can use a regular expression well actually before we even give the final answer just to be clear a set has a bunch of different commands that you can run with this so for example if I run CF instead of D if I run P that printed out a whole bunch of stuff what that did is whenever it fine finds CF in the file it runs P meaning print so you actually see it run so it prints out the whole document but every time it finds a CF it actually prints that line out twice because we're telling it to to print that out or we could say CF and Q then that means when you find C the first CF you fine quit the file etc or we were talking about the other day right so said 11 Q just means when you get to the 11th line if it's not in slashes or if it's just the number non slashes that's assumed to be the line number but said 11 Q just means when you find the 11th line quit okay so anyway so what we want to do again we know how to delete patterns the pattern we really want to delete is the blank line now of course there is no symbol for blankness but there is we can do the following we can do carrot thingy Patrick whatever whatever you call it plus dollar sign now if you don't know anything about regular expressions I will tell you what this means the hatch ik Caron whatever it is that stands for the beginning of a line so that is gonna match every beginning of the line and the dollar sign is going to match all the ends of lines so when we put these two things together that basically means every blank line because a blank line is just the beginning of a line followed by an end of a line and we're saying whenever you find that on the line delete it we don't want it so if I are on this BAM that is what we want look how beautiful that looks so now we with one again with one little set command which is technically two sub commands but it's one set command we can take an input file that might have a whole bunch of comments people can comment however they want on this file we will take that and it will automatically turn into this much more parsable document so this is actually what I do now again as I mentioned this is a real-life implementation just a little said command I added to get rid of comments okay very simple now just to give you some ideas for other things you can do and said I mean this is all the technical portion of the video is now over but just to give you another idea of the kind of things you can do with said one thing I have is I'm going to turn on my transmission daemon one thing I have up here is I have this little module that I guess I've talked about in other videos a little torrent module and it shows little emojis based on how many of my torrents are done or downloading or waiting for seeders or seeding or something like that now that might sound like you know you have to download a 5,000 Meg's of Python modules or something if you don't know what you're doing but really to get something like that you just need a kind of a said command so I'm gonna actually go to the script that runs this so I'm gonna open up my this this particular file and I will show you now what this module is in effect is the input to it is the transmission remote list command which what that does is it just prints out all the torrents you have and it gives a bunch of information about them like how finished downloading they are and stuff and really to get this little module up here all I do is I pipe it in the said and I have a bunch of replace commands so for example really in effect what it's going to do is whenever you know it finds that a torrent is seeding because it says seeding it's going to replace that with a little sapling up here right so this thing means seeding now notice that I actually have two said commands that's just because first I replace them with letters so I can run a sort on them to get them in the order I want then I'll replace those letters with the actual emojis but in essence it's it's pretty simply just me replacing it's me just taking text input and replacing all of that with some kind of more something more visually I guess intuitive for a status bar icon or something like that so that that's all this thing up here it's just a little shell script line well this I guess it could technically be a line just a lot a line of shell script that is really just said replace command commands I'm just replacing sequences with other sequences so anyway that's about it for this video I hope that you've learned a little bit about said if you didn't know anything about it or even if you did know something about it I hope you learned something if you want more videos like this just feel free to say so because you know a lot of people have asked me to do I guess more general scripting videos and hopefully this has been a good example so anyway that's that's about it see you guys next time [Music]

Channel: Luke Smith

Views: 128,326

Keywords: gnu, linux, tutorial, introduction, unix, philosophy, minimal, minimalism, minimalst, computer, programming, program, i3, i3-gaps, vim, emacs, vi, tex, latex, markdown, git, github, groff, arch, gentoo, distro, distribution, distrobution, ubuntu, fedora, thinkpad, floss, free, open, source, foss, software, sed, awk, core, grep, replace, substitute, delete, line, pattern, regular expression, regex, regexes, kleene, star, pipe, technology, computers, virus, command, frog pepe

Id: QaGhpqRll_k

Length: 14min 56sec (896 seconds)

Published: Sat Dec 15 2018