Advanced Bash Scripting Tutorial

Asch provides some handy shortcuts called expansions and you may already be familiar with one of them that's called if you've ever used the tilde character to represent your home directory you've used tilde expansion the tilde character represents the value of the user's home variable it tilde followed by a - or - represents the bash variable called old PWD which is the directory that you were just in if you've recently changed directories there's some other features as well and I encourage you to check out the documentation for till the expansion to learn more I'll step into the fruit directory here another handy type of expansion is called brace expansion this is written with braces around an expression and it can help with repeated commands with different terms or interpolation within a range let's say I wanted to create a bunch of files but I didn't want to type each command to do so with brace expansion I can type touch apple banana sherry durian and I can see that that created for files of course I could do that with touch just by itself but I can use interpolation for things that I really don't want to type out each time let's say I want to create a thousand files it would be a little tedious to touch each of those individually so I can type touch and then the beginning of the file name file underscore just can be whatever you want one dot dot one thousand and close the brace and then when I list the directory there's a thousand new files now at the upper limit of this range it seems to vary by platform on my Mac I can get up to about fourteen thousand three hundred and forty my lytx machine here it goes much higher than that but that's not really the point bash version 4 and the ability to pad these series of numbers with zero normally if I ask for the numbers one through ten I'd get this but if I started naming files like this they wouldn't sort the way you might want as you saw before for example file 428 file 429 file 43 file 430 of course that's not an incorrect sort there's a logic to it but like I said it may not be what you expect I can fix that with zero padding but first let me clear out all these files and I'll create new files notice the zero in front of the one it doesn't matter how many places you need to pad one zero is all it takes so now the files sort the way I'd expect and they all have the same filename length as of bash for you can also specify an interval I'll clear the screen here and then type echo and curly brace one 10.2 in this case I'll get the numbers between one and ten counting by twos or threes or any absolute value integer you might need brace expansion works with letters as well as you can see the capital letters come before the lowercase letters the couple symbols in between and like I mentioned with bash 4 and above you can specify an interval with letters as well Oh brace expansion will interpret the range that you give it so as an example I said W dot D which is not only backwards but it's also a sub range of the alphabet if you do use this in your scripts make sure to keep in mind that your target audience may not have this feature available if they're on certain platforms and I can chain together these expressions as well I'll get rid of all of those files I created just a minute ago and type clear to clear up the screen and then I'll create a series of files aimed with the fruit underscore a number underscore and a lowercase letter and suddenly I have thousands of files in fact I can take the output of LS - 1 and piped into WC - L which is a word count with the L flag giving me the line count and I can see I have 8,000 files obviously this has applications beyond file creation and it's important to be able to recognize when it's being used and when to use it a lot of what you'll be doing with scripts involves getting something to go somewhere else Bosch has a couple ways of doing this first let's look at pipes piping takes the result of one command and sends or pipes it into another command for example take a look at the result of listing the contents of this directory which has a lot of things on it from the previous movie that's a bunch of stuff it would be nice to take that output and display it page by page I can do this by piping the output of LS into the more command and if you're looking for the pipe character it's usually shift backslash the key for rich on most keyboards is right above the return key but your keyboard might vary and now I can take advantage of Morris pagination redirection on the other hand works with a standard input standard output and standard error that is the input from the command line environment the output to it and the errors that arise when something goes wrong let's set up a little bit of an example here first I'll make another folder to copy to and I'll make it so that I don't have access to read some of the files in this current folder this will cause some errors later on and that's what I want so I'll copy these files into the other folder the V here tells the CP command to be verbose so we see each copy operation and I'm specifying star to use file name expansion to match all of the files then I'm using the two dots and a slash in the destination path to indicate that other folder is a child of this folders parent folder that is the CP command we'll have to go up a folder level to find other folder and if I scroll up here there's some errors I couldn't copy some files the ones named with 0 1 5 which you remember I blocked everyone from reading and the other files copied just fine so I'll clear the screen I'll get rid of everything in that other folder I'll write another command here and I'll explain it in just a minute these numbers one and two represent the standard output and standard error respectively the greater than symbol represents redirecting the output from each of those somewhere else so the successes will go to a file called success not txt and the errors will go into a file called error dot txt I'm putting them one level up so if I want to run this command again I won't be copying my success and error files into the other folder and that time there was no feedback that's because I redirected both regular messages that would normally appear at the standard output and the errors to text files instead this can be pretty handy if you want to capture the outputs of commands that you're running let's take a look at these files I'll type cat dot dot slash success dot txt and then cat dot dot slash error dot txt cool I'll clear out that folder again you can also tell bash to redirect standard output and standard error to the same place here using the ampersand to represent both 1 & 2 and I can see that everything is there you can also redirect to a special location called they have null if you just want to get rid of and ignore the output from a command for example I could redirect the output of LS to Devon all and the results just go off into the great blue nowhere never to be seen again there's a handy tool called grep which lets you search files for specific patterns of text this can come in handy when you're searching through log files or if you just want to get one piece of output from a command that returns a lot of information of course there's way way too much cool stuff that grep does for me to cover in this movie but I'll show you a couple examples if you want to learn more about what grep can do check out its man page with a command man grab so first let's take a look at a log file I've got a log file here called auth dot log it's a modified version of an authentication log from a computer that was left exposed to the Internet for a few days it's got a whole bunch of failed login attempts from different addresses and it's over 18,000 lines long not something you'd want to have to scroll through by hand now of course I've changed the addresses to example values so the IPS and servers you see don't exist on the Internet so I could use nano or cat to see the contents but I want to search for my username so I'll clear the screen here so I'll use grep followed by the search term in this case my username followed by the name of the file that I want to work with auth dot log notice how the search term is highlighted you may not see this on your system I like to enable the color output because it makes things stand out a bit more you can use the color option that's - - color equals auto on individual lines or you can type export grep underscore options all caps equals apostrophe - - color equals Auto Tune able it all the time so let's clear the screen and get back to the log now look for access attempts that the system thinks is a break in here I'm using - I to make my search case insensitive and there's all the break-in attempts I can pipe this output into a command called awk to extract just a list of specific things in my case I want a list of IP addresses but this print statement does is to count the 12th thing that it comes across line by line space - limited and only return that value so I have print dollar sign 12 which will return 1 2 3 4 5 6 7 8 9 10 11 12 the 12th thing right there CIP address in brackets awk is powerful and there's another program called said sed that you'll see used all over the place for string manipulation within bash scripts I encourage you to explore them and pages for them both with man awk and man said to become familiar with them but since they're beyond the scope of bash itself I won't cover them in depth here so ok that's using grep directly on a file but what about using it to pull specific information out of the output of a command let's take a look at that using the ping command now you're probably familiar with ping it's really handy for checking if you have network connectivity and for getting an idea of the response time for a particular server or device but sometimes I just want one ping to check something really quickly rather than letting the command run until I stop it with ctrl C of course ping has an option for this the - C flag to set the count of the pings that it sends but it still outputs a bunch of other information and all I'm interested in is the response time so to get that I can pipe the output of ping into grep in this case I'll look for bytes from all right this is getting closer I just have one line so to get where I want I'll pipe the result of grep into a command called cut which lets you slice up lines based on particular criteria for this I'll use the D flag which lets me specify a delimiter in this case I'll use the equal sign and I'll follow that up with - f4 because I want field four similar to how awkward this value will go along the line that I give it and look for the delimiter which is equals and then it'll find the fourth thing that's delimited with equals this here is all one thing then we have a delimiter this is a second thing delimiter this is a third thing delimiter and then the fourth thing and there it is now of course this is a pretty long and unwieldy command to use all the time and you're probably seeing that it would be really handy to be able to change certain values within that command programmatically and that's exactly where we're headed so in a later movie I'll show you how to wrap it up in a script to make it more convenient and versatile while Bosch refers to the whole interactive shell most of the time we just issue single commands LS c d RM grep and things like that a bash script can contain many of these commands and often makes use of variables arguments and procedural flow control logic of course you can enter these things line by line at the command prompt but the whole point of having scripts is that you don't have to do that your script will start with an interpreter directive that sometimes called a hash bang or shebang that nickname comes from the first two characters of the line a number or pound sign or hash sign and an exclamation mark which is sometimes called bang that's followed by the path to the bash executable and this can vary on different systems but usually it'll be at vin bash the hash or number sign is the comments signifier in bash if you put it at the beginning of a line the interpreter ignores the text on that line together with the exclamation mark and a path to the bash binary the whole line tells the shell that this is a bash script and should be run as such this is especially important when you mark your script as executable which we'll take a look at in more detail later after the first line we get into the rest of the script bash can be picky about white space within expressions but it doesn't mind white space in indentations and between lines so now we're ready to create a script frequently if you're creating or editing a bash script you'll be doing so in a terminal window I'm using the Nano editor here on my system and I'll create a file by typing Nano my SH the dot SH on the end signifies that it's a shell script and of course we'll want to put the shebang line in first next I'll add a comment to describe what the script does it's a good practice to document things as you write them so you're not faced with coming back months or years later and trying to figure out what a script is for at this point we're ready to write something that does something and as it turns out we already know a whole bunch of bash commands let's just use a basic one for now that's just the command to list the current directory so we know what to expect out of the script I'll save the file with ctrl o and then exit which is control X back at the command line if i type LS i can see that i've got the script here but I can't just type the name of the script to run it I have to use an interpreter for that so I can type bash my dot s H and we see the contents of this folder just like we'd expect but most of the time you won't want to type out bash in front of your script so let's take advantage of that shebang line which tells the system where the bash interpreter is I'll type chmod plus x minus H to make the script executable and now I can just run it by typing dot slash my Sh we need the dot slash since the current working directory isn't part of the path environment variable if we put the script somewhere that is like user bin then I can run it just by typing the name but we wouldn't really want to do that in practice and that's getting a little ahead of ourselves of course now I can do Nano my dot Sh and jump back into the script and make changes if I need to but for now let's leave it as is you one of the most basic commands that you'll use in a batch script is the echo command echo prints out information normally to the standard output but it can't be directed elsewhere this syntax for the echo command is pretty simple you just type echo followed by whatever you want to output however there are a couple different things you should keep in mind about quote marks there's three conditions to be aware of here no quotes single quotes and double quotes here's a simple example don't worry about the greeting stuff for now that's a variable and we'll dig into those soon if you write a statement with no quotes bash goes along and interprets things as it finds them I got an error here because if you need to use special characters you have to escape them with a backslash there's also single quotes or strong quotes where nothing inside of the quotes gets interpreted so everything comes out literally even if I put a variable inside I get back literally what I typed not the variable and finally there's a middle way using double quotes I'll turn these all on so we can see the differences of course if you wanted to have batch displays something literally inside of double quotes you could escape it with a backslash if you want to add empty lines to your output just type echo with nothing after it I'll be using echo throughout the rest of the course for now let's move on to using variables as with any language you'll probably find yourself needing to work with variables in your scripts they're named with a sequence of alphanumeric characters and the names must start with a letter to declare a variable I'll type its name directly followed with an equal sign directly followed with a value note that there's no spaces if you put spaces around the equal sign they'll get an error when you run the script likewise if your variable is a string with spaces in it such as good morning you'll need to put that string in quotes in the variable declaration of course a variable could also be a number so to use these variables we call them with a dollar sign in front of their name so I'll save that and I'll run it at the command line and we can see the values printed out there these variables also work inside of other variables or strings if they're double quoted and again I'll save and run and here I have a string that has variables included in it of course that can be very handy for generating output to the standardout for a string using a file name or something like that or to a log file you can give variables special attributes if you like using the declare keyword declare I in front of the variable marks that variable as an integer declare our before a variable marks it as a read-only that means that it can't be modified later arithmetic Lior with string manipulation you can also convert arbitrary strings to lowercase or uppercase with - l and - u respectively there's some special variables to be aware of - listed a few of them here with the output from both my Mac and my Linux machine there's home which returns the users home directory PWD which returns the current directory just like the lowercase PWD command does at the batch command line mach type which returns the machine type which could come in handy for determining file locations if you're writing a script to work on different platforms there's host name which returns the system name - underscore version which returns the version of bash running on the box and seconds which returns the number of seconds a bash session has been running if you type echo seconds at the command-line you'll see how long that said it's been open but if you use seconds in a script it'll start counting from when the script started that can be handy for timing things another variable dollar sign 0 contains the name of the script there's a bunch more - and you can explore them here variables are everywhere in scripts so it's important to have a good handle on them chances are you'll probably want to get some information back from command Z right in a batch script instead of just running them luckily it's very easy with command substitution that's accomplished using a dollar sign with parentheses around a command in this case PWD which prints the working directory what this does is run the command inside the parentheses and assigns the result to the variable specified in this case D without the command substitution stuff around PWD the value would just be PWD I'll put that back and I'll save the script and then I'll run it so that's a pretty basic example but let's take a look at another one at some point in your script writing adventures you'll once I use the value of some complex command as an example let's take a look at finding out how long it takes to get a response from a server you may recognize this command from an earlier movie what matters here is that I'm setting the variable a to the result of this command-line statement so if I run this command I see the result but now it's available as a variable within my script that I can use for other operations working with numbers in bash is pretty straightforward to tell the interpreter that you're going to do math you need to wrap an expression up in double parenthesis if you're assigning the result of an arithmetic expression to a variable the parens need to start with the dollar sign the expression inside the parentheses can use either literal numbers or variables bash supports six basic arithmetic operators exponentiation with double asterisks multiplication with a single asterisk division with a forward slash modulo with a % addition with a plus and subtraction with a minus I'll define a variable D to equal to and then I'll define E to equal the result of D plus two then I'll echo out the value of E and I'll save and then I'll run my script and I see four two plus two we can also use the increment and decrement operators that you might be familiar with and the combination assignments ie plus equals five would increment the value of e by five e x equals 3 y / e equals 3 and E - equals five one thing to watch out for if you don't have the double parens around an expression you might end up with a string concatenation if you're using plus equals assignment so if I take the parens off of the e plus equals five now it treated that as a string four plus five is no longer nine but forty five and that's not something that I want there's something else to be aware of with numbers and bash if I write F equals one over three and then I run it I get zero that's because bash math only works with integers not floating-point numbers if you need to get a floating-point value you can use the BC program with a predefined math routines to return that here I'm using command substitution to echo one over three piped into ECL and then if I run that I get the result that I'm looking for you can learn more about the bc command by typing man bc at the command prompt but suffice to say if you're doing a lot of math you probably want to use something other than bash for it what are the primary reasons you might want to write a script rather than execute commands line by line is to incorporate some logic into what you're doing for this comparisons can be very useful we do this with double square brackets a notation borrowed from kornshell it's important to keep spaces between the sets of brackets and the expression this expression returns 1 or 0 for failure or success bash supports the standard complement of comparators less than greater than less than or equal to greater than or equal to equal and not equal these work within the double square brackets to compare strings let's take a look at how these work I'll test to see if one string is equal to another and then echo the return value with dollar sign question mark in this test context you could use a single or a double equal sign to test for equality I'll save that and then I'll run it and I see the first test for a cat equals cat returns zero for success and the next one cat equals a dog returns 1 or failure I'll jump back into my script and I'll add another test asking if 20 is greater than 100 now if I run that I get back zero success what's going on here well the square brackets are comparing the two values 20 and 100 as strings and 2 0 is a higher value lexically than 1 0 0 if I want to test them arithmetic alee I have to use a set of operators a tell bash to compare the terms as numbers instead for working with integers we've got the same operations but the operators are different a dash and a few letters - LT for less than - GT for greater than - le for less than or equal to - ge for greater than or equal to - EQ for equality and - NE for not equal given the difference from other languages this can take quite a bit of getting used to and they're changing the operator I see what I expect there are also logic operators available in bash and with two ampersands or with two pipe characters and finally not with a single exclamation mark there's also two operators for testing whether a string is null or not null - is e + - n let's take a look at that here I'm using the logical and operator and using it to ask whether both a is no and B is not null I'll set a to be an empty string and I'll set B to be the shorts drink then I'll type a test expression and echo out dollar sign question mark for the return value I'll save and I'll run it and I get 0 success that's the result of the logical and operator right here the two ampersands and I'm using that to ask whether both a is null and B is not null one last note if you're finding examples online or working with older scripts you might see a notation with single square brackets the square bracket is the test command which was supplemented in batch 2.2 with a double bracket extended test keyword so now we've got tests that return a value in the next chapter we'll put that value to use in conditional statements Bosch provides for string manipulations like concatenation substring extraction and replacement probably the simplest string manipulation you'll use is concatenation sticking one string on to the end of another and it's really straightforward for this movie I'll just use the command line instead of a script file I'll set a equals to hello and be equal to world and then to create a string out of both of these put together or concatenate it together I'll define the variable C equal to a and B just right next to each other and if I echo that you can see I have my concatenated string you can find out how long a string is using a pound or hash sign in front of the variable name here I'm using parameter expansion and you can see when I run that I get the length of each of those strings a and C five and ten characters respectively you can also ask to just have a certain piece or a substring from an existing string and for that you'll use a colon followed by an index number to start from keep in mind that the first character is at index to 0 so I'll define the variable D equal to C colon 3 then echo out D and you can see here I have the substring starting at the third character I can also ask for a specific number of characters after that position in this case starting at character 3 and asking for four characters after that and I get back oh oh and if I use a negative number I can count from the end of the string instead of the beginning here I'm getting four characters at the end of the string note that you need to put a space before the - or it won't work and here's the first three letters of the last four letters a chunk of three letters starting from four in from the end of the string you might use this to extract a date code from a file name for example if you know exactly where that date appears in the string you might also find yourself needing to replace text in a string with some other text there's a couple ways to go about this first I'll define a string here I have the name of the variable I want to work with in this case the string fruit followed by one slash the term I want to replace in this case banana another slash and the string I want to replace the first instance of the search term with in this case durian and when I run it I can see that it replaced the first instance of banana even though there's two so what if I want to replace both of those well that's just a minor change and instead of having one slash right before my search term I put in two and it replaces all of the instances of that string there's a couple modifiers you should know about too if you use a number or pound sign in front of the search term in a single replacement it'll replace the term only if it's the very beginning of the string in this case it were played at the very beginning of the string if I try something that's not like banana it leaves my string completely unchanged if you use a percent sign instead it will replace the term only if it's at the end of the string as you can see here and if I use it on a term that's not at the end of the string again it leaves the string unchanged of course you can use matching with these two you can see how powerful this is and you'll probably use it in your scripts quite a bit you can style text in bash to draw attention to specific things or just to make your output a little more interesting you can do this two ways the first way is with ansi escape codes for this i'll use the echo command with a dash lowercase e option to enable escape sequences the escape sequences I'll be using look like this cool but what the heck's going on here let's take a look at this in a little bit more detail the first part echo - eat like I mentioned earlier tells echo that I'll be using an escaped string which are special codes following a backslash or escape character that lets me do the second part the escape sequence which is backslash 0 3 3 and a bracket to start followed by a couple numbers with a semicolon between them and that's all followed with a lowercase M to indicate the end of the escape sequence the numbers represent various formatting options I'll talk about those in just a minute then following that I have the string that I'm going to print out in this case color text then I have another escape string bracket 0 M I'll talk about that a little bit too okay so the numbers I mentioned correspond to certain styling options in this case foreground and background colors here's a table of some common options in this example I'm using colors you've got a range of 8 colors to work with black red green yellow blue magenta cyan and white which you can see isn't as bright as you might expect these numbers correspond to the color used as the foreground or background color for example a green is 32 in the foreground and 42 in the background and here's some examples of what the results could look like as you can see in each case I have two numbers separated by semicolon to set the foreground and background color there's also the ability to change the style of the text using different numbers for example bold is one and blinking is five up at the top here notice that zero is no style I used that in the example earlier to end the color selection if I hadn't anything that followed that string would have been styled as well so let's say I wanted to write out an error message I'll use echo - lowercase EE and then the slash 0 3 3 escape code followed with a bracket and five for blinking 31 for red foreground and 40 for a black background then I'll end that string with an M and I'll write error followed by a colon so that error text is going to blink but I don't want the message that I'm going to write to follow it to also be blinking so I'll use an escape code again 0 3 3 bracket 0 M and that'll clear out all of the formatting next I want to just have solid red text so I'll use another escape code slash 0 3 3 and then 31 for red foreground 44 black background and M then a message something went wrong and then I'll end the formatting with you clear out escape code and I'll save this and I'll run it I see I made a mistake here my error should be blinking so I'll go in and take a look an yep and there it is I have an extra semicolon with these kind of escape codes it's really easy to make simple typos let's run that again and there we go so it's blinking for me it's red and then the error message isn't blinking but it's also red it's kind of annoying but it'll definitely catch your attention and otherwise boring terminal window so it's a little bit tedious to type this kind of thing over and over in the script but since we're using bash we can make it a little bit easier on ourselves using basic string concatenation I can break out that flashy red code as a variable and the solid red and the reset string as well then I can just use them wherever I need to so I'll replace that first one with the variable flash red then I'll replace the clear out code with none they don't replace the solid red string with the red variable and I'll make sure that my coding is right so that this all lines up just the way that I expected to and over here I'll use the none variable again to clear it out for the final time so now I'll save and I'll run my script and it looks exactly the same way that it did before so that could definitely make using this kind of color and styling values in long scripts a whole lot easier there's another way of specifying this formatting as well using a utility called T put this ways a little bit more verbose but it doesn't involve as many arcane looking escape strings let's take a look at those commands here's a selection of them the ones that match up with the options I showed you earlier the foreground and background styles have numbers from 0 through 7 to specify colors and I'll show you those in a minute there are keywords here for the other styles as well and here's the colors you'll see here a particular color like say green for example is 2 using the set AF command and it's also 2 using the set a B command for the background whereas in the ANSI 1 they were different values I'll switch over to my script and I'll change it to use the teapot notation and since teapot is a command I'll need to use command substitution so I'll change flash red using T put set a B for the background color to 0 which is black a semicolon since I'm using more than one command here at the same time T put set a F for the foreground color to 1 which is red and then sync because I want that text to be blinking next I'll change the red one and I'll do T put set a B zero for black background Antti put set a f-14 read for ground and then for none I'll use T put s gr0 which resets all of the styles so I'll save I'll go out here to my script and it runs exactly the same way there's a lot of options available for T put and to check them out you can take a look at the man page for term info so if you scroll down to where you see these things like enter dim mode and enter reverse mode enter stand out mode you'll see that the second column in this case dim and Rev and SMS oo 4 stand out those are the values that we were using with the t put command and you can see all of the options that are available here and there's a lot of them I want to take a couple minutes to talk about date and printf date is not part of bash but it's really useful for certain kinds of scripts the date command gives you a format like this and you can use a string formatting option the character plus followed by a format specifier in quotes to get pretty much any format you need here I've used percent lowercase D - percent lowercase M - percent uppercase Y and that gives me the day the month and the year here's another one percent capital H - percent capital M - percent capital s and that's the hours minutes and seconds of the current time you can explore all of the other format specifiers on dates main page and there's a couple pages down as you can see there's a whole bunch of things that you can use and I encourage you to explore those I think you to quit out of here another handy tool is printf a built in that comes with bash which lets you print out data with a particular format well you can use echo to lay out text manually printf gives you a lot more options let's take a look at some of them so I'll start to print a string with printf then I'll use a name colon which is a label that I want then backslash T which is a tab character this is going to help me line up my information they don't use percent s which is the format specifier for a string which I'll specify later it'll use backslash n which is a newline character and then for the next line I'll use ID colon a label for an ID number another backslash T to keep things lined up and then % 0 4d what that does is say I want to pad this filled with zeros to 4 places and they'll specify a number or a series of digits in a little bit then I'll use backslash n again for another new line and I'll use an ending quote after that I have a string that I want to pass in for the first % s and then a second string which in this case is the ID number notice that it's only two characters I'll run this and you can see that everything got lined up nicely because of the tab characters it's broken across a couple lines because of the newline characters and my ID number is padded with zeros two four digits I can change that information without having to mess around with the formatter string so let's put together a little script that uses both of these helpers I'll defined a variable called today using command substitution using that format string that you saw earlier and another variable called time again using the format string you saw earlier they don't write a printf string using the option - V which assigns the output to a variable in this case D I'm using this because command substitution strips out the new lines and I'm specifying new lines so I definitely want those to show up in the result and then opening the values that I want to pass it into my string variables the first one current user I'll use the batch variable user which is the current logged in user and then today which is the date for today and time which is the current time and then I'll echo D using quotes to preserve the new lines and then I'll run my script and there it is there's a lot more format specifiers that you can use and I encourage you to check them out here's a list of them on the bash hackers website in a script you might want to keep track of a collection of things rather than one-off variables for that you can use an array in bash you declare that a variable isn't array by enclosing it in parentheses there's an empty array called a and an array called B that I'll give three elements notice that there aren't commas in between these elements like there are in a lot of other languages to retrieve an element inside an array you use it's a zero-based notation along with the variable name inside of curly braces so in this case when they run it I'll get cherry because that's the item that's an index to remember that arrays are zero based you can also set array values by index number and you don't have to populate every element these sparse arrays can be very useful when you need to record data with a particular position in a list instead of just adding to the end of the array but if you want to add something to the end of an array you can use the plus equals operator make sure you put the value in parentheses if you don't do that special append the value to the 0th element in the array so I'd end up with apple mango banana cherry Kiwi and you probably don't want that now I'm going to do that I'll take a dollar sign curly brace and then in square brackets the @ symbol which represents the whole array I'll save that and then I'll run it I can also grab the last element using the notation that I showed you in the movie about strings and in this case that's a mango now if you're using bash for and above you can also make associative arrays that is you can specify a key and a value to do this you'll use the declare keyword clear on my script here then I'll type declare and then - capital a I'll call this one my array if your key or value has a space in it you'll need to use quotes and then I can access those values by using the key in this case I'll access the value HQ West by using the key office building it'll access the value blue by using the key color and I'll run that that looks like I've got a problem there there it is I'm missing a quart at the end of this key and there we go HQ west is blue both of the values that I set in my associative array if you're using bash 3 or below you won't have support for this so keep that in mind when you're building your script arrays don't have a maximum length and they can be really handy for keeping track of things in your script eventually you'll probably want to work with files if you want to work with the contents of binary files you're on your own for that that's a territory better served by C or something a bit lower level but working with text files and bash is pretty easy the greater than unless then symbols are key here for example if I write echo some text greater than sign file dot txt bash writes the string some text to a file called file txt creating it if it doesn't and overwriting anything that's already in it you're probably familiar with this from working with output redirection you can use the greater than operator by itself to zero out of file getting rid of anything that's inside it but let's put something back in there if you want to add onto the end of a file rather than replace it you can use two greater than symbols so what about reading files well for that you can use a while loop along with the read command this syntax reads the file file dot txt into the variable F line by line to see this in action I'll add an incrementing variable and print that out alongside each line and I'll save this and I'll run it and just like I can use a file as an input for a loop I can also use a file as an input for a command like a set of instructions a simple example of this would be to use cat to read a file but a more interesting use would be to set up a list of instructions to execute automatically for instance I'll write a quick text file and I'll call it FTP dot txt so here I've written instructions to connect to one of the mirror servers for Project Gutenberg this one's hosted on X mission comm I'm logging in as the user anonymous and I'm using the password nothing here because you know nervous user doesn't need a password then I wrote ASCII FTP want to know whether you're going to be using an ASCII or a binary connection in this case I'm using ASCII because I know I'll just be transferring text then I have CD space Gutenberg because the file that I'm looking for is in the Gutenberg directory and then I'm issuing the command get good index zero zero which is just one part of the index so I'll save this and then I'll use the FTP command with a dash and option so the FTP respect to the user I specified in the file and doesn't try to connect just as whoever I'm logged in to my session ask and then I'll use the less than symbol and referred to FTP txt as the input and when I less the directory I see my good index dot zero zero file so you can see that can be pretty handy one of the Handy features of Bosch is what's called a here document this lets you specify input freely up to a specified limit string which can be handy for displaying long passages of text in your script or for specifying options to an interactive command like reading a command file as we saw earlier in the chapter so a here document looks like this I used two less than symbols in a limit string to say take all of this input and feed it into the given command in this case I'm just using cat up until you see the limit string now obviously that limit string needs to be unique enough that it won't appear as part of whatever you're passing to the command so I'll save this and when I run it I see the text that I entered this is used a lot for instructions or other long bits of text that would be tedious to echo out line by line and because of that there's another option to know about if I put a dash after the West End symbols Bosch will strip out the leading tabs from the text that follows and that can help make your script a lot more readable you could also use the here document as a built-in way to download or upload something that your script uses if you host files on an FTP server let's take a look at a previous example getting part of the index of books from Project Gutenberg you'll notice a couple minor differences in this case I'm getting good index dot 0 1 since I've already got good index dot 0 0 in a previous movie I'm also using the FTP command by to disconnect from the FTP server if I don't do that I'll get an error when the script runs you could wrap this up in a menu option or specify a flag to trigger it we'll take a look at how to do that later on in the course but if I save this and run it it goes out and gets the file so that's a really handy and powerful option that you can use in your scripts for this challenge imagine that you've been asked to create a script that output some information about the system to the screen and save some information to a text file use variables some string manipulation some text formatting and some command substitution you'll also need to use redirection to get information into a file of course there's no right or wrong answer in this challenge but try to get a few different pieces of information and display them in a nice way this should probably take you about 20 to 30 minutes good luck my solution to this challenge looks like this yours will probably look different and as long as it returned some information about the system and looks nice that's just fine let's take a look at the script that I built for this here at the very top I have the shebang line and then I start defining a few variables the first one is free space which uses the DF command and a couple options to get the free space available on the route volume which was represented by the backslash that information is piped into graph with a simple regular expression to match that line and then that's piped into another utility to print out the specific information that I want to get at after that I have green text bold and normal which are three formatting strings that I'll use later on and then there's a variable called log date that's using the date command to print out the current date the four-digit year represented by capital y the month on the date after that I define a variable called log file which concatenates together that log date with underscore report dot log so that'll generate the name of a file using today's date then I have echo for the first line of my output and I'm using some of the formatting strings making the whole string bold and then I'm using the host name variable which comes from bash to get the host name of my system and that's gonna be in green and then at the end of that line I reset the styles to normal then I use printf a couple times using pretty much the same construction I have a tab character here at the beginning and then a label system type bash version free space and files in directory and generated on and then I have another tab character to keep everything lined up and then I have a format specifier for string and then a new line and so into this first one into this string I have mock type which tells me the Machine type and that comes from bash in the next line I have bash version which is pretty self-explanatory then I have free space which comes from this DF command up here using grab and hog then I have a listing of the files in the current directory which is using command substitution with the LS command typing into a utility called WC for word count with the option - L which counts the number of lines with LS equates the number of things in the directory in the next line here I'm using command substitution as well with the date command using that format specifier that we saw earlier to get the data as a month day and year and then I have a comment over here this is the US date format so if you're in Europe or Africa or Asia or somewhere else it has a different standard date format you'll probably have a different date format if you used to date in your script then I have echo with a formatted string again saying that a summary of this information has been saved to the log file and then I start writing to the log file using a here document basically passing two lines terminated by EOF in this case into cat and I'm using the - modifier here so that these tab lines aren't represented in the file that I'm writing out and then I'm using redirection here to take the output of cat and put that into the log file which is as you'll remember the date underscore report dialog I'm using printf again with some format specifiers to print just a summary of that information that I displayed on the screen out to a file so I have sis and then the mock type bash for the bachelors in and then the bachelors in variable and you'll notice here I'm using two double greater-than symbol so that I don't step on what's in log file I just add to the bottom of that log file so that's my script and then you can see that it's created this date underscore report dot log file so I can take a look at that with cat and I see here's the information that I specified that two lines of text that I used to here document four and then the two pieces of information that I use printf to print similar to how I did up here remember the goal of this challenge was to get some practice using bash to solve a problem you an if-statement exit an if statement executes code based on the truth value of a given expression in bash that's a gif followed by a statement usually in single or test brackets or extended test or double square brackets or it can be contained within double parentheses for an integer comparison or have no brackets at all whatever is appropriate for the comparison you're doing that's followed up with a semicolon and the word then or you can drop the semicolon and put one on its own line after then goes whatever you want to execute if the expression is true optionally you can specify an else condition which executes if the tested expression evaluates is false and you close the if statement with fi if backwards if you need to do additional evaluation within an else block you can use the Elif or LIF statement with its own then since it's another evaluation let's take a look at how you might use this in a script I'll open up my working file - H and I'll set a equal to 5 and then I'll use an if statement with integer comparison and then I'll echo a is greater than 4 if a is in fact greater than 4 otherwise I'll echo a is not greater than 4 then I'll close the F with fi so I'm testing a 5 is greater than 4 5 is in fact greater than 4 so I see the message that I specified when the condition is true I can edit the script and flip that around and we'll change that to 2 and then I'll run the yes and I see 2 is not greater than 4 well that's right you can also use an if statement with a regular expression to test if a string matches what you're looking for for that I'll use the extended test notation with a double brackets I used the equals tilde operator here to indicate a regular expression and then I use the very basic regular expression square bracket 0-9 square bracket to look for a digit and then plus 2 say one or more of these regular expressions are very powerful and I encourage you to check out using regular expressions with Kevin's Gobind here on lynda.com for more in-depth information about them now I'll change my echo line since those don't make sense anymore in here I'm using ctrl K to cut lines so I'll save this and I'll run it and I get the result I'm looking for of course I can change that by putting a number in the string this is my number one string now and now it sees that there are numbers in the string because it matched the condition for my regular expression so checking to see if something's true or not is only good up to a certain point to make it even more useful let's use it in a loop we don't always want to have a loop work on a specific range of values we might want to loop to continue while some condition is true or false or until some condition occurs for that there's the while ends until loops first let's look at the while loop with a simple example that comes up to 10 here I'm using both the integer comparison to loop while I is less than or equal to 10 and doing a little bit of arithmetic to increment the value of I by 1 well notice that the syntax here is basically the same as the F statement which of course makes sense because we're asking for an evaluation that is where there I is less than or equal to 10 and if it evaluates true then we do everything within the loop and then at the end of the loop is the word done I'll save it and run and I see the numbers 0 through 10 the until loop is the counterpart to the while loop here I'm echoing the value of J until J is greater than or equal to 10 and when I run it I see that the loop ran until J was equal to 10 keep in mind both of these can cause infinite loops so be careful to check your logic now let's take a look at the for loop a for loop sets up a loop based on particular criteria usually a sequence or a range of things with it we get a variable that contains the current value with each iteration here's a simple example so if for loop looks like this it starts with 4 and has a variable in this case I it contains the current variable each time that it steps through the list in the word in and then whatever list of things 0 in years in this case 1 2 3 then the statement do followed by anything you want to do with the current value at that iteration of the loop and then that's all ended up with done so in this case just like about 1 2 3 so let's see that and there it is so this could get really tedious if we want to use a large range say from one to a hundred so for that we use brace expansion so this will list all the numbers from one to hundred and there it goes so already this syntax has saved me a whole lot of typing in bash 4 and above I can specify an interval and there we go counting by twos another variation of this for loop is available too which might look familiar to you if you've worked with a C style language setting a variable equal to some starting value and then a test to see if that variable meets a certain criteria and then some operation on that variable usually an increment or decrement so I'll save this and I'll run that of course you can leap through an array as well get rid of that line with ctrl K and I'll specify symbol array and then I'll set my for loop for I in and then using parameter expansion here I'll read the array then I'll run my script and there's each of the items in the array working with an associative array is a little bit different and again remember this will only work on bash 4 and above again I'll get rid of those lines using ctrl K to cut the lines and then I'll declare an associative array and I'll start populating that array of a key called name and I'll set the value to my name and I'll have a key called ID I'll just set to some value and then in the for the I started just like normal but so there's something a little bit different going on I'm using an exclamation mark to access the keys in the associative array and since my keys are strings they could have spaces in them so I'm using quotes to accommodate that so when I leave through the variable I contains the key and so I need to modify my echo statement to get the value of their ray that corresponds to that key so here I'm echoing the value of the key and then I met going the value as referenced by the key in the array so I'll save that and I'll run it and there it is I have a listing of the key and the value for each of the key value pairs that I specified so finally let's take a look at using command substitution if I say before I in and then use command substitution just to do something basic in this case like LS this is as the output of the LS command as a series of things for the for loop to loop through so if I run that I see each of the items in my folder of course there are much better ways of running the LS command but the shows how easy it is to integrate some logic with the operations you use at the command line so what if you have a number of different things you want to test for well you could use a long series of if statements but there's a better way the case statement case checks a value against a series of values you provide and it shows up pretty often in conjunction with scripts that take an argument to control an outcome I'll talk about those in more depth in the movie about arguments I'll set a variable a equal to dog now a case statement starts out with the word case followed by the variable you're testing and then the word in then on the next line I'll put a condition to test this will test if the value of a is equal to cat and the right parentheses to indicate the end of that test that's followed up by whatever I want to do in the value matches in this case echo feline and then two semicolons to tell bash that you're done with that test and you can just keep adding test conditions you can use a pipe character to specify a list of things to match this will match a dog or puppy and if nothing matches there's a way to catch that as well using the asterisk to close the case statement I'll use es AC case spelled backwards now I'll save and I'll run the script I see k9 which is the response I expect because the variable a is set to dog if I go in and change that to cat I get feline and if I change it to bird that doesn't match any of the conditions so I get that fall three response obviously using a case statement is handy for menus and working with user input and you'll see more about that in a little bit during your script development you may find yourself repeating code blocks it's a good practice to not repeat yourself if you don't have to and it's a whole lot more maintainable to keep things organized bash functions can help out with that to declare one just type the keyword function and the name of your function followed by curly braces inside the function goes whatever you want to happen when the function is called and then to call the function just call it by name I'll save that and go out to the command line to run it and there's hi there echoed by calling the function greet and of course the contents of these functions can be as complex as you like but there's one more thing that functions are great for wrapping up code that will process some value that you give the function to do that you pass one or more arguments to the function that's accomplished by putting values right after the function call and up here in the function I'll make a little bit of a change I need to accommodate a name and I'll use the variable dollar sign one which represents the first argument passed into a function now if I save and run the script I see my name passed into the function great now I said that the dollar sign one represents the first argument passed and you've probably already figured out there's a dollar sign two dollar sign three and more well if you go to 10 or above you'll have to put that number in curly braces but you get the picture so I'll change the output of my function to accommodate another variable I'll pass some more arguments into the function and if I run that I get the response that I expect but maybe I have a list of things that I want to run through my function like the output of a list command let's make some space here and I'll define a new function called number things so there's a lot going on here first I'm declaring a function called number things and then setting I to one then I'm using a for loop for F in dollar sign at which is a special array variable that represents all of the arguments passed to a function then I'm saying echo I : f which on each loop around is going to be the next item in that dollar sign at array of arguments then I'm incrementing I by 1 and finishing a loop down below I call the function number things and I'm using the shell expansion to get the results of command LS which will contain the name of all the files and folders in the current directory this could just as easily be a list of things does you'd find them in on the Ray and I'll run that cool now we're going to get somewhere you until now I've been writing self-contained scripts that don't accept any input but in the real world chances are you'll want to get some kind of information from the user to use in your script for this you'll use arguments if you're following along you'll see that these arguments are exactly the same as they were when you're working with functions if you're just running in I'll start with a little bit of review arguments are a special type of variable that are specified by the user when the script is run these can be pretty much anything a file or folder you want to create or work with a user name or a string of text to search for they're passed into the script by typing them after the script name arguments are represented by numbered variables and they're assigned in the order that they're provided at the command line so the first argument is assigned to dollar sign one and let's take a look at that here I'm using Apple as the first argument and since I'm not coming out the first argument in my script I see Apple the second variable would be dollar sign two etc and just like a regular variable anything with a space in it would need to have quotes around it that's pretty useful for a couple of arguments but what if I wanted to deal with any number of arguments rather than creating a variable for each one manually well Bash gives us an array of arguments which I can get to with a dollar sign at and I can use a for loop to walk through it there's also a variable dollar sign pound which contains the number of arguments let's run that you can see that that will accommodate an arbitrary number of arguments apple orange banana kiwi lemon and they counted them up and there were five so I didn't have to define a variable for each one of those which definitely saves a lot of time if you've been working with the batch command line environment for a while you're bound to have come across flags there's specific options that you can use to pass information into a program and they usually start with a - you can make use of these in your batch scripts by way of the get opt function let's say I wanted to make a script that accepts a username and password I could certainly use arguments for this but then I'd need to make sure that the user name and password were entered in a certain order let's take a look at how to set this up with a while loop using the get ops function I'll specify an OP string which defines what flag is I'm looking for here I'll use u : P : which means that I'm looking for a - you and - P so I'm using the case statement using option as a variable and when that option matches you it's it's user equal to opt Arg which is the value that's fasted from the user following bash you at the command line the same thing happens for - key so let's take a look at this running and I see that the values that I wanted or assigned one of the handy things about Flags is that they don't have any particular order now there's a couple other things you can do with the OP string the string like I have it with the colons after each flag means that I expect there to be a value along with each flag Def you username - P password but if I add other flags without colons after them that means that I just want to know whether or not that flag was used so I add a and B there and then I'll update my case statement I'll run this first without any Flags and then with one of the flags - a messy got the a flag I can also do that with B forgot the B flag and I can do both got the B flag got the a flag there's another change that I can make to the opt string adding a colon at the beginning tells bash that I want to know about flags that were used at the command line that I haven't specified and in the case statement I'll use question mark to capture that I'll save it and then if I run it using an option that I didn't specify I see that message I don't know what Z is and that's how you work with flags so you just saw how to get user input at the command line but sometimes it's not practical to have all of the values you're looking for specified when you run the script luckily there's a way to get input during the execution of a script already a typical question and then write read name the read keyword pauses the script to wait for input followed by return or enter and it stores that input in the variable you specify here I've used name there are also some options that Reed can accept let's ask for a password and here I'll use read - s followed by a variable name the - s means silent which means that I won't show the characters that I type in I can also write everything on one line with a - key option which writes out a prompt before the input area and let's I call that out just to see what's going on and if I run this first I'm prompted for my name and then my password which you can see is not appearing when I type and then our inline prompt what's your favorite animal I like cats and now you can see the values that were collected there are some other options as well and I encourage you to explore them if you need more flexibility in your script there's another way of getting input as well in a handy menu form let's make some room here and for that I'll use the keyword to select select is followed by the variable name that you want to store the selection in in this case animal and then the word in followed by a list of the options to select from after that comes a do block and that will encapsulate whatever you want to do with the selection it's important to break out of the loop when you have a response otherwise the loop will continue forever so I'll save this and then I'll run it to the command line and you can see I get a very nice and numbered list of words and a prompt to enter a number when I pick a number I get back the string associated with it of course this kind of input pairs very nicely with a case function to respond to the user selection let's change the selection list a little bit and I'll add a quit item and then here inside the do block a lot a case statement I'll change the variable name here as well to make a little bit more sense then I'll add a case block here I've got options cat dog and quit and in the case block I respond to each of those selections and then I have a fall through condition in case the user enters something that doesn't match an option that I've provided let's see what this looks like when it runs I'll type the number 1 and even though I didn't type the word cat the case statement is matching that term because it was provided by the Select statement it asks for input again and I'll type 2 for dog and I get back a similar response I'll type three for quit and it quits because I said to break for that option so that's all pretty straightforward and easy to implement in your own script well it's great to ask for input you probably want to build some error tolerance into your scripts what happens if someone skips a prompt by just pressing enter there's a few ways to deal with us pretty easily first let's take a look at a case where we have a script that requires arguments to run so this group checks for the number of arguments and because it wants to have three it won't run the program if it doesn't instead it returns a usage recommendation instead of just exiting or failing somewhere along the road when it comes time to use an argument that wasn't provided I'll save this exit and run the script since I haven't provided any arguments I get the usage recommendation if I provide just a few arguments the condition of having three arguments is still not satisfied so I get the usage recommendation again but if I give it three arguments which is what it's looking for then the program executes alternatively we can set up a loop to not allow the user to continue without specifying some kind of input this uses the - is the option in the evaluation to check that the variable in this case a is not empty let's take a look at how this works I'll save and then I'll run favorite animal I'll answer cat and it says cat was selected so if I run that again and I press ENTER instead of having an error this great prompts me for an answer and I can continue like this until they decide to finally answer and then when I do it continues the program but that can get kind of needy and irritating so instead what you might do is assume a default answer if the user skips the question by pressing enter this can also be handy for streamlining scripts where input is probably going to be a predetermined value but where you want to give the user the option to change it on the off chance that it needs to be different so I'll edit the script that we have and in the prompt I'll add what the answer is going to be assumed to be if the user presses enter here cat in brackets that's something you'll see a lot on an assumed answer in brackets and then inside the while loop instead of asking a question again I'll just set the variable to what I told the user that it would be set to in this case a equals cat so I'll save this and clear up the screen here and then I'll run the script favorite animal and if I hit enter the script is continues along making the assumption the cat was going to be selected anyway so if I run it I can also put in other input like fish for example and then fish becomes a value of a and that can be used in the script later you can also do some basic validation of input rejecting input that doesn't match a regular expression here I'm asking for a year and then using a regular expression to test whether the input does not match it in this case I'm looking for four digits from 0 through 9 I'm also giving the user an affordance of what I'm looking for in this case 4 ends in brackets to say that looking for four numbers so if I save and they run this script it Trump's me what year well what about the Year 300 I didn't like that very much so I could put it in two or three hundred and it likes that or if I run the script again and put it in what year last year it doesn't like that either so I'll have to put in 2012 and since that matches the regular expression it continues checking for input that matches parameters you expect can help to avoid errors later on in your code for this challenge let's build a little game make a script that takes interactive input from the user to guess a random number use a loop and a condition to check if the answer is right and keep asking if it isn't make the script respond to having an argument and give it some functionality if there isn't an argument provided as well and finally make sure to use a function in your script this should take about 20 minutes good luck so here's my result if I just run my script with no argument I get a random number and if I use the argument game I get this fun guessing game I'll guess one nope that's not right - nope three there it is six I only got that on my sixth try I must be great at guessing let's take a look at the script up here at the top I've got the shebang line and then I set the variable R and equal to the bash random variable so every time the script runs that'll generate a new value and then I set the variable secret equal to the very first character in the string for random I don't want my users having to guess a four or five digit random number so I'm just taking the first digit off of the string that Bash generates for me let's go down to the bottom first so here I'm asking if dollar sign one which is the first argument matches game lowercase or game with a capital G or game all in capital letters if it does then I call the function game if not I call the function generate so let's take a look at those functions the game function asked me to guess a random one digit number and then it stores that value in the variable guess then I have a while loop to compare the value of guess to the value of secret if it's not equal I'll ask for another try when the guest does equal the secret the while loop is done and I echo good job then further down here we have the generate function which gets called if there's no arguments at the command line it shows me the random number and it also gives me this formatted string as a hint in case I forget what argument this takes so that's what I came up with and I hope you came up with something fun as well you there are a lot of fantastic resources out there for beginner intermediate and advanced batch users and one of the best is the Bosch manual itself which you can get to in your terminal window using the command man Bosch or you can find it online there's a community at Bosch hackers with a lot of great guides and information as well the Linux documentation project has a detailed list of gotchas or things to watch out for along with code examples to describe them if you run into problems or have a question you can't find the answer to in the documentation Stack Overflow is a great option I hope you had as much fun with

Channel: Cybersecurity Shandy

Views: 16,335

Rating: 4.7994428 out of 5

Keywords: Scripting, Hacking, Cybesecurity, Ethical Hacking, Linux, GNU/Linux, MAC OS, Bash, Bourne Again Shell, Shell

Id: emhouufDnB4

Length: 60min 26sec (3626 seconds)

Published: Thu May 14 2020