Wget | Linux Command Line Utility to Download Files

what's up guys this is josh from cupid techie and today i wanted to show you guys the wget command [Music] now as i stated in the intro the video i want to talk to you guys about the linux w get command line tool this is a pretty cool utility used to download files over the internet so if you ever want to know how to just strictly download files directly from the internet from the command line then this is the tool you want to use it's typically used when you want to download like a tor bar or a zip file and sometimes devs and rpm packages from like different websites and with this command you can actually download over all the different protocols like http http https or like ftp and it also has a whole bunch of options that go along with the with the wget command that make it very powerful so in this video i want to go down and show you guys how to install it if it's not installed on your system as well as a few ways of how to actually use the wk git command to download random files from the internet so let's get started okay cool so i have my terminal up and i want to go down and start off by showing you guys how to install it in case it's not installing a system uh most modern linux distributions come with w get installed on the system but in certain cases it may not be installed i just want to show you right fast it's simply simple it's in all of the distributions so this is a utility that's been around for a while so it's in all the different linux distributions so you can install it on any of them so for example uh if it's debian or ubuntu you know sudo [Music] apt uh install and then get that's the package name so it's wget and so it's the same on all you know the distributions you know you use uh the young package manager you know what i'm saying or pacman they all you know it's the same package name because like i said wget has been around for a long time that's why it's included in most distributions nowadays so now that we got that out the way let's go down and show you guys how to actually use it and i already found like a tor file on the web that i wanted to use as an example and it's basically sublime uh text editor that i'll be downloading i'll be downloading the tor file for it it's the latest version so if you go to you know whatever website you're trying to download some from you can right click on the actual link that actually downloads whatever you're trying to get and copy that link and then paste it into the terminal so i'm gonna do that right now i already have the website open so i'm not gonna gonna show it to you guys but i'm gonna just copy the link and the base way to actually run the command is simply w get but before i do that i totally forgot to do something i always like to go to the main page just so you guys can know that when you're using a application for the first time or utility for the first time you want to check out the man page that way you can get a clear understanding of how to actually use the application so let's read a little bit here so wget the non-interactive network downloader uh so here's the synopsis and here is you know the main way of running the command so it's wget then the options and then the url and if i want to read the description it says gnu w get is a free utility for non-interactive download of files from the web it supports http https and ftp protocols as well as retrieval from http proxies get is non-interactive meaning that it works in the background while the user is not logged in this allows you to start a retrieval and disconnect from the system letting w get finished to work by contrast most of the web browsers require constant users presence which can be great can be a great hindrance when transferring a lot of data wget would can follow links to html x html and css pages to control to control local versions of remote websites fully recreating the directory structure of the original site this is sometime sometimes referred to as recursive downloading while doing that w gets w get respects the robot exclusion standard and basically i'm not going to read the rest of that that uh paragraph but just to quickly uh uh so you guys can understand this you can basically download a full website you can copy all the html all the css files everything down to a local your local system uh and you can view a website offline so that's that's basically what that means you could you could download everything to your to your local system and that's another option i'll show you guys later on is you're basically cloning the website down uh but the only thing is you won't have like the sql information all that back-end information you won't have any that you'll just have the front-end files that allows you to look at the website and this is great like let's say it's an article that you want to read later on you can actually download the website or download it to your system and read that article later or read it locally you know on your system so let's go to that next paragraph it says wget has been designed for robustness over slow and unstable network connections and that's another option i'm gonna show you guys you can actually set the max amount of download speed so it doesn't fail like you can slow down the connection or the download speed you can specify that in the command so i'll show you that option as well but i won't read too much more uh let's go down and just show you guys a couple examples right fast but here are the basic startup options or basic options right here you know you have your version help background uh and like i said i'm gonna go through some of this stuff you can create log files you can do quiet mode verbose where it shows a lot more information you know all that good stuff but we're gonna go ahead and quit this right fast and go right to actually showing you guys how to actually use it so let's clear right fast and show you guys the first way i want to show you guys how to run it and i'm gonna download this specifically to my home directory so it doesn't matter i don't have to put a or a specified directory so i'm just type w get and then i'm gonna paste that link in there and that's what you want to do as you can see this is the download link off of sub sublime text dot com page and it's actually referencing the torah file the latest build of it so if you try just trying to download a iso or a tar file or whatever you can use the command line to actually do it so let me run it right fast and just show you because it's not that big of a file but i'm going to show you the output right fast just by running it you know the regular way so let's press enter and let's just wait for this thing to finish as you see is it wasn't that big of a file it's only like 13 megabytes uh and i was able to download it at 2.19 megabytes per second and i got it in 6.5 seconds and it's cool that it gives you some output by default because a lot of applications you have to specify to give me output i want to be able to see output you know directly from this command and i didn't create any aliases for it it's all the base level of of the w get command no options or anything but if we store it back up here at the top you know it gives you a date and time stamp and then it also references the file that you actually downloading and then here's your search and then it says resolving download it resolves it and then connects and then it sends the request and then it waits for a response and then it starts with the actual download it says saving to your local directory which that doesn't specify that but that's it's basically whatever working directory you're in that's where it'll save the file and right now like i said i was in my home directory so it saved it right to my home directory and then it gives you the output as is downloading it you know and the percentage which is cool you know it it uh filled in the board just letting you know how how much longer you got left as well as you know it was successfully saved and if we run let's say ls uh and then let's grab it into or let's run it into grip and pipe it in the grip and i want to look at sublime because i want to show you everything in my home directory right now um press enter and there you go you'll see that it's downloading on my system so good to go and actually let's get a little bit more information uh so if we go la boom there we go so as you can see it was downloaded and everything um today well actually this is when it was created so that's the latest version was created last year in september so it's one years old uh since they updated this actual this actual package so but this is the most recent package so i just wanted to show you guys how to actually download it using that now there are a few more options that i wanted to show you guys was a couple more that i wanted to show you guys and the first one i want to show you guys is actually the dash o option or capital is actually capital o option and what you want to do is or what that option does is it allows you to change the name of the file that's being downloaded so i'm gonna up arrow on the command that i ran uh the wget command i'm gonna go up here to the front because i don't want i don't want to be pasting that link but uh dash capital o and then let's say we want to name it something else let's say we just want it to be sublime uh text dot tor dot bz2 so let's say we want we only want it to be you know that we want to shorten the name basically well you could do that by using the dash capital o option so let's press enter on that and that'll download it and actually save it as that file name that where we gave it as you can see it's taken us through the whole thing process again and it's downloading that file and it changed it to that name so now if we run that ls that's la grab grep uh sublime and actually just put h in there so you guys can see the human readable version of the size press enter as you can see we got both of them there so the sublime text the one that we renamed and then the original download that we did is right here so let's go down and clear this up right fast but that was the first way i wanted to or one of the first options i wanted to show you guys now let's roll right into the dash capital p option this allows you to specify a certain directory uh and that's why i want to you know kind of show you guys in order of how most people would actually use this uh so you and then also you can use a lot of these options in tandem or together so dash p and let's say we want to um put it in my downloads directory so let's go home and then josh and then i have a downloads directory that i want to save it to so let's put a space in there but that's where we want to download it we want to download it to the downloads directory so we press enter that'll download it and it'll download it to my actual downloads directory in my home directory cool so that's going on in uh in ls that directory as well uh let's put uh the directory that we wanna you know ls and actually let's just put downloads uh and press enter as you can see we got it in my downloads uh folder as you can see i list you know the downloads directory and i grep for sublime and that's the one we downloaded right there i'm gonna have and i'm gonna i'm gonna go crazy trying to find out all the different places i stored sublime text uh torah.bz2 on my system uh i can quickly grab it and delete it or find and delete it but the next thing i want to do is show you guys how to actually limit the rate of the download now this is a option let's say you want to download something slowly you know over time you don't want to raise too many eyebrows with your isp if you're downloading something or or if you try not to go over your internet limit that you have set then you might want to use this that way to download over time is something you don't need right away it may be like let's say it's like 100 gigs of stuff you're trying to download from uh from somewhere and you want to download it slowly or it could be multiple archives that you're trying to download and i'll get to that option later but you can you could download multiple you know links at one time but i'll get to that later but let's say it's a huge amount of data that you're trying to download and you want to limit your speed so i wanted to show you guys how to actually do that so we're going to let's see back this out right fast but the option for that is actually a longer one so let's type it out though but it's dash dash uh limits and i'll space i'll tab it out so i don't have to type the whole thing in but it's limit dash rate equals and you want to specify that rate so let's just do example of so let's say 500 kilobytes so all you got to do is type 500 k and it could be lowercase you want it to be lowercase and then press enter and that will limit the download speed now i'm not sure how long it's going to take it's probably won't take too long well it'll download pretty quick but anyway it'll it'll limit the speed that you download this actual file and as you can see is adding a one to the end of it because i'm downloading it to my working directory and if as you can see right here too i wanted to point this out but you see that it's being limited right by it only using 500 kilobits per second so that lets you know that that actually works and so you can specify whatever you can put it on 100 kilobits per second you know i'm saying it'll take his time and download but yeah another thing i was gonna say it put the dot one on the end of it because it's already there uh within my home directory so actually let me just remove a couple of them right fast actually i'm gonna remove them all um so subline and then we'll put a star and press enter and that'll remove all of my home directory and then let's remove out of my downloads directory as well um and then sublime text and delete that as well now let's go down to clear again now i want to go and show you another option and i have to do something right fast in order to actually do it uh so let's let's run the let's actually run the uh the command where i limit it and this will download that like i said they'll it'll download it at 500k so let's go down and press enter and wait until it starts and then what i'm gonna do is i'm gonna kill it i'm gonna kill the actual download so let's say it's a big file and you end up killing it or you lose connection let's say you you lose internet connection for whatever reason well it'll kill that download so if we let's say let's run that ls again and actually yeah there we go so that's not the full file as you can see the size of it is 3.9 megabytes uh and the file is actually 14 megabytes which is the size of the actual file but since you know it killed it or we killed it or whatever or we lost internet connection so to speak then we can actually continue that download i mean let's say this is a 20 gigabit or 20 gig download that you're trying to do and you've already you know downloaded about 25 of that which is about five gigabytes you don't want to have to start from the beginning you can continue from that point forward where it's the option called that option dash c so let's go down and run this thing again but since we know this is the command that we use to start downloading it let's go and add that option to the beginning so let's go dash c lowercase c and press enter and that will continue from where we started as you can see it started like halfway through it and i mean you see this uh these pluses that lets you know that that's the port that was already downloaded and now it's continuing on from that point so this is a a great way to you know re start a download that we had already previously started and let's say your connection failed so that option is there for you guys if in case that happens to you using w get okay so let me go down and clear right fast because this will be the next one i want to show you so let's go ctrl l and this option i want to show you is the dash b option and this one is pretty cool this is how you download things from the background or in the background so we type w get and then dash b and then i already copied a link for our arch linux uh and so this is a rather big was small download because orange linux uh the base uh download is like 600 megabytes so it doesn't take that long but i just wanted to show you this option by using a bigger like iso so that but the dash b option is for you to download things in the background and i'll show you how to how to actually look at the status of it so let's say it's like a 20 gig 20 gigabyte back uh 20 big 20 gigabyte file you're trying to download or something and i'm just you know hypothetically speaking uh 20 gigabytes but this is 600 megabytes so don't worry about it i just want to show you how to actually look at the status so let's go down and press enter on this and like i said this will go in the background now what you want to do is type tail which is a command i think i did a video on this in the past but tail uh dash f and then let's look at the w get log dash log and press enter as you can see that gives you the status right there and you can press uh c on it but you can see the status is is uh saving it as you go you know i'm saying that you could uh just press ctrl c to stop it uh so you can you know let's say you want to download it on a server or remote server uh you can start to download and come back like an hour later just to see where it is and then disconnect from it you know by pressing ctrl c that'll stop whatever you have running but you know that w get is still running the background because if we run it again uh tail dash f uh and then w get dash log we press enter and you'll see that it's it's still running in the background it's still downloading in the background so let's go down and uh and kill that now just let that run in the background it doesn't matter it's not gonna you know affect the rest of the commands i want to show you guys so let's press ctrl l uh and like i said i'll let that keep running in the background but that's how you actually run something in the background using wwg now the last thing i want to do is show you guys how to multiple how to download multiple files from one command so one thing you need to do is create a text file so i'm gonna create a text file right fast so let's go uh nano i'm gonna create in my own directory so nano and let's say wget uh example dot txt and let's press enter on that and all you have to do with the downloads file is you actually just copy and paste your two links or more two or more links into the file and it'll go through them one at a time and i'm gonna show you the option that actually works with this so uh i'm gonna save this right fast so so i'm gonna hit ctrl x uh yes and then press enter and that'll save it as that file name and the option that you want to use is the dash i option so all you have to do is type w get you can use it with uh all of your you know other options that you want to use and then we want to specify that that file name and then let's say we want to limit it as well we can type that in at the end of it so we can go dash dash limits rates and then we type 500k at the end of it and press enter and this will go through and download it and it'll it'll also limit it to 500k on the download speed and it shouldn't take too long because i think the second file is just a like a md5 hash for porch linux so it shouldn't take too long it's just the sublime text is it's probably a little longer because i and then plus i'm limiting it to 500k so once that's done it'll be like instantaneous for the second one cool and we're done it didn't even take a second you know it took a millisecond to actually download it but you can see right here it gives you a little bit more information you know it'll tell you at the end you know it finished it'll give you the finish time the total time for both of the files combined you know and they'll tell you downloaded two files you know 13 minutes i mean 13 megabytes and 27 kills 27 seconds and that's pretty much it how you do it okay so the last thing i wanted to show you guys was how to actually clone a website or you can actually pull all the files down you can pull down the images you can pull down you know the html the css file javascript you know all that good stuff by using the dash m option now let me go down and do it uh i'ma just run it against orangelinux.com or org i'm sorry so and i'm not gonna run the full thing because it'll download the full site uh and everything on the site so i'm just running for a couple minutes and then i'll bring up my browser and actually open up the files that it downloads uh now it won't look you know real good because let me download not downloading the full site you know you won't be able to see it exactly the way it is on the website but i'll open it up locally uh so you guys can see what it actually downloads so i'm just download a little bit like i said um let's just let it go a little bit longer uh just so you guys can see so let's go down press ctrl c and that'll kill it but let me actually open up i'll be right back with the website up okay cool so yeah this is orange linux.org or whatever so as you can see you know that's the main website their main website and everything so i'm gonna open up the actual directory that it downloaded on my system using wget i'm gonna open it up in the same browser so let's just run open let's see open with brave and what i'm looking for is the index.html file which is what i'm going to use to open up the site just so you can see what it actually looks like when it downloads so this is basically an example of the site being downloaded so a lot of the files did download some of them like html files but not you know not that many not all of them you know i'm saying as you can see it doesn't look the same a lot of the images are messed up but you can actually download a lot of the text that's one good thing about it is just getting a really getting the text which is you know one of the benefits for me when i've when i've looked at it in the past and as you can see this is the directory on my home directory well in my home directory so home josh uh www.orchlinux.org and then the index.html i just opened that file up which was you know ported at downloads folder and actually we may be able to see the full directory if i back that out press enter yeah you can see it downloaded most of the directory of the actual website and actually it would have had a long way to go to be honest i think it it you know it'll go all these folders deep into the actual structure of the website so you can actually open it up but that's pretty much all i want to show you guys today i hope you guys got something out of it uh check out the w get command you know is there to handle downloading you know different archives from the web as well as you know cloning websites just to kind of play around with it and also another thing about this you know i'm saying this has the you know the full html you can look at the html you can open up the html and like bluefish or whatever text editor you want to look at you can open it up and actually see the html that was written to build the website so it's pretty cool to check that out you know i'm saying especially if you're trying to get into web development let's say you want to uh clone a website down just to see how to actually accomplish something uh then you use w kit to actually accomplish that so you can see the text but like i said i hope you guys enjoyed the video please like share and subscribe to the channel if you have any questions please leave comments down in the comment boxes below and of course keep it techy [Music] you

Channel: KeepItTechie

Views: 512

Rating: 5 out of 5

Keywords: wget, howto, tutorial, command, tar, tarball, zip, deb, rpm, ftp, download with wget, wget download, web wget, wget web, curl wget, curl, KeepItTechie, learn linux, linux, wget command in linux, wget command, wget tutorial

Id: KukVFf0hEwE

Length: 27min 45sec (1665 seconds)

Published: Sun Sep 20 2020