Wget is a command line software that can be used to download everything available on the internet. It should be available over HTTP, HTTPS, or FTP protocols otherwise Wget won’t be capable of download it. There are various approaches in which Wget may be used. for example, Use Wget to download each image on a website, or download a single file, as well. Wget is even capable of resuming downloads, when possible.
[ad type=”banner”]Installing Wget
1.Installing Wget on macOS Sierra
Previously we get started with using Wget, we will need to install it on our system. Now, most Linux distributions come with Wget pre-installed, so you will not have to do any additional effort. If you’re on a Mac, you will have to install Wget. This can be done easily using Homebrew (download).
Here are the steps you’ll need to follow:
- First, you will have to install Homebrew on your Mac, in order to be able to download Wget. To do this, just launch the Terminal, and type the following command:
/usr/bin/ruby -e "$(curl -fsSL
https://raw.githubusercontent.com/Homebrew/install/master/install)”
- After the command completes, Homebrew will be installedon your system. Now You can use Homebrew to install packages, such as Wget. In the Terminal, just run the following command:
brew install wget
- Homebrew will automatically start downloading, unpacking, and installing Wgeton your system. Once the command completes, you should be able to use Wget on your Mac. To check if it has been installed, just use the command:
wget -V
2. Installing Wget on Windows
Installing Wget on Windows is easily. If you want to install Wget on your Windows PC, just follow the steps below:
Download Cygwin, and then run the downloaded setup file.
When it asks for the default download source, choose the option that says “Install from Internet“, and click on “Next”.
Pick an installation directory for Cygwin, as well as a directory where it will save package files for installation of utilities.
In the next screen, you’re required to select your internet connection. If you’re using a Proxy, choose the type of Proxy it is, and fill up the details. For most users, though, the choice would be “Direct Connection“.
You will then be shown a list of mirrors, choose any of these to download Cygwin. If the download fails, you can try again with a different mirror.
[ad type=”banner”]
In the next window, search for “wget”, and click on the plus next to “web” in the list that comes up. Here, the checkbox next to “wget” will be selected automatically. Simply click on “Next”.
The next screen will show you the packages it needs to download. Just click “Next”.
Once installation is completed, you will be asked if you want to create a desktop icon, and/or add an entry to the start menu. This is completely a matter of personal preference, but I’d suggest you at least add it to the start menu, so you can find it easily.
Now Wget is installed on your Windows PC. However, you’ll not be able to run it from the command prompt. Instead, you’ll have to launch the Cygwin Terminal, and use Wget from there.
How to Use Wget
Now we’ve got Wget up and running on our system then we can use Wget to download files, folders, and even entire websites from the internet. Here are a couple of interesting things you can do with Wget on your system.
Note:
If you are using Wget on a MacBook Air running macOS Sierra 10.12.3 beta. However, since the utility we’re using is the same (Wget), the commands will be the same on any operating system you may be using Wget on.
1. Use Wget to Download Files
You can use Wget to download files from the internet with ease. Just use the following command
wget <url>
- Download a File and Save it With a Specific Filename
Files on servers sometimes have the inexplicable names, and you may want to download the file, and have Wget automatically rename it. To do this, just use the following command
wget -o <output_file.extension> <url_to_download>
- Download Only Specific Filetypes
A web server may be hosting a number of different files. So, what if you only want to download all the PDFs on the server, or maybe all the GIFs then simply use the following command:
wget -r -A pdf <url_to_download>
This particular command was put to use by Mark Zuckerberg in his Harvard dorm room, to download all the images with just one command, and build the infamous website “facesmash.com”.
wget -r -A jpg, jpeg, png, bmp <url_to_download>
2. Download Files from Multiple URLs with Wget
If you need to download files from a number of different locations on the internet. There may be a PDF file on url1, an important ZIP file on url2, and something else in url3. If you used Wget in the traditional method, you’ll have to run it three times. But, there is a much easier way to do this.
- First, create a file with all the URLs you need to download data from. Ensure that you keep each URL in its own line. Save the file, and make sure you remember the name.
- In Terminal, use the following command to download files from all of the URLs:
wget -i <file_name>
where “file_name” is the name of the file you created with all the URLs you needed to download stuff from.
3. Download an Entire Website with Wget
If You can use Wget to download the contents of an entire website. Clearly, if the website is too large, this command will take a lot of time to finish. So if you want to use Wget to download an entire website, just use the following command:
wget --mirror -p <url> -P ./LOCAL-DIR WEBSITE-URL
This will download the entire website, into a folder “LOCAL-DIR” inside your working directory. The files of the website will be stored in a folder with the same name as the website.
4. Use Wget as a Web Spider
Additionally, you can make Wget act like a web crawler or a web spider. This is helpful when you want to confirm that the URL for the file you want to download is valid. To use Wget as a web spider, and check the validity of URLs, just use the following command:
wget --spider <url>
This will check if the URL you’ve mentioned exists, and you’ll get a response to the results of the check.
- Positive Response:
- Negative Response:
5. Continue Incomplete Downloads
Sometime if you were downloading a file with Wget, but for some reason, you had to shut your pc down. The next time you want to try downloading that same file, Wget can actually continue the download. You just have to ensure you’re in the same folder where you were downloading the file the last time, and then use the following command:
wget -c <url>
When Wget gets that a part of the file is already present in your system, it will automatically continue the download.
Note:
If the file on the server has been changed since the last time you tried to download it, then you shouldn’t use this command, because it will result in a corrupted file that you won’t be able to read.
Also, if the server doesn’t allow continued downloads, then Wget will refuse to start the download from scratch. If you want to start the download from scratch, first you will have to delete the partially downloaded file from your system.
6. Limit the Download Speed
In case you’re using Wget to download files, and browsing the internet together with it, now you’d possibly not need all of your bandwidth to be used up by Wget. Fortunately, Wget consists of an argument that you can use to limit the amount of bandwidth that Wget can use for downloading files on your system:
wget --limit-rate=<speed> <url>
Note:
Remember that the speed is mentioned in Bytes, and not Bits. So, if you use Wget –limit-rate=20k, it will limit the bandwidth usage to 20 kB, and not 20 kb.
7. Use Wget to Download Files from FTP Servers
If you need to download a file from an FTP server, you can use Wget as well. Simply use the following command to download files from FTP servers:
wget <ftp_url>
- Download Files from FTP servers with Login
IF your FTP server requires a login to allow download files, you can simply pass the credentials with Wget, as well. Just use the following command:
wget --ftp-username=USERNAME --ftp-password=PASSWORD <ftp_url>
