Black God

wget example tips and tricks for command line download

Some times before I was looking for command line download tool which can download in background. Going through the man page of wget, a lot of interesting features I noticed, which we are not familiar much. Now a days I am using those features in my day to day linux life.

wget is a highly used tool to manage your internet download from command line. But many of us don’t utilize it to its full capacity. wget makes your command line life more easier and both resource and time saving. I believe you need not to run a browser just for download, which consumes a consderable CPU time and memory. Apart from this download managers in browsers are less featured than command line download manager like wget. Here I am going to present a few less used but highly useful and handy features.

A typical usage of wget is,

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2

This command just downloads the file pointed by the url given.

Background download (wget -b)

$ wget -b http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2

This is command is highly useful when you initiate a download in remote machine through SSH terminal. This will start downloading in background, so that you can disconnect the terminal once the command is issued.

Continuing the broken partial download (wget -c)

$ wget -c http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2

This switch is useful for big downloads, which may get interrupted before completion. If already there is a file with this name already, it will check its size and start downloading the remaining portion of the file instead of full download again.

Retry attempts setting (wget –tries=<n>)

$ wget –tries=10 http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2

It is obvious from the switch name, it retries number of times mentioned to download the file, in case the server is busy and slow.

Rename the file downloaded (wget -O <filename>)

$ wget -O latest-kernel.tar.bz2 http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2

This switch will download the file and store with the file name mentioned in command line.

Bulk download (wget -i <input-file>)

$ wget -i to_download.txt

Content of to_download.txt

http://www.kernel.org/pub/linux/kernel/v2.6/next/patch-v3.0-rc4-next-20110624.bz2

http://www.kernel.org/pub/linux/kernel/v3.0/testing/patch-3.0-rc4.bz2

The above example is to download the files listed in input file called to_download.txt. You can store list of URLs in this file.

Mimic wget as your browser (wget –user-agent=<string>)

Some web sites validates the download client and deliver the content according to it. For example, when you visit firefox site from Linux machine, it automatically provides firefox for linux download link. But in some cases, the web server doesn’t allow less used download tool like wget. In this case you can mimic wget as your browser like firefox. Here is how I do it.

Get your browser’s user agent by visiting “http://getright.com/useragent.html”. Mine is “Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0″.

$ wget –user-agent=”Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0″ http://www.kernel.org/pub/linux/kernel/v3.0/testing/patch-3.0-rc4.bz2

 

Limit the download speed (wget –limit-rate=<speed>)

$ wget –limit-rate=1k http://www.kernel.org/pub/linux/kernel/v3.0/testing/patch-3.0-rc4.bz2

In my home networking, my 24×7 NAS server use to downloads files. During this time, my internet access from laptop may be slow if the server download occupies entire bandwidth. In this case I restrict the server download speed with –limit-rate switch. In the above case, it is 1kB/s.

Downloading behind a Proxy

You need to set the Proxy server through http_proxy environment variable as shown in the following example, which sets proxy server as “myproxyserver” and proxy port as “8080”.

$ export http_proxy=”http://myproxyserver:8080″

In case you need to get authenticated by proxy server, you can use the following form.

$ export http_proxy=”http://username:password@myproxyserver:8080″

3 comments for “wget example tips and tricks for command line download

  1. November 8, 2012 at 3:07 pm

    Good to know this..

  2. latedreamer
    August 5, 2012 at 11:08 am

    Thanks to wget, being organized is not that hard anymore.
    I just like to know if there will be more features sooner. I can’t wait for it.
    I highly recommend wget, great to know more info about it.

    Minda
    My blog : portail coulissant motorisé 

Leave a Reply

Your email address will not be published. Required fields are marked *