Wget Tips & Tricks

Download to a different folder :

wget -O /etc/yum.repos.d/CentOS-Testing.repo http://dev.centos.org/centos/5/CentOS-Testing.repo

Download File with Authentication:

wget –http-user=Username –http-password=password  ‘url_to_download’

Using Curl :

curl –user $USERNAME ‘url’

Download with Rate Limited Bandwidth :

wget –limit-rate=512k 

Resume , restart a download stopped in the middle :

wget -c 

Download in the Background :

wget -b

You can check status by running :

tail -f wget-log

Make wget to be used from a browser :

wget –user-agent=”Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3″

Test Download url :

wget –spider

Download from a list :

wget -i download_list.txt

Abort download if greater than certain size :

wget -Q5m -i

Download only files of certain types :

wget -r -A.pdf 

Download from ftp:

wget (annonymous)

wget –ftp-user=USERNAME –ftp-password=PASSWORD (user/pass)

Mirror a website :

wget –mirror -p –convert-links -P ./local_dir <website_to_download>

–mirror : turn on options suitable for mirroring.
-p : download all files that are necessary to properly display a given HTML page.
–convert-links : after the download, convert the links in document for local viewing.
-P ./LOCAL-DIR : save all the files and directories to the specified directory.

Other Options :

–tries=

–reject=gif (Reject file type gif)

wget -o download.log ( log to a file instead of console)