A Comprehensive Guide to the Linux wget Command for Beginners

Introduction

In this article, we will explore the Linux wget command in detail, covering its history, use cases, and how to use it effectively. We will also discuss its commonly used parameters, other supported parameters, and some tricky skills.

History

The wget command was initially created by Hrvoje Nikšić in 1996 and has since become a popular tool for downloading files from the internet on Unix-based systems.

When and why to use it

wget is a command-line utility that allows users to download files from the internet. It supports HTTP, HTTPS, and FTP protocols, and is best used for downloading files non-interactively in the background, mirroring websites, or even fetching data from APIs.

How to use it

To use wget, simply provide the URL of the file you wish to download, and wget will save it to the current directory.

wget [<http://example.com/file.txt>](<http://example.com/file.txt>)

The commonly used parameters

• -O Save the downloaded file with a specific name.

wget -O new_file.txt [<http://example.com/file.txt>](<http://example.com/file.txt>)

• -P Specify the directory to save the downloaded file.

wget -P /path/to/directory [<http://example.com/file.txt>](<http://example.com/file.txt>)

• -c Resume an interrupted download.

wget -c [<http://example.com/large_file.zip>](<http://example.com/large_file.zip>)

• -r Download a website recursively, following all links.

wget -r [<http://example.com/>](<http://example.com/>)

• --limit-rate Limit the download speed.

wget --limit-rate=200k [<http://example.com/file.txt>](<http://example.com/file.txt>)

Other supported parameters

• -q Quiet mode, suppress output
• --no-check-certificate Don't check SSL certificates
• -N Download only if the file is newer than the local copy
• --no-clobber Don't overwrite existing files
• -i Download files listed in a specified text file

Most common use cases

• Downloading files non-interactively
• Mirroring websites for offline browsing
• Fetching data from APIs or scraping web content

wget [<http://api.example.com/data.json>](<http://api.example.com/data.json>)

The tricky skills

• Downloading files from a list of URLs stored in a file:

wget -i urls.txt

• Downloading files with specific extensions using pattern matching:

wget -r -A .pdf [<http://example.com/documents/>](<http://example.com/documents/>)

What needs to be noted

• Be aware of the server’s usage policies and limitations.
• Respect the robots.txt file on websites when using wget for web scraping.
• Be cautious when using wget with authentication or sensitive data, as command history may store sensitive information.

Conclusion

The Linux wget command is a powerful and versatile tool for downloading files and data from the internet. With its numerous parameters and use cases, it's an essential utility for any Linux user. This article has provided a comprehensive overview of wget, enabling you to utilize it effectively in various scenarios.