Downloading Text and Binary Objects with curl

Transcription

1 Downloading Text and Binary Objects with curl Many orchestration and automation processes will need to download content from external or internal sources over protocols like HTTP and FTP. The simple way to do this is to leverage lightweight, commonly supported and available tools. The most common and popular tool that I ve found for managing scripting of downloads inside configuration management systems and other script management tools is with curl. What is curl? curl stands for command Line URL and is a simple, yet powerful, command line utility that gives the ability to download content using a lightweight executable that provides crossplatform support. curl is community supported and is often a packaged part of some *nix systems already. You can download revisions of curl for a varying set of platforms from even including AmigaOS if you so desire Why use curl? The most common comparative tool to curl is wget. There is a fully featured matrix of options that are available across a number of different tools, but for simplicity, curl and wget tend to be the goto standards for *nix and Windows systems because of the small footprint and flexibility. curl and wget have many similarities including: download via HTTP, HTTPS, and FTP both command line tools with multiple platforms supported support for HTTP POST requests curl does provide additional feature support that isn t available from wget including: many protocols including DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more. (source: curl.haxx.se) API support with using libcurl across platforms Let s take a look at our example code to see how to make use of curl. Downloading HTML or Text with curl It s frighteningly simple to download text and non-binary objects with curl. You simply use this format: curl <source URL>

2 This will download the target URL and output to STDOUT, which will be to the console in most cases. If you wanted to output it to a file, you just add -o to the command line with a target file name (note: that is a lower case o): Downloading Binary content with curl If we had a binary file, we obviously can t write it to STDOUT on the console, or else we would get garbage output. Let s use this image as an example:

3 The raw URL for this file is to use for our examples. Using the same format, we would try to do a curl which gives us this rather ugly result:

4 To download a binary file, we can use the -O parameter which pulls down the content exactly as the source file specified dictates, including the name as such: curl -O It isn t that the -O was required to succeed, but it meant that it treated the output exactly as the input and forced it to output to a file with the same name as the source on the target filesystem. We can achieve the same result by using the -o parameter option also and specifying a target filename: curl -o StillReal.jpg

5 This is handy if you want to change the name of the file on the target filesystem. Let s imagine that we want to download something and force a different name like anyimagename.jpg for example: curl -o anyimagename.jpg You can pass along all sorts of variable goodness into the name for output when you want to do something programmatically, which is all sorts of awesome as you get to using curl for automated file management and other neat functions. We will tackle more curl options again in the future, but hopefully this is a good start!