Accessing DigitalOcean Droplets via Command Line Using SSH Keys on OSX

As you get rolling with using DigitalOcean and other VPS providers, one of the features that many folks see in the configuration is the “user SSH key to access your instance” options. The trick is that many newcomers to using cloud instances aren’t totally comfortable or fully understand setting up an SSH key for password-less access to your instance.

Is it Secure Without a Password?

A resounding yes! In fact, it’s much more secure. You’ve uploaded the public side of your key to the instance already from within the cloud infrastructure and you’re now using the private side to match up for access. By not using a password, you’re removing the risk of sending authentication information over the public network. Brute force attacks are not as effective with public/private key pairs whereas they are successful in password hacking attempts.

It’s assumed that you’ve already uploaded your key. I won’t dig into all the different providers and ways to upload the keys. Make sure to do that for your individual provider to create and upload a key from your machine.

Adding your key to the SSH agent from the command line for OSX

When you launch your instance through the GUI, make sure that you have a SSH key selected to match the private key you have on your local machine. I’ve nicknamed mine as Eric-MacbookPro. For extra safety, I also keep copies of the keys in an offsite vault to ensure that I never lose access to the instances that are attached to that key.

When your DigitalOcean droplet is launched, the key is added as part of the init process. Once you have your IP address, you just have a quick process to run to set your key up. Because I use a key that is stored in a folder that isn’t the default, it has to be added to the ssh agent.

Run the eval `ssh-agent -s` command. NOTE: those are backpacks, not apostrophes. That character is found on the same key as the tilde (~) symbol.

The second command you run is ssh-add [yourkeyname] where [yourkeyname] is the full filename and path of your private key. IN my case, I have it stored in my Documents folder under a keys subfolder. This is my process:

ssh-add ~/Documents/keys/id_rsa

Connecting to your DigitalOcean Droplet via SSH with your Private Key

Now we simply run the command line ssh using the administrative account. For CentOS and Ubuntu on DigitaiOcean, it is the root account. For CoreOS instances, you use the core account.

My Ubuntu instance is accessible now by the ssh root@ip-address:

Now you’re in! Keep your keys safe, and keep your DigitalOcean droplets safe with those keys. Happy SSHing!




Downloading Files with cURL, Including Text and Binary Objects

Many orchestration and automation processes will need to download content from external or internal sources over protocols like HTTP and FTP. The simple way to do this is to leverage lightweight, commonly supported and available tools. The most common and popular tool that I’ve found for managing scripting of downloads inside configuration management systems and other script management tools is with cURL.

What is cURL?

cURL stands for command Line URL and is a simple, yet powerful, command line utility that gives the ability to download content using a lightweight executable that provides cross-platform support. cURL is community supported and is often a packaged part of some *nix systems already.

You can download revisions of cURL for a varying set of platforms from https://curl.haxx.se/download.html even including AmigaOS if you so desire 🙂

Why use cURL?

The most common comparative tool to cURL is wget. There is a fully featured matrix of options that are available across a number of different tools, but for simplicity, cURL and wget tend to be the goto standards for *nix and Windows systems because of the small footprint and flexibility.

cURL and wget have many similarities including:

  • download via HTTP, HTTPS, and FTP
  • both command line tools with multiple platforms supported
  • support for HTTP POST requests

cURL does provide additional feature support that isn’t available from wget including:

  • many protocols including DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling and more. (source: curl.haxx.se)
  • API support with using libcurl across platforms

Let’s take a look at our example code to see how to make use of cURL.

Download Files With cURL

As we discussed, one of the most useful thing to do with curls is to download a file with cURL.  You can download multiple different file types with cURL, so now let’s take a closer look at how to do this, paying special attention to parameters we need to download different file types.

cURL, Download File type HTML or Text

It’s frighteningly simple to download text and non-binary objects with cURL. You simply use this format:

curl <source URL>

This will download the target URL and output to STDOUT, which will be to the console in most cases.

curl-google

If you wanted to output it to a file, you just add -o to the command line with a target file name (note: that is a lower case o):

curl-google-file

Downloading Binary content with cURL

If we had a binary file, we obviously can’t write it to STDOUT on the console, or else we would get garbage output. Let’s use this image as an example:

target-still-real

The raw URL for this file is https://raw.githubusercontent.com/discoposse/memes/master/StillReal.jpg to use for our examples.

Using the same format, we would try to do a curl https://raw.githubusercontent.com/discoposse/memes/master/StillReal.jpg which gives us this rather ugly result:

curl-binary-fail

To download a binary file, we can use the -O parameter which pulls down the content exactly as the source file specified dictates, including the name as such:

curl -O https://raw.githubusercontent.com/discoposse/memes/master/StillReal.jpg

curl-binary-success

It isn’t that the -O was required to succeed, but it meant that it treated the output exactly as the input and forced it to output to a file with the same name as the source on the target filesystem. We can achieve the same result by using the -o parameter option also and specifying a target filename:

curl https://raw.githubusercontent.com/discoposse/memes/master/StillReal.jpg -o StillReal.jpg

binary-specified-name-same

This is handy if you want to change the name of the file on the target filesystem. Let’s imagine that we want to download something and force a different name like anyimagename.jpg for example:

curl https://raw.githubusercontent.com/discoposse/memes/master/StillReal.jpg -o anyimagename.jpg

binary-specified-any-name

You can pass along all sorts of variable goodness into the name for output when you want to do something programmatically, which is all sorts of awesome as you get to using cURL for automated file management and other neat functions.

We will tackle more cURL options again in the future, but hopefully this is a good start!




Clean up Exited Docker Containers With Docker PS, AWK, and Grep

As we start to mess around more with Docker, one of the things that I have to do regularly is to purge out the containers that are exited and unused. I will put the disclaimer here that this is a way to remove ALL exited containers. In other words, any container that is not running in detached mode will be destroyed. This is very handy for development systems where you test out lots of containers, and we will learn one of the most useful Docker commands out there – docker ps!

Docker PS and Pattern Matching the Awk-ward Way

I won’t dive into the goodness of awk, but I do recommend that you get familiar with it as it is a powerful command line tool to do pattern matching and parsing of data. First, we will take a look at how to view containers that are running.

This is done with the docker ps command, which simply lists containers, and you can can pass some parameters to it.  docker ps is one of the most useful commands you will use in docker.

Running the Docker PS Command

By default, docker ps shows you the active containers:

docker-ps-running

Since we know we have been mucking around with other containers that are not running, we need to run docker ps -a to view all containers:

docker-ps-all

Adding Options to the Docker PS Command

We can also add a filter option to the command by using docker ps -a -f STATUS=exited to give os the list of containers which have been exited:

docker-ps-notrunning

Combining awk and docker ps

Let’s refine the results using awk to pull only what we need to pass to our docker rm command to remove the containers. In this case, we want the container ID which is the first column value. We will use the docker ps -a -f STATUS=exited | awk '{print $1}' command to do that:

docker-ps-containerid-list

You can use a lower case for the filter parameter, but for some reason I like to use it the way it displays in the table. So, now that we have a command that can give us a list of the container IDs that we want to remove, there is only one step. Let’s remove all of these using the docker rm $(docker ps -a -f STATUS=exited | awk '{print $1}') command.

This command does a docker remove for all values pulled from the docker ps command:

docker-ps-rm-awk

You can see that there will be an error thrown because we pulled the entire first column including the header which said “CONTAINER”, so the docker rm command tripped on this one.

We can confirm the results by running our docker ps -a and you will see that now we only have our running containers in the list:

docker-ps-all-clean

You could also do further limiting by adding a grep command. Perhaps we only want to remove exited containers from the Ubuntu image. That can be done using docker rm $(docker ps -a -f STATUS=exited | grep -i ubuntu | awk '{print $1}') as an example:

docker-rm-ubuntu

Between awk and grep you can do some pretty powerful stuff with a little practice. Hopefully this is a helpful tip for you.

More About docker ps

docker ps is one of the most useful and commonly used commands in Docker, since it lists containers.  Remember, if you want to list ALL containers, including running and stopped, use the docker ps -a command.

Be sure to check out the Docker Documentation for the official and complete docker ps reference.