AWS Summit Toronto Presentation: Open-source Infrastructure-as-Code with Terraform and AWS

This is the summary post which will cover my Open-source Infrastructure-as-Code with Terraform and AWS presentation that I gave at the AWS Summit in Toronto.  The landing link from my presentation may have brought you here and this post has all the links to the Github repositories, the PDF of the presentation, and also the video of the entire presentation.

Make sure to drop in a comment if you have any questions and want to get connected.  You can set yourself up for Terraform Cloud at app.terraform.io 

Video Replay

Here is the video of my session that I recorded based on the presentation at the AWS Summit in Toronto

Presentation File

Here is the presentation which the team at AWS has confirmed can be shared.  Big thanks go to @RossBarich for allowing me to present at the event and to @mistwire for getting me connected with Ross.

Click here to view the PDF file: https://discoposse.com/aws-summit-toronto-2019-ericwright-pdf/

Code Links

These are the links to my Github repos used in the presentation examples.  Note that some code is currently static references to some resources which will be made into completely variable-driven resources shortly:

Simple EC2 image using Terraform Cloud: https://github.com/discoposse/terraform-cloud-aws-summit-toronto

Classroom example using EC2 from AMI with ALB and TLS: https://github.com/discoposse/terraform-cloud-aws-toronto-classroom-example

 




Getting Terraform Provisioning Parameters from the Packet.net API

Provisioning on Packet.net is super easy using Terraform. One of the tricks you will need to know up front is that for Terraform and for many other provisioning tools, you need to provide a minimum set of parameters to launch.

As a minimum, you need to provide these following parameters as shown in the Terraform docs for the Packet provisioner:

  • hostname – gotta name ’em all
  • project_id – you need to know, or create the project to launch into
  • facility – which location are you deploying into? (EWR1, SJC1, etc.)
  • plan – which node type?
  • billing_cycle – hourly or monthly
  • operating_system – which OS will the node run?

Some are simple to use because they are your criteria. We choose the hostname, and we choose the billing cycle as either a static choice of hourly or monthly. How can we get the other details about our deployment? You can gather some data using a browser such as browsing to your project and then pulling the project ID from the URL. That still leaves us in search of the plan type, operating_system, and facility.

For completeness, let’s learn how to simply gather all four items (operating system list, project ID, plan types, facility) from the Packet.net API.

You’ll need a terminal session, your API key to query the Packet.net API, and the JQ tool for parsing out JSON results into something a little more friendly.

Querying the API is as easy as sending your token to the API using the cURL command and selecting which entities you want to query. This is the basic framework:

curl -s -X GET -H 'X-Auth-Token: YOURAPITOKEN' 'https://api.packet.net/OBJECT'

Now we can dig into the four easy examples we have.

Finding the Packet.net Facility Name

The simple one-liner will pull a JSON result that gives you the locations and subsequent Facility name that you can use and then parses out just the location codes to use. If you remove the '.facilities[].code' portion of the command it will show you the full pretty-printed JSON results including the full facility descriptions.

curl -s -X GET -H 'X-Auth-Token: YOURAPITOKEN' 'https://api.packet.net/facilities' | jq '.facilities[].code'

Finding the Packet.net Project ID

You’ll want the full JSON result so you can choose from your active projects if you have more than one. Just drill into the JSON results and you can locate the id field:

curl -s -X GET -H 'X-Auth-Token: YOURAPITOKEN' 'https://api.packet.net/projects' | jq

Finding the Packet.net Plan Names

Plans don’t shift around too much, just like facilities. Here is the simple query to get all the plan names and match them to what node type you want to use:

curl -s -X GET -H 'X-Auth-Token: YOURAPITOKEN' 'https://api.packet.net/plans' | jq '.plans[].slug'

Finding the Packet.net Operating System Types

By now, you can guess where we are going wth the next one. Query the API, parse out the results, and provide the slugs for the Operating System names which we will use for Terraform and other provisioning tools which consume the Packet API.

curl -s -X GET -H 'X-Auth-Token: YOURAPITOKEN' 'https://api.packet.net/operating-systems' | jq '.operating_systems[].slug'

The result will give you all of the slug names that are usable as the operating_system parameter. In the case of vSphere 6.5, it happens to be vmware_esxi_6_5 which may not have been obvious if you were to try guessing it out.

Now you can take those easy JSON results and feed them into a Terraform file or you may also use these raw queries as part of other configuration management and provisioning solutions. Hope you find this helpful!

Also, you can sign up for Packet.net to kick the tires on this goodness and you can use VDM25 as a referral code to get a 25$ credit to use. Make sure you tell them DiscoPosse and the Virtual Design Master crew sent you!




Using Terraform to Install DevStack on DigitalOcean

There are a few times where having a persistent OpenStack lab on a shared infrastructure is handy. I’ve been revisiting DevStack a lot more lately in order to help a few folks get their labs up and running. DevStack is the OpenStack project which lets you run non-production OpenStack using either a single or a multi-node configuration. Running on DigitalOcean means that I can have a lab that can spin up quickly (about 40 minutes) and also lets me find another handy use for Terraform.

NOTE: This uses an 80$/month DigitalOcean droplet, so please keep that in mind as you experiment.

Requirements for this are:

Getting the Code

All of the scripts and configuration are on GitHub for free use and are also open for contributions and updates if you see anything that you’re keen to add to. Remember that Terraform uses state files to manage your environment, so when you pull down the GitHub repo and launch your environment, it will create the .tfstate and .tfstatebackup files after you launch for the first time.

Grab the code using git clone https://github.com/discoposse/terraform-samples to bring it down locally:

Change directory into the /terraform-samples/DigitalOcean/devstack folder where we will be working:

Make sure you have the environment variables setup including the DigitalOcean API token, SSH key file locations, and your SSH fingerprint. These can be exported into your environment using a script or as one-off commands:

The process that is run by the code is to:

  • Pull the DigitalOcean environment needs (API and SSH info)
  • Launch an 8 GB RAM droplet in the NYC2 region and attach your SSH fingerprint
  • Insert the DevStack build script (files/devstack-install.sh) as a cloud-init script

Those are the pre-requirements. Now it’s time to get started!

Launching the DevStack Build on DigitalOcean with Terraform

It’s always good to use a health check flow of your Terraform builds. Start by validating, running the plan, and then launching. This ensures that you have a good environment configuration and the process should work smoothly.

terraform validate

No news is good news. The code validated fine and we are ready to run the terraform plan command to see what will transpire when we launch the build:

We can see a single droplet will be created because we have nothing to start with. There are a number of parameters that are dynamic and will be populated when the environment launches. Time to go for it!

terraform apply

This is where you need a little bit of patience. The build takes approximately 45-60 minutes. We know the IP address of the environment because we requested it via the Terraform outputs. You can confirm this at any time by running the terraform output command:

Checking the DevStack Install Progress using the Cloud-Init Log

Let’s connect via SSH to our DigitalOcean droplet so we can monitor the build progress. We use the build script as a cloud-init script so that it launches as root during the deployment. This means you can keep track of the results using the /var/log/cloud-init.log and the /var/log/cloud-init-output.log files.

Install completion is indicated by a set of log results like this:

Let’s try it out to confirm using the OpenStack Horizon dashboard URL as indicated in the cloud-init output. There are two accounts created by the script which are admin and demo, both of which have secret-do as the default password.

NOTE: Please change your OpenStack passwords right away! These are simple, plain-text passwords that are packaged with the build and you are vulnerable to attack

That gets us up and running. You are incurring charges as long as the environment is up, so when you’re ready to bring the environment down and destroy the droplet, it’s as easy as it was to launch it.

Destroying the DevStack DigitalOcean Build Using Terraform Destroy

In just two quick words and a confirmation we can remove all of the environment: terraform destroy

Just like that we have installed an all-in-one OpenStack DevStack node on DigitalOcean and learned another nifty way to leverage Hashicorp Terraform to do it.




Adding SSH Access for DigitalOcean when Using Terraform

We’ve been looking at how to add a little Terraform into your IT infrastructure provisioning toolkit lately. Using DigitalOcean is also super easy and inexpensive for testing out processes and doing things like repetitive builds using Terraform.

The first post where we saw how to do a simple Terraform environment build on DigitalOcean appeared at my ON:Technology blog hosted at Turbonomic. That gave us the initial steps for a quick droplet deployment.

We also talked about how to access your DigitalOcean droplets via the command line using SSH keys here which is very important. The reason that it is important is that without SSH keys, you are relying on using the root account with a password. DigitalOcean will create a complex password for you when deploying your droplet. This is not something you can find out without actually resetting the root password and restarting your droplet. This is both insecure (reverting to password access instead of SSH key pair) and also disruptive because you are rebooting the instance to do the password reset.

Now it’s time to merge these two things together!

Adding SSH key details to the Terraform DigitalOcean provider

We are going to add a few things to what we have already done in those two other posts. You will need the following:

Getting your SSH fingerprint is a simple process. Start by going to the top right of your DigialOcean console to the icon which has a dropdown for your account settings:

In the profile page, choose the Settings option from the menu on the left-hand panel:

The SSH fingerprint that you’ll need is in the security settings page. Keep this somewhere as safe as you would your SSH keys themselves because this is an important piece of security information.

Using the SSH Details in Environment Variables

Our settings are going to be stored using local environment variables just like with our DigitalOcean key was in the first example blog. Because we have a few other things to keep track of now we will see the changes in the provider.tf file:

Our environments variables are going to have the same format which is TF_VAR_digitalocean_ssh_fingerprint which is your fingerprint you got from the security settings. The other two things we need are the TF_VAR_digitalocean_pub_key and TF_VAR_digitalocean_private_key parameters which are the paths to your local SSH key files.

NOTE: The use of the file locations is actually not needed for basic key configuration using Terraform. I just thought we should set that up which will come to use later on in other blogs around using Terraform with DigitalOcean.

Use the export command to sett up your variables.  Our Terraform file contains an extra config parameter now which you’ll see here:

These new parameters will read in all that we need to launch a new droplet, attach the appropriate SSH key by the fingerprint in DigitalOcean, and then to allow us to manage the infrastructure with Terraform.

Time for our routine, which should always be: terraform validate to confirm our syntax is good followed by a terraform plan to test the environment:

Now we run our terraform apply to launch the droplet:

Now we have launched a droplet on DigitalOcean with Terraform. Use the SSH command line to connect to the droplet as the root account. Make sure you’ve done all the steps in the previous blog to set up your ssh-agent and then you should be all set:

This is the next step in making more secure, repeatable, and compassable infrastructure using Terraform on DigitalOcean. These same methods will also be showing up as we walk through future more complex examples on DigitalOcean and other providers.

Let’s clean up after ourselves to make sure that we take advantage of the disposable and elastic nature of our public cloud infrastructure by very easily running the terraform destroy command to remove the droplet:

Hopefully this is helpful!