Building a HashiCorp Nomad Cluster Lab using Vagrant and VirtualBox

One of the greatest things about open source and free tools is that they are…well, open source, and free!  Today I’m sharing a simple HashiCorp Nomad lab build that I use for a variety of things.  The reason I want to be able to quickly spin up local lab infrastructure is so I can get rid of the mundane repetitive tasks.  Being able to provision, de-provision, start, and stop servers easily means you can save a ton of time and

Now for the walk through to get you to show just how easy this can be!  Skip ahead to whatever part of the steps you need below.  I’ve built every step so that you can easily do this if you’ve got no experience.

Step 1:  Getting the Tools

You’re going to need Vagrant and VirtualBox.  Each are available free for many platforms.  Click these links to reach the downloads if you don’t already have them.  There is no special configuration needed.  My setup is completely default.

HashiCorp Vagrant: https://www.vagrantup.com/downloads.html

Oracle VM VirtualBox: https://www.oracle.com/virtualization/technologies/vm/downloads/virtualbox-downloads.html or https://www.virtualbox.org/wiki/Downloads

Step 2:  Getting the Code

The GitHub repository which contains all the necessary code is here: https://github.com/discoposse/nomad-vagrant-lab

Just go there and use the Clone or Download button to get the URL and then do a git clone command:

git clone https://github.com/discoposse/nomad-vagrant-lab.git

If you want to contribute to the repository, you can also fork the repository and work from your own fork and then submit pull requests if you wish.

Step 3:  Configuring the lab for 3-node or 6-node

Making the choice of your cluster and lab size is easy.  One configuration (Vagrantfile.3node) is a simple 3-node configuration with one region and a single virtual datacenter.  The second configuration (Vagrantfile.6node) will create two 3-node clusters across two virtual datacenters (toronto and Vancouver) and across two regions (east and west).

Picking your deployment pattern is done by renaming the configuration file (Vagrantfile.3node or Vagrantfile.6node) to Vagrantfile

Seriously, it’s that easy.  If you want to update the names of the regions or datacenter then you will do that by editing the server-east.hcl and server-west.hcl files.  The 3-node configuration only uses server-east.hcl by default.

Step 4:  Launching your Nomad Lab Cluster

Launch a terminal shell session (or command prompt for Windows) and change directory to the folder where you’ve cloned the code locally on your machine.  Make sure that everything is all good by checking with the vagrant status command:


Next up is starting the deployment using the vagrant up command.  The whole build takes anywhere from 5-15 minutes depending on your speed of network and local machine resources.  Once you run it once you also have the Vagrant box image cached locally so that saves time in the future for rebuilds.

Once deployment is finished you’re back at the shell prompt and ready to start your cluster!

Step 5:  Starting the Nomad Cluster

The process is easy for this because we are using pre-made shell scripts that are included in the code.  Each VM comes up with the local Github repository mapped to the /vagrant folder inside the VM.  Each node has a file named after the node name.  It’s ideal if you have three (or six for a 6-node configuration) terminal sessions active so you can work

  1. Connect via ssh to the first node using the command vagrant ssh nomad-a-1
  2. Change to the code folder with the command cd /vagrant
  3. Launch the Nomad script with this command:  sudo sh launch-a-1.sh
  4. Connect via ssh to the second node using the command vagrant ssh nomad-a-2
  5. Change to the code folder with the command cd /vagrant
  6. Launch the Nomad script with this command:  sudo sh launch-a-2.sh
  7. Connect via ssh to the third node using the command vagrant ssh nomad-a-3
  8. Change to the code folder with the command cd /vagrant
  9. Launch the Nomad script with this command:  sudo sh launch-a-3.sh

The reason I’m detailing the manual clustering process is that this is meant for folks getting started.  I will run a different process to start services automatically in another blog.

Step 6 (Optional 6-node configuration):  Starting the Second Nomad Cluster

The process is just as easy for the second cluster because we are using pre-made shell scripts that are included in the code.  Each VM comes up with the local Github repository mapped to the /vagrant folder inside the VM.  Each node has a file named after the node name.  It’s ideal if you have three (or six for a 6-node configuration) terminal sessions active so you can work

  1. Connect via ssh to the first node using the command vagrant ssh nomad-b-1
  2. Change to the code folder with the command cd /vagrant
  3. Launch the Nomad script with this command:  sudo sh launch-b-1.sh
  4. Connect via ssh to the second node using the command vagrant ssh nomad-b-2
  5. Change to the code folder with the command cd /vagrant
  6. Launch the Nomad script with this command:  sudo sh launch-b-2.sh
  7. Connect via ssh to the third node using the command vagrant ssh nomad-b-3
  8. Change to the code folder with the command cd /vagrant
  9. Launch the Nomad script with this command:  sudo sh launch-b-3.sh

Step 7:  Confirming your Cluster Status

Checking the state of the cluster is also super easy. Just use the following command from any node in the cluster:

nomad server members

This is the output showing you have the 3 node cluster active:

Step 8:  More learning!

There are lots of resources to begin your journey with HashiCorp Nomad.  Not least of which is my freshly released Pluralsight course Getting Started with Nomad which you can view as a Pluralsight member.  If you’re not already a member, you can also sign up for a free trial and get a sample of the course.

Hopefully this is helpful and keep watching the blog for updates with more lab learning based on this handy local installation.




Talking HashiCorp with CEO Dave McJannet

This week at AWS re:Invent, I was lucky enough to be able to sit down with Dave McJannet, CEO (and fellow Canadian, eh!) of HashiCorp. As a huge fan of the HashiCorp products and story, this was especially enjoyable for me. I wanted to share some of the conversation and my thoughts with everyone here.

The HashiCorp Ecosystem

HashiCorp is the company that produces many of the products that we use today, whether you realize it or not. The array of products is astounding for a company of its size and age.  There are a variety of open source products:

open-source-hashicorp

There are also the enterprise suite of products:

enterprise-hashicorp

The first homework assignment I’ll give you is to go and read this article on the HashiCorp blog: https://www.hashicorp.com/blog/hashicorp-devops-and-the-application-delivery-process.html

To give a simplified view of it, we see the three stages and a few key items that are a part of each phase.

provsion-secure-run-hashicorp

Provision

  • How do you do your infrastructure provisioning?
  • Bare-metal? Virtualization layer? Public cloud? Container? Any or all of the above?
  • Image packaging and creation
  • Programmatic, composable methods

Secure

  • Secret management
  • Security in volatile and ephemeral environments
  • Programmatic, composable methods

Run

  • Launch and discover instances, VMs, and containers
  • Programmatic, composable methods

You can see the common theme that I’ve highlighted. Programmatic, composable methods. The key feature across all of the different products in the HashiCorp suite is the ability to programmatically create and manage infrastructure. The next generation of applications will need this. Truthfully, today’s infrastructure needs it too, but we still have some traditional hand-craft infrastructure that has held on and continues to operate in many, many environments.

Solving the Workflow Challenge

Every piece in the HashiCorp puzzle shares a common goal: solve the workflow challenge.

Workflow challenges come in the ways that we do things including (but not limited to, of course):

  • Provisioning our underlying infrastructure
  • Deploy on multiple physical, virtual, and cloud infrastructure platforms using common methods
  • Build applications to deploy on heterogeneous infrastructure
  • Continuous management from infrastructure to application layers

As I covered in a recent post around the Top 3 Skills that Every IT Person Needs, the key focus was on automation, orchestration, and getting closer to a developer’s view of infrastructure. I also highlighted how important security is at every single level.

This is where the HashiCorp story becomes so important. Again, the lifecycle approach that is being taken across the suite of products helps to illustrate how it all can fit together.

lifecycle-hashicorp

Interchangeable Lifecycle Products

The full stack has a view like this:

full-stack-hashicorp

What you’ll find interesting around how the full stack looks across many of the current HashiCorp customers and community product users. There are many interchangeable products that fit in amongst the layers. There are products that fill out the overall suite, or you can also cleanly integrate with your own infrastructure solutions.

Heterogeneity is a fundamental part of the story. I was especially impressed with the candid chat with Dave and he noted that “our reality is around heterogeneity, which means we need to be able to provision, secure, and run on any infrastructure” which is echoed in all of the product documentation, the well-stated Tao of HashiCorp, and in conversation with folks using the HashiCorp products.

There’s a reason that great community advocates and technologists like Kelsey Hightower of Google will often feature HashiCorp products in his talks and demos. They fit into important places, and solve very specific and interesting challenges.

Next Steps

Hopefully that gives you an idea of the story that is important to how HashiCorp products come together in general. You’re not done yet, because your second piece of homework is the DevOps blog article: https://www.hashicorp.com/devops.html

If you walk away from all of this with anything, it should be that you need to look at what you’re doing today, and how you can look towards products like the HashiCorp suite of products to solve challenges within your overall application and infrastructure lifecycle management.

If you don’t think that you have challenges, you just haven’t looked close enough. If you don’t believe that HashiCorp is an important part of what you could be doing, you also haven’t looked close enough. Not because they are the only ones attacking these challenges, but because the companies you use today for banking, infrastructure hosting, cloud management, and more, are already using Consul, Terraform, Vault, Packer and other products.

I’m looking forward to seeing what’s in the future for the company and the products. Look for some technical HowTo blogs in the near future as I give you some insight to how I use the products already today.




Using the –no-provision directive for Vagrant when resuming machines

It seems that there is an active issue when resuming machines using Vagrant that triggers provisioning scripts on resume and not just when doing the original vagrant up command.

EXAMPLE:

vagrant-resume-provisioned

The issue here is that I already have the build script executed on the original vagrant build of the machine.  The scripts may not be idempotent, and could overwrite content or damage the active machine.

In our Vagrant file, we use the provision capability regularly, so we would not want to have to build all sorts of logic around that unless necessary because Vagrant did this natively in the past.

provision-script

Workaround Using the –no-provision Parameter

Rather than run a vagrant resume as you saw above which triggered the build script again, you can simply use a vagrant up --no-provision which will bring the machine up and reconnect any SSH connections and NFS shares, but it will ignore any provision directives from the Vagrantfile:

vagrant-noprovision

Hopefully this will be solved in a patch or future update to Vagrant.  The post deals specifically with version 1.8.1 that presented the problem.  It may also be present in other versions.




Deploying a Docker Sandbox using Vagrant and VirtualBox

Even with the addition of more client-side tools for running Docker, it still requires installing development tools that may impact your local environment. Plus, I’m a fan of using VirtualBox to make sandbox environments that I can spin up and tear down as needed. This means that I never have to worry about version conflicts with development languages and dealing with package management on my Macbook or on my Windows machine. Every machine will use the same recipe.

Deploying the VirtualBox Docker Sandbox Environment

We are assuming you’ve read the initial post, you have already installed Vagrant, VirtualBox and Git. Once you are up to that point and ready to go, we just need to pull down the code to deploy our Docker sandbox using git clone https://github.com/discoposse/virtualbox-docker-sandbox.git first:

git-clone

Next, change directory into the folder cd virtualbox-docker-sandbox and then run the vagrant up command:

cd-vagrant-up

That’s going to run for a while, and you will see after about 10 minutes that you’re back at the prompt. Now we can use vagrant ssh dockersandbox to get onto the console and confirm our environment:

ssh

You’ve got the nested instance running, and as you may have read in the build.sh file from the code we cloned, we have installed docker as well as docker-compose and docker-machine which will help us test the waters on a few different Docker features:

docker-versions

Now you can start up some Docker containers without having touched your local machine at all, and this ensures a clean sandbox deployment for you that is able to be quiesced using the vagrant suspend dockersandbox command:

suspend

To bring the environment back up, just run a vagrant resume dockersandbox

Running Docker as Sudo

One thing to note is that we have deployed the Docker runtime as sudo, and because of that we must always run the Docker commands as sudo. This is a best practice in general, so when you try to run docker pull or any docker commands, you will see errors about the daemon not being available:

docker-fail

Run the sudo docker pull nginx instead of docker pull nginx and you will see a marked difference in the results:

docker-pull

Happy Dockering!