Resetting vSphere 6.x ESXi Account Lockouts via SSH

VMware vSphere has had a good security feature added since vSphere ESXi 6.0 to add a root account lockout for safety. After a number of failed login attempts, the server will trigger a lockout. This is a good safety measure for when you have public facing servers and is even important for internally exposed servers on your corporate network. We can’t always assume that it is external bad actors who are the only ones attempting to breach your devices.

VMware vSphere ESXi Root Account Locked – Where to Start

Using the vSphere web client shows us the settings which are used to define the lockout count and duration. The parameters under the Advanced settings are as follows:

Security.AccountLockFailures
Security.AccountUnlockTime

Are you seeing this dreaded message?

 remote access for esxi local user account ‘root’ has been locked for 900 seconds

Do not worry, you are in the right place.  Now, let’s look at what to do if your ESXi root account is locked.

Resetting your ESXi Failed Login Attempts with pam_tally2

There is a rather simple but effective tool to help you do this. It’s called pam_tally2 and is baked in with your ESXi installation. The command line to clear the lockout status and reset the count to zero for an account is shown here with the root account as an example:

pam_tally2 --user root --reset

In order to gain access to do this, you will need to have SSH access or console access to your server. Console access could be at a physical or virtual console. For SSH access, you need to use SSH keys to make sure that you won’t fall victim to the lockouts for administrative users. In fact, this should be a standard practice. Setting up the SSH keys is relatively simple and is nicely documented in the Knowledge Base article Allowing SSH access to ESXi/ESX hosts with public/private key authentication (1002866)

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002866

Uploading a key can be done with the vifs command as shown here:

https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-392ADDE9-FD3B-49A2-BF64-4ACBB60EB149.html

The real question will come as to why you have the interface exposed publicly. This is a deeper question that we have to make sure to ask ourselves at all times. It’s generally not recommended as you can imagine. Ensuring you always use complex passwords and 2-factor authentication is another layer which we will explore. Hopefully this quick tip to safely reset your accounts for login is a good first step.




Adding SSH Access for DigitalOcean when Using Terraform

We’ve been looking at how to add a little Terraform into your IT infrastructure provisioning toolkit lately. Using DigitalOcean is also super easy and inexpensive for testing out processes and doing things like repetitive builds using Terraform.

The first post where we saw how to do a simple Terraform environment build on DigitalOcean appeared at my ON:Technology blog hosted at Turbonomic. That gave us the initial steps for a quick droplet deployment.

We also talked about how to access your DigitalOcean droplets via the command line using SSH keys here which is very important. The reason that it is important is that without SSH keys, you are relying on using the root account with a password. DigitalOcean will create a complex password for you when deploying your droplet. This is not something you can find out without actually resetting the root password and restarting your droplet. This is both insecure (reverting to password access instead of SSH key pair) and also disruptive because you are rebooting the instance to do the password reset.

Now it’s time to merge these two things together!

Adding SSH key details to the Terraform DigitalOcean provider

We are going to add a few things to what we have already done in those two other posts. You will need the following:

Getting your SSH fingerprint is a simple process. Start by going to the top right of your DigialOcean console to the icon which has a dropdown for your account settings:

In the profile page, choose the Settings option from the menu on the left-hand panel:

The SSH fingerprint that you’ll need is in the security settings page. Keep this somewhere as safe as you would your SSH keys themselves because this is an important piece of security information.

Using the SSH Details in Environment Variables

Our settings are going to be stored using local environment variables just like with our DigitalOcean key was in the first example blog. Because we have a few other things to keep track of now we will see the changes in the provider.tf file:

Our environments variables are going to have the same format which is TF_VAR_digitalocean_ssh_fingerprint which is your fingerprint you got from the security settings. The other two things we need are the TF_VAR_digitalocean_pub_key and TF_VAR_digitalocean_private_key parameters which are the paths to your local SSH key files.

NOTE: The use of the file locations is actually not needed for basic key configuration using Terraform. I just thought we should set that up which will come to use later on in other blogs around using Terraform with DigitalOcean.

Use the export command to sett up your variables.  Our Terraform file contains an extra config parameter now which you’ll see here:

These new parameters will read in all that we need to launch a new droplet, attach the appropriate SSH key by the fingerprint in DigitalOcean, and then to allow us to manage the infrastructure with Terraform.

Time for our routine, which should always be: terraform validate to confirm our syntax is good followed by a terraform plan to test the environment:

Now we run our terraform apply to launch the droplet:

Now we have launched a droplet on DigitalOcean with Terraform. Use the SSH command line to connect to the droplet as the root account. Make sure you’ve done all the steps in the previous blog to set up your ssh-agent and then you should be all set:

This is the next step in making more secure, repeatable, and compassable infrastructure using Terraform on DigitalOcean. These same methods will also be showing up as we walk through future more complex examples on DigitalOcean and other providers.

Let’s clean up after ourselves to make sure that we take advantage of the disposable and elastic nature of our public cloud infrastructure by very easily running the terraform destroy command to remove the droplet:

Hopefully this is helpful!




Accessing DigitalOcean Droplets via Command Line Using SSH Keys on OSX

As you get rolling with using DigitalOcean and other VPS providers, one of the features that many folks see in the configuration is the “user SSH key to access your instance” options. The trick is that many newcomers to using cloud instances aren’t totally comfortable or fully understand setting up an SSH key for password-less access to your instance.

Is it Secure Without a Password?

A resounding yes! In fact, it’s much more secure. You’ve uploaded the public side of your key to the instance already from within the cloud infrastructure and you’re now using the private side to match up for access. By not using a password, you’re removing the risk of sending authentication information over the public network. Brute force attacks are not as effective with public/private key pairs whereas they are successful in password hacking attempts.

It’s assumed that you’ve already uploaded your key. I won’t dig into all the different providers and ways to upload the keys. Make sure to do that for your individual provider to create and upload a key from your machine.

Adding your key to the SSH agent from the command line for OSX

When you launch your instance through the GUI, make sure that you have a SSH key selected to match the private key you have on your local machine. I’ve nicknamed mine as Eric-MacbookPro. For extra safety, I also keep copies of the keys in an offsite vault to ensure that I never lose access to the instances that are attached to that key.

When your DigitalOcean droplet is launched, the key is added as part of the init process. Once you have your IP address, you just have a quick process to run to set your key up. Because I use a key that is stored in a folder that isn’t the default, it has to be added to the ssh agent.

Run the eval `ssh-agent -s` command. NOTE: those are backpacks, not apostrophes. That character is found on the same key as the tilde (~) symbol.

The second command you run is ssh-add [yourkeyname] where [yourkeyname] is the full filename and path of your private key. IN my case, I have it stored in my Documents folder under a keys subfolder. This is my process:

ssh-add ~/Documents/keys/id_rsa

Connecting to your DigitalOcean Droplet via SSH with your Private Key

Now we simply run the command line ssh using the administrative account. For CentOS and Ubuntu on DigitaiOcean, it is the root account. For CoreOS instances, you use the core account.

My Ubuntu instance is accessible now by the ssh root@ip-address:

Now you’re in! Keep your keys safe, and keep your DigitalOcean droplets safe with those keys. Happy SSHing!




PowerCLI: Enabling and Disabling SSH on vSphere hosts

I’m all about running PowerCLI to replace GUI management. Every once in a while there is a requirement to access the ESXCLI on my vSphere hosts, and this is a classic task that is a multi-click process and if you’re using the old 4.x or 5.0/5.1 web client, you may find the process very tedious.

PowerCLI to the Rescue

My suggestion is that the first window you should open up on your workstation each day, and that is the PowerCLI shell to do your VMware administrative work. Here is a great way to use that nifty little shell.

First, let’s walk through the process that we need to follow.

  1. Attach to your vCenter server
  2. Query your cluster to find the hosts
  3. On each host, enable SSH by starting the TSM-SSH service

On paper, it looks pretty easy. The surprising thing is that in PowerCLI it is just as easy as that 3 point list. Even more exciting is the fact that we are going to use the magic of the pipeline to create a one-liner for the whole process.

So for item number one you will most likely do this when you launch your PowerCLI shell. This is done with the Connect-VIServer CmdLet:

Connect-VIServer <yourvCenterHost>

stop-ssh

This will prompt you for credentials as you can see here:

auth

Now we are authenticated and we can move to step 2 which is to query the cluster. The Get-Cluster CmdLet is just what the doctor ordered:

Get-Cluster -name <yourclustername>

get-cluster

This returns the name of the cluster and nothing more. So let’s add in a pipe to a Get-VMHost CmdLet to get back a list of our vSphere hosts in that cluster:

Get-Cluster -name <yourclustername> | Get-VMHost

get-cluster-get-host

Alright, we have gotten to this point which is where it gets a little tricky. Now we have to turn on the TSM-SSH service on each host, but because we have our hosts in a pipeline, we need to get a little bit fancy.

Starting SSH with PowerCLI on One Host

We have to walk before we run, so the first thing we need to do is understand how to turn on SSH on one host, and then we can use that method to apply to each host from the query that we got from step 2.

This section has some challenges because we are going to do a Start-VMHostService using results from a Get-VMHostService using results from a Where clause query. That’s a bit weird sounding, so let’s just get to the code and it should make sense.

Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq “TSM-SSH” } )

Hopefully it makes sense when it’s written out like that. The command blocks are in parentheses which should help to differentiate each section. The great thing about PowerCLI and PowerShell is that you can run a CmdLet and pipe content into it from another query and you can even nest queries which is both awesome and confusing.

With our nice nested query in place, we just have to make this run for each of our hosts in the cluster that we queried. Hmmm…how would we do this for each of the hosts (HINT: The CmdLet is right there for us!)

Gluing it all together with our ForEach Loop – The Final One-Liner

The command line has come together easily now in bits and pieces. The last step is to put it all together into a single command line:

Get-Cluster | Get-VMHost | ForEach {Start-VMHostService -HostService ($_ | Get-VMHostService | Where {$_.Key -eq “TSM-SSH”})}

Make sure that you keep track of the opening and closing brackets there because some of them are curly and some are round brackets.

Let’s see the results!

enable-ssh

Reversing the Process to Stop the SSH Services

Using the very same logic, we can replace the Start-VMHostService CmdLet with the Stop-VMHostService CmdLet and the SSH services will be stopped which is the default configuration for our vSphere hosts.

Get-Cluster | Get-VMHost | ForEach {Stop-VMHostService -HostService ($_ | Get-VMHostService | Where {$_.Key -eq “TSM-SSH”})}

One More Thing

columbo

If you ran the Stop-VMHostService one-liner you will notice that you are prompted in the shell to confirm the action for each host:

prompt

Assuming you know what it is that you are doing with this process, there is one simple parameter to drop into the command line on the Stop-VMHostService section which is -Confirm:$FALSE as we see here:

Get-Cluster | Get-VMHost | ForEach {Stop-VMHostService -HostService ($_ | Get-VMHostService | Where {$_.Key -eq “TSM-SSH”}) -Confirm:$FALSE}

stop-ssh

Now that we have that command together, we can start and stop SSH on our clustered hosts with ease. You can also do the same for a list of non-clustered hosts by removing the Get-Cluster and the first pipe symbol.

Happy SSH-ing!