The better alternative for Vault and AWS Secret Manager

Where do you store your application secrets? That’s the first question I ask many of our clients at ProdOps. I usually get one of two answers, either they thought I was talking about a user password manager like 1Password or LastPass, or I get really weird answers like Google Drive, AWS S3, environment variables in the code etc.

So I sharpen my question, asking where are API tokens are stored, RSA encrypted keys, and any other authentication methods our applications use to interact with external services like GCP, AWS, NPM, etc etc etc…

The untold truth is that 85% of the software companies store their application secrets hardcoded, or the better solution — a secret store in their CI server e.g TravisCI, Jenkins, Drone and the like.

The few saints that do take care of their security like pros, do it with Hashicorp’s Vault, which is an awesome product, and a most suitable solution for this exact use case.

But, Vault is somewhat tedious to begin with, it requires a lot of API work to configure it so that it’s really compartementalizing your services from one another, while using slim access policies that only allow read permissions, and from the single location that the application really needs.

Don’t get me wrong, I’m all in for Vault. I think it’s the best at its field of encrypting secrets and integrating with software applications. I wrote about it myself and I still stand behind my words. However, Vault requires work to deploy it correctly and to configure it with it’s internal policies and access management. I recently found out that AWS provide it as a free service of their own as part of their Systems Manager. It’s called “Parameter Store”. Being an AWS oriented shop, we decided to implement a solution that is natively integrated to IAM, allowing us to control access from one central place, not only to users and cloud deployed services, but to applications’ secrets as well.

Rumor has it that the AWS SSM Parameter Store is actually based on Vault, and when the guys over at Hashicorp heard about it, they obviously were not too excited about a competitor running their application as a service.

So? I still need to interact with an API, define roles and policies, how is that any better?

Well, not exactly. Yes, you still need handle access management, but it’s done via IAM. So assuming you’re already handling your cloud resources’ access via policies and roles, you can apply these with Parameter Store. The encryption is done using AWS KMS, so you get the encryption out of the box too.

And the API interaction to fetch and store parameters and secrets?

Good question! For that, I give you… “Chamber”

“Chamber is a tool for managing secrets. Currently it does so by storing secrets in SSM Parameter Store, an AWS service for storing secrets.”

  • github.com/segmentio/chamber

Chamber is a very convenient tool, that gives you very easy access to the Parameter Store, and it structures your work with KMS. It requires a KMS key and an alias (which takes around 50 seconds to set up) and you’ve got yourself a read/right abstraction layer on top of the API; it’s as easy as:

# Write:
$ chamber write <service> <key> <value>

# Read:
$ chamber read <service> <key>

Many other options are available but these are the core of the ability to easily interact of the system securly. Assuming your services are attached to IAM roles, you simply add permissions to read/write values to SSM Parameter Store, here’s an example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:DescribeParameters",
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DeleteParameters"
            ],
            "Resource": "*"
        }
    ]
}

Pros & Cons

I’ll handle this as if my two options are Vault and SSM. Both are great, each has it’s own pitfalls and advantages.

The SSM Parameter store is a free managed service, it means that financially you don’t have a reason not to use it, it requires no running resources except the KMS key, it’s basically free. On the other hand, it’s a service, so in case of cloud migration you’ll have to migrate your keys to a new system. Now, Hashicorp users tend to make the argument of “not being vendor locked” when using Vault / TerraForm and other tools. I can’t agree with this argument due to two major factors;

  1. You’re vendor locked to Hashicorp, I’ll take Amazon any day.
  2. Cloud migration is something done very rarely. Your secrets will be the least of your concerns if that’s the case.

Yet, if you’re on the edge case of running a hybrid setting of cloud providers, or use on-premise resources then 1. I feel for you, and 2. You can still use the SSM but you’ll have to manage your authentication less natively, e.g using one-time rotated access keys by AWS Cognito. But generally, if that’s the case then yes, Vault maybe the better option.

“But Vault has IAM backend access as a feature, even EC2!” Yes, that argument is valid, however, setting it up is no fun. You’ll need to create internal Vault policies allowing the access of a certain role to read and write using this backend authentication method, and then attach the role to your relevant resources. Again — no fun. Which is why if you’re AWS oriented, SSM is the more obvious solution, with easier safe access management. Tools like chamber exist for Vault, one of them is our own vault-get, but it still constitutes a single point of failure in the form of an injected token / key / password / whatever method is used to authenticate and pull data.

For these reasons, I chose SSM+Chamber as the AWS “native” alternative to Vault, and currently the cheaper alternative to the new AWS secret manager.

On the day of writing the last word of this post, AWS had announced their new secret manager. I don’t think it makes the parameter store obsolete, since it’s de-facto the best way of running AWS oriented secret managing service for free. The new secret manager is a great solution, but for $0.40 for secret per month, not to mention the API calls that read them, parameter store finds it’s place.

Let me know your thoughts in the comments below, or connect with me directly on Twitter @0merxx. Clap if you liked it, it helps me focus my future writings!

My name is Omer, and I am a consultant at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the Devops culture.

You may also enjoy