Using Vault to secure, encrypt and manage secrets
Last year, I’ve found my client’s AWS account access and secret keys pushed to git, and visible in 23 different locations in the application’s codebase. After requesting an immediate removal, I used the awesome GitLeaks tool to scan the project’s history. GitLeaks allows you scan a project’s entire history, rather than its current state so that it provides a very reliable output of the sensitive data hiding in your code.
The process yielded a 4.3MB of sensitive data, listing the same keys in 309 commits throughout the project’s history, along with private keys, private secrets used to communicate with cloud providers and other gems.
In the absence of tools, engineers tend to find their own solutions to everything. Whether it’s for delivery, storage, build-pipelines and unfortunately secrets and keys, developers require certain tools to surround them and provide solutions to their basic application needs.
A secret manager, is a tool that allows safe storage and encryption of sensitive data, providing access through strict and secure methods. Without it, keys are being stored on shared locations, hardcoded, committed and pushed to VCS. Sometimes it’s even publicly visible in the code downloaded to the client’s browser. #truestory Right, it’s not even remotely safe or done with the minimum level of security in mind, we can blame the responsible ones as much as we want, but educating is worthless without providing a proper alternative. Common sense should be trusted, but not with your company’s life, and yes, I mean that in the very literal way.
In order to illustrate how destructive a data leak can be, I used a few python lines exploiting AWS root keys as described in the first paragraph; I’m sure you understand this can be done just as easily with any language or the CLI. Please do not copy this code, let alone run it.
This code can be altered and used on any AWS resource the keys allow access to. Deleting instances, databases, users, DNS records and everything in its path!
To be very clear about the risk of such a key pair falling in the wrong hands, and without disclosing any names, it could have completely ruined the company with its 65 engineers and $50 Million in the bank, turning it to ashes within a single command. A tool like aws-nuke can also be handy for the bad guys. The destructive power of such a leak cannot be stressed enough.
Not convinced?
Here are some real world examples:
#1
A small startup in the retail business, discovering and comparing retail prices of premium brand top sellers, had been hacked four times in 90 days at the end of 2017. A private SSH key used by the production cluster was part of a library open sourced by the organization and aggressively marketed to create traction and buzz around their product. After each hack they rotated all AWS and GCP keys, and 3rd party tools access credentials including taking a security audit of their internal code base. Only a complete scan over the public code base exposed the leaked secret keys.
#2
Another small company in the mobile gaming field, used AWS S3 to store XML files. Upon launching their Android app, XML data from their AWS account was downloaded, using an AWS key. Only retrospectively, it turned out to be lacking any kind of monitoring on resources and monthly bills alike. In order to fetch the data, under the hood, the application used to authenticate with S3 using the account’s root credentials. When their new iOS application was launched, they were disapproved by Amazon’s Appstore, and communicated by Amazon describing the leak. Only then, they found some public unfamiliar instances running on their expense. Luckily, except for money loss, no real harm was done.
#3
One of the biggest known digital media and advertising companies in the world had discovered a very suspicious activity in their account on march 2017: instead of running their normal cluster ranging from 500 to 1000 instances every day, they were constantly growing and requesting Amazon to raise their instance count limits, assuming they were experiencing a natural growth, they thanked their marketing division for their fruit-baring efforts. The instance count curve went very steep in a single 24 hour period reaching a peak of 8000 R3 (intensive memory) instances. The activity was so out of the ordinary they were contacted by AWS, and found a Bitcoin mining cluster running on their account resulting in over $600,000 in charges, taking into account the entire month’s activity since the exploit started. In the post mortem investigation, it was discovered that one of their publicly accessible instances was left open, and contained a file you may be familiar with in the path ~/.aws/credentials allowing the use of all EC2 permissions, which was exploited to launch a huge cluster of machines by an opportunist hacker.
How can you mitigate the situation?
Based on the Pareto principle (a.k.a the 20–80 rule), by managing secrets within a single, secure, encrypted, yet reachable location, you can take care of the 20% that can potentially impose 80% damage.
The best example for such a solution, is Hashicorp’s Vault:
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
With Vault, solving the problem is easy. It is a secret manager that literally seals your keys in a safe. Every unplanned change in the vault system (such as crashing, multiple bad authentications, etc…) seals the system. It moves to a “frozen” state, and requires a minimum number of pre-configured personal keys to be inserted, in a very specific way to unseal the system and allow access.
Visibility to secrets is granted based on configured “paths” and user-based policies. So, for example, an application called “UserAccounts” can be attached to a policy that allows the service to only read i.e vault.company.io/secrets/UserAccounts from its own path. This actively applies compartmentalization between the different services, so that even if one of them is compromised, it has no access to others’ keys. Such a process comes hand-in-hand with a general rule to create dedicated and slim roles and policies to each service, providing it with only the minimum permissions required to complete its job.
sys/auth
Vault allows different authentication methods like user/pass, tokens, LDAP, AWS, GitHub tokens and more. This way developers can easily interact with the system, reading and writing secrets, and so can applications interacting with Vault natively using wrapper-libraries.
* Security TIP: If you use GitHub as an organization, allowing GitHub tokens as backend authentication is very comfortable for developers to interact with Vault directly. But allow it only after enforcing 2FA on the GitHub organization.
GDPR Compliancy
As the world heads toward the GDPR due date, companies and organizations plan their information compliancy according to the new rules. Vault can help with strengthening personal data using roles and access policies. Learn more about it in a recorded webinar, from Feb/2018 — Preparing for GDPR Compliance with HashiCorp Vault.
A much simpler integration with application code
In order to maintain strict security, Vault requires each application to send an authentication request and only then it allows API interaction.
Since developers wouldn’t normally jump on the boat when it comes to adding code which doesn’t contribute to an applicative feature, we have to ease the on-boarding process and make it as seamless as possible; this can be simplified using Vault-Get.
Vault-get is a free and open source CLI tool which we developed in-house to interact with Vault in a single request that 1. Authenticates 2. Pulls secrets, and 3. Injects them to the local environment. This way, when the instance fires up, it pulls the secrets from vault and then launches the application with secrets injected into its environment.
Vault-get is as easy as:
eval "$(vault-get —vault_host [https://vault.example.com](https://vault.example.com) —vault_token mytoken —vault_path secret/my-secret-path)"
This line is the command that is fired before running the application. For more information visit github.com/devops-israel/vault-get. That’s all it takes, secrets have been returned and set as environment variables, ready to be used.
Lessons learned
Vault is just an example, it has alternatives and competitors, some may be better or more fitting for each specific use case. It’s not the tool that matters, but the change in methods of work, mindset and understanding. It’s important to understand that organizational secrets and keys have to be concealed and protected, and that a software code base should be consistently monitored and scanned for leaks. Investing in education is far more effective than just announcing new strict rules; let your developers and infrastructure employees learn about the environment they work in, and the risks and threats that surround them, in order to motivate them to improve security, and apply what they’ve just been taught.
So, as I like to say, don’t wait for the st to hit the fan, because the st is there and the fan is on:
Be proactive; be a professional; protect yourself.
Some additional information to complete the security process of concealing your secrets
Disaster Recovery
Even with the best security measures, accidents, and exploits still happen. Make sure you are ready for any scenario, even the one where your entire resource stack is being terminated or deleted in a non-retrievable way. In order to recover, you want to make sure your infrastructure is templated and backed-up in git. Recommended read: Infrastructure as code
Concealing visible sensitive data at all times
As presented earlier, Vault has many ways of authenticating, some include roles, some require token as a string delivered. Try working with authentication methods that don’t require any additional steps that may expose data. A very good example is using the already assumed IAM role that the instance or service use. This way, you’re not using keys or generating any unnecessary points of failure in your security flow. I’ll use a quote by Ryan Kroonenberg from acloud.guru:
“If you don’t have keys, you can’t lose them.”
It’s funny, cause it’s true 😉. Keeping this quote in mind is a very good way to keep your data out of unwanted hands.
Using IAM
Follow AWS’s / GCP’s best IAM practices, in order to learn about slim policies, roles creation and assuming by resources. Knowing in-depth IAM internals may very well be your saviour in terms of access management.
Let me know your thoughts in the comments below, or connect with me directly on Twitter @0merxx. Clap if you liked it, it helps me focus my future writings!
My name is Omer, and I am a consultant at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the Devops culture.
