Most advances in DevOps today are being leveraged for speed. Continuous integration and continuous delivery (CI/CD) is driven by the need to meet internal demands, meet customer demands, and adapt to the competition. However, as the headlines constantly remind us, security can’t be overlooked in our rush to deploy.
No matter where your organization is on the spectrum from monolithic code to microservices, or annual to continuous releases, security is often considered an impediment to progress. Of course, the ideal situation is for tools to deliver speed and security.
Why native security is not sufficient
Starting with applications, there’s often a need for one application to access another application or database either within or outside the enterprise. This is how secrets (passwords, keys) end up hard-coded, or barely better, in configuration files, into the applications themselves. We’ve seen many instances of hard-coded credentials posted publicly in Github, and we’re sure this happens in internal repositories as well. DevOps tools often provide their own method of secrets management. Kubernetes provides a method to store secrets, which I’ll discuss later. Jenkins, Chef, and Puppet all have some type of vault. So why not use them?
Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools
A single centralized vault provides efficiency and security. Many of your applications and DevOps tools need access to the same assets, so having one vault enables the creation of a unified policy for access to those assets. Secrets rotation is far easier from one vault location as opposed to coordinating several locations in a variety of different tools. A centralized vault also enables simplified auditing and alerting from that single source rather than having to pull logs from various tools.
It’s important to note that none of this is very easy, if possible at all, with hard-coded credentials in applications or secrets spread over several DevOps tools. For both applications and DevOps tools, when secrets are needed, an API call to the centralized vault is ideal.
Kubernetes security with DevOps Secrets Vault
As mentioned, Kubernetes provides a mechanism for the applications in pods to access secrets. They are managed in the etcd (pronounced et-cee-dee) distributed database along with all the other cluster configuration information.
Beyond the non-centralized vault issues mentioned above, there are two known issues with this method. The first is that any secrets stored in etcd are available to any node in the cluster. So an intrusion into any node in the cluster allows access to all the other nodes via lateral movement. The other issue is the secrets are stored in plain text (base64 encoded) in the etcd database.
The DevOps Secrets Vault Kubernetes Plug-in uses a different approach. It consists of a Broker in its own pod and a Client that acts as a sidecar within the pod of any application that needs a Secret. Let’s look at how this technique works. (See diagram)
The Secret volume is only available inside the Pod and is already built into Kubernetes. It’s used by the DevOps Secrets Vault plugin and secrets are updated in the Client rather than the etcd database. To enable the automated connection to DevOps Secrets Vault, the client build file sets an annotation to “DSV.” The Broker then watches for any Pod that has the client with the “DSV” annotation and registers it.
The Broker and Client then work together to keep any Secret(s) updated in the secrets volume by connecting to the DevOps Secrets Vault API. The secrets are cached in the Broker for as long as you set them to allow for secrets rotation. If for security reasons you didn’t want to cache secrets, you could set the policy to 0 seconds so it must connect to the API every time the Client requests the Secret.
The Broker will keep the secrets updated as long as the Pod remains active. If the Pod’s not active, then the Broker will delete the secrets. The Broker will also constantly watch for new pods and automatically connect them to DevOps Secrets Vault, which is especially helpful when you have autoscaling in your Kubernetes environment.
The plugin also allows for the communication between the Broker and Client and the Broker and DevOps Secrets Vault to be TLS encrypted. In this case, it makes sense for the TLS keys to be managed by etcd because this is a Cluster-level function. Alternatively, a service mesh like Linkerd or Istio could be used to set up and manage the TLS communications between Pods.
Streamlining secrets management for DevOps
DevOps Secrets Vault is an API-as-a-Service, which makes getting up and running easy. No installation of the vault or database is required and Delinea even handles all the updates. A command-line interface (CLI) is provided for Windows, Mac, and Linux. In addition to the Kubernetes plugin, we have a Jenkins plugin now, will release Terraform, Ansible, Chef, and Puppet plug-ins in January 2020, and will be adding more DevOps tool plug-ins over time.
To help applications go to DevOps Secrets Vault directly, we have a Java SDK now and will release Go, Python, and Ruby in January 2020. The CLI, plug-ins, and SDKs are open source, so you have the freedom to add your own customizations. Try DevOps Secrets Vault for free to get full capabilities to automatically create, archive, and retrieve up to 250 passwords and other secrets.
Manage DevOps secrets safely
