7 Tips to Simplify Kubernetes Secrets Management

You're an exception to the rule if you have a mature tech stack and don’t use Kubernetes (k8s). Kubernetes is the undisputed leader in orchestrating, scaling, and deploying containerized applications. 

61% of the cloud-native community used Kubernetes in 2022, and a further 31% were considering adopting it. Its Infrastructure-as-code approach, unparalleled ecosystem, and first-class Docker integration make it difficult to justify any rival. However, security considerations are essential to prevent downtime and malicious activity with any powerful and complex service. 

Maintaining a secure deployment can be a difficult task, especially when it comes to keeping your secrets safe. So, in this article, we give you seven essential tips on managing and protecting your secrets in Kubernetes. 

What are Kubernetes' secrets?

Kubernetes Secrets are objects containing private information, whether internal (business data) or external (customer data). Often, these are keys to other services like passwords, API keys, and OAuth tokens. These secrets enable containers to function correctly and communicate with other technologies. For instance, a container might need an API key to access real-time data over the internet or connect with a third-party product.



The Importance of Secrets Management

Since secrets are often passwords to access other products and services, you must keep them hidden and secure. If someone can access your secrets, they can run up huge bills on your account, steal intellectual property, or change your password and hold your infrastructure to ransom.

Managing secrets in K8s can also be a pain. If you are orchestrating hundreds of containers, it can be a huge undertaking to keep track of where all your sensitive information is stored. Changing the secrets when necessary is even more complex, especially when multiple containers need access to the same secret. Therefore, consider how you design Kubernetes’ secret management to keep your data secure, reduce ongoing maintenance, and mitigate the chances of error.

7 Tips to Simplify Kubernetes Secrets Management

1. Encrypt all data in transit or at rest

If you store your passwords in plain text, anyone accessing your account can easily read the data. This is the K8s equivalent of writing your bank details on a Post-it note on your fridge.

Instead of storing passwords as plain text, encrypt them to ensure they are safe, whether in transit or at rest. When it comes to deployment configurations like YAML, avoid embedding secrets directly. Instead, opt for the use of environment variables or external secret managers to add an extra layer of protection (more on this below).

Kubescape enables you to scan for unsecured secrets and understand which secrets need encryption. Plus, Kubernetes provides native encryption options for secrets at rest and in transit. You can enable encryption at rest using the Encryption Configuration API. Start by creating a YAML manifest as follows:

apiVersion: v1 kind: Secret metadata:   name: my-secret type: Opaque data:   username: <base64-encoded-username>   password: <base64-encoded-password>

In this example, <base64-encoded-username> and <base64-encoded-password> should be replaced with the Base64-encoded versions of your username and password. However, these credentials should not be hardcoded directly into the YAML file. Instead, they should be retrieved from a secure location, like an environment variable or a secret management tool, and only referred to from the file. Kubernetes will then automatically encrypt this secret data at rest when it is stored in etcd.

To protect secrets during communication between resources within Kubernetes (in transit), use TLS. For instance, the communication should be encrypted when a container needs to access an external API using an API key. First, ensure that the external service you are connecting to supports TLS. Then, use the appropriate URL scheme (e.g.,

2. Avoid secrets sprawl 

The more secrets you have, the more challenging they are to manage, and the more doors you open for hackers to enter your systems. Secrets sprawl means having an ever-growing number of secrets that become increasingly harder to track and secure. 

A centralized secrets management solution like Kubernetes’ own Kubernetes Secrets or an external tool like HashiCorp Vault can help you merge your secrets across clusters. To use HashiCorp Vault with Kubernetes, you'll need to install and configure the Vault server and integrate it with your Kubernetes cluster. Once set up, you can create and manage secrets in Vault and access them securely from your applications running in Kubernetes.

HashiCorp Vault provides a robust and flexible platform for secret storage, encryption, and dynamic secrets generation platform. With Vault, you can define fine-grained IAM policies, audit secret access, and manage secrets centrally, regardless of the number of clusters or namespaces.

3. Use anti-affinity rules

Co-locating your secrets in the same node is the same as putting all your eggs in one basket - the hacker gets a field day, but you risk having all your secrets compromised. To avoid these, use anti-affinity rules to isolate and split secrets between different nodes. Anti-affinity minimizes the consequences of a potential hack and helps you manage your secrets more efficiently. For instance, even if one node fails, secrets are available on other nodes. 

You can define Pod Affinity and Pod Anti-affinity rules in Kubernetes. There are two types of Pod Anti-Affinity: 

• "preferredDuringSchedulingIgnoredDuringExecution” - shows a preference for not locating secrets on the same node.
• "requiredDuringSchedulingIgnoredDuringExecution” - more than a preference, this is a hard rule for not co-locating secrets. 

Label the nodes you want to use to spread your secrets across nodes, then connect the Pod Affinity or Anti-Affinity rule to a specific label. Check out this example defining a Pod Anti-Affinity: 



4. Reduce the secrets’ scope 

To enhance the security of your Kubernetes secrets, it's crucial to apply the principle of least privilege. This principle advocates granting each container access to only the specific secrets it requires for its proper functioning and no more. 

Limiting the scope of secrets means that containers will only have access to the sensitive keys or information they genuinely need to operate rather than having unrestricted access to all secrets in the cluster. This reduces the risk of unauthorized access to critical data by malicious actors who might gain access to a compromised container.

Just like Role-Based Access Control (RBAC) allows you to control access to Kubernetes resources, secrets also support scope limitation through fine-tuning permissions. When creating a secret, you can configure it to be accessible only by specific pods or containers, ensuring that only those authorized entities can access the sensitive information.

5. Avoid hardcoding secrets in YAML files

As mentioned in our first tip, storing sensitive information, such as passwords and API keys, directly in YAML configuration files can make it vulnerable to accidental exposure or unauthorized access. Never hardcode your secrets directly into YAML or any other configuration files. Instead, refer to the secrets from a secure location or through reference mechanisms provided by the platform, such as Kubernetes secrets.

One practical approach is to use environment variables to obfuscate secrets from the code and configuration files. By utilizing environment variables, you can store sensitive data separately from your codebase, enhancing the overall security of your Kubernetes deployment.

To exemplify how to use environment variables, consider the following YAML configuration:

‍



In this example, the my-app-pod Pod contains a container named my-app-container, which fetches the username and password secrets from the my-app-secrets Kubernetes Secret and sets them as environment variables within the container.

6. Use external secret management solutions

Consider integrating an external secret management solution such as Vault by Hashicorp and Azure Key Vault. These services securely store and encrypt your secrets in a central location where it is easy to update, rotate, and expire them (more on that later).

You can use the External Secrets Operator with Kubernetes to integrate your secret management system into your cluster. It securely fetches and synchronizes your data from your external vault and allows you to keep all your secrets separate from your deployment.

To use HashiCorp Vault with Kubernetes, deploy the Vault server and integrate it with your Kubernetes cluster. Once configured, you can securely create and manage secrets within Vault.



With HashiCorp Vault integrated into your Kubernetes environment, you can securely manage and access secrets, ensuring a higher level of protection for your sensitive data.

7. Secret rotation and expiration

Rotation and expiration limit the lifetimes of your secrets. It regularly updates your keys to a new value and revokes permissions for your old ones so they can no longer be used. This practice puts a time limit on the utility of a key for a malicious actor. If they gain access to one, they only have until you rotate its value to inflict damage. Similarly, if an attacker acquires an expired secret, they can do nothing.

You can implement this practice by manually switching over keys at regular intervals, using an operator such as the Secrets Store CSI driver, or using a secret management tool such as Kubernetes CronJobs to automate the process and trigger the rotation of your secrets at specific times. 

Here's a simplified example of a Kubernetes CronJob that automates the rotation of a secret named my-database-secret every 30 days:



In this example, the CronJob runs every 30 days (schedule: "0 0 */30 * *"), triggering the secret rotation process in the secret-rotation-container.

How should I get started?

Secret management in Kubernetes is a complicated business that requires thoughtful design architecture and ongoing maintenance. You should work to hide and encrypt your secrets, limit access for users and keys, and regularly log and monitor your deployment for security flaws.

Consider integrating tools that automate secret exposure and security vulnerability across your entire CI/CD pipeline to make it easier. Solutions like Kubescape support static monitoring and runtime scans and reveal misconfigured policies, overprivileged containers, and other violations. 

Jit orchestrates open-source security tools such as Kubescape and GitLeaks to effortlessly facilitate product security across your entire stack, including Kubernetes. Plus, when using Kubescape and other OSS tools with Jit, you get much more optimized reporting, including insights from other security tools. If you want to learn more, you can start for free here.

‍