Introduction
According to the statistics, credentials breach was the most common type of data breach in 2022, accounting for 61% of all breaches. Therefore, managing credentials also called referred to as secrets, has become a highly crucial aspect of running applications and services.
However, managing secrets can be challenging, especially in DevOps environments. To properly do this, a good understanding of secret management and integrating it into your workflow is important.
In this article, we will discuss in detail what secrets management is and why it is important, what GitOps is, how it works with Kubernetes, and the benefits of using GitOps for secret management. We will also provide guidance on implementing GitOps with Kubernetes and best practices for using GitOps in secret management.
What is Secrets Management?
Secrets management is the process of storing, protecting, and managing sensitive information, such as passwords, API keys, and certificates in the highest-risk layers of a modern organization’s tech infrastructure: clouds, code, data, and devices.
It is an important aspect of information security because it helps prevent unauthorized access to sensitive data and ensures that only authorized personnel have access to it. There are many different types of secrets that may need to be managed, including:
Passwords: Passwords are used to protect access to accounts and resources. It is important to store passwords securely and to use strong, unique passwords to prevent unauthorized access
API keys: API keys are used to authenticate access to APIs and other resources. It is crucial to store API keys securely and to rotate them regularly to prevent unauthorized access
Certificates: Certificates are used to establish trust between systems and secure communication. It is important to store certificates securely and to manage their expiration dates to ensure that communication remains secure
User data: Sensitive user information such as ATM pins, user profiles, etc can also be considered secrets.
Why is Secrets Management important?
Secrets management is important because it helps to protect sensitive information, such as passwords, API keys, and security certificates, from being accessed by unauthorized individuals or systems.
Secrets management enables you to store, transmit, and audit secrets securely. It minimizes the involvement of humans in the management of secrets to reduce potential points of failure.
If sensitive information is not properly protected, it can be accessed by malicious actors who may use it to compromise the security of a system or steal sensitive data.
Implementing effective secrets management in modern DevOps environments is important for several reasons:
Protecting sensitive data: In modern DevOps environments, sensitive data such as passwords, API keys, and certificates are often stored and used in multiple locations, including version control systems, build servers, and deployment environments. It is important to implement secrets management to ensure that this sensitive data is protected and that only authorized personnel has access to it
Ensuring compliance: Many organizations are required to comply with various regulations and standards that mandate the secure storage and management of sensitive data. Implementing secrets management helps organizations meet compliance requirements and avoid fines and other penalties
Reducing risk: By implementing secrets management, organizations can reduce the risk of data breaches and unauthorized access to sensitive data. This can help to protect the organization's reputation and assets
Secrets management best practices
To implement effective secrets management in modern DevOps environments, organizations can follow these best practices:
Use a dedicated secrets management solution: There are many dedicated secrets management solutions available that can help organizations securely store and manage sensitive data. These solutions often include features such as encryption, access controls, and auditing capabilities
Store secrets in a secure location: Secrets should be stored in a secure location that is not accessible to unauthorized personnel. This could be a separate server, a secure storage device, or a cloud-based storage solution
Use strong, unique passwords: Passwords should be strong and unique to reduce the risk of unauthorized access. Organizations can use password managers to generate and store strong, unique passwords
Update secrets regularly: Secrets should be rotated regularly to reduce the risk of unauthorized access. This could involve generating new API keys, updating passwords, or renewing certificates
Implement access control: Access to secrets should be strictly controlled and granted only to authorized personnel. This can be achieved through the use of access controls such as authentication and authorization policies
By following these best practices, organizations can implement effective secrets management in their DevOps environments and protect sensitive data, meet compliance requirements, and reduce the risk of data breaches and unauthorized access.
Now let's discuss GitOps and how to use it for secrets management in containerized applications.
What is GitOps?
GitOps is a set of practices that enables continuous delivery, integration, collaboration, version control, etc using Git as a single source of truth for declarative infrastructure and application code. This means that the entire application stack – including infrastructure, applications, and secrets – is managed in a Git repository.
In a GitOps workflow, developers commit their code changes to a Git repository via pull requests (PRs), which triggers a pipeline of automated processes that build, test, and deploy the changes. The pipeline is defined in code and stored in the Git repository alongside the application code, making it easy for developers to understand and modify the delivery process.
GitOps is often used in conjunction with infrastructure as code (IaC) tools and container orchestration tools, such as Kubernetes, to manage the deployment and scaling of cloud-native applications. It can also be used to manage the configuration and deployment of infrastructure resources, such as cloud resources and networking configurations.
To learn more about the comparison between DevOps and GitOps, check out Stakater’s blog on DevOps vs GitOps.
GitOps and Kubernetes for Secrets Management
There are two main methodologies for managing secrets in GitOps, they are:
Encrypted Secrets
Reference Store
Encrypted secrets with GitOps
Storing encrypted secrets can be done in Git repositories and using automation tools to decrypt and convert them into Kubernetes Secrets. Two methods of doing this are using Bitnami sealed secrets and Mozilla SOPs
Sealed Secrets: Sealed secrets offer the ability to easily encrypt and decrypt secrets without having to manually handle the encryption process. This makes it simple for developers to store and access sensitive information as needed.
Sealed Secrets has two parts: a Kubernetes controller that is aware of the private and public keys used for decrypting and encrypting encrypted secrets, and performs reconciliation tasks, and Kubseal, a command line interface that developers use to encrypt their secrets before adding them to a Git repository
Sealed secrets also include a number of security features to protect against unauthorized access. For example, the project uses a private key to encrypt secrets, which are only accessible to authorized users. This ensures that even if someone gains access to the repository, they will not be able to access the sensitive information.
Secrets OPerationS (SOPS): Mozilla SOPS (Secrets OPerationS) is an open-source tool designed to help secure and manage sensitive data within an organization. It uses encryption to protect secrets and allows for the creation of secure workflows for handling and sharing sensitive information. Mozilla SOPS uses encryption to protect secrets and allows for fine-grained access control through the use of GPG keys. When a secret needs to be accessed or updated, a pull request is created in the Git repository, allowing multiple team members to review and approve the change.
This helps organizations ensure that their sensitive data is kept safe and secure, while also providing the flexibility and convenience needed to effectively manage it.
Reference store in GitOps
References to secrets can be stored in Git repositories, and automation tools can be used to retrieve the actual secrets based on these references.
In a GitOps workflow with Kubernetes, the Kubernetes manifests – including the definitions of secrets – are stored in the Git repository. When a change is made to the manifests, a PR is opened and reviewed by the team. Once the PR is approved and merged, the continuous delivery tool automatically applies the changes to the Kubernetes cluster.
For example, if a new secret is added to the Git repository, the continuous delivery tool will automatically create the secret in the Kubernetes cluster. If an existing secret is updated, the tool will automatically update the secret in the cluster. This ensures that the actual state of the secrets in the Kubernetes cluster always matches the desired state defined in the Git repository.
Two projects that utilize this approach are External Secrets and the Kubernetes Secret Store CSI Driver. Both of these approaches depend on the presence of a key management system, which is typically a comprehensive enterprise system used to manage secrets for the entire company and provide additional capabilities beyond simply distributing encryption keys.
Another reference store tool to manage secrets in a GitOps workflow is Hashicorp's Vault. This tool allows us to store sensitive information, such as encryption keys, tokens, passwords, and API keys, in a secure and centralized location.
In a GitOps workflow, Hashicorp Vault is used as the centralized store for all secrets. These secrets are encrypted and protected by access controls and policies. A Git repository is then used to store manifests that reference the secrets stored in Vault. Once the manifests are deployed, an operator retrieves the secrets from Vault and applies them as Kubernetes Secrets within the cluster.
Hashicorp Vault integrates with most applications providing static and dynamic secrets management, Authentication, and Token renewal. It also supports fine-grained access control, allowing administrators to control who has access to specific secrets and what actions they can perform on them.
For a detailed discussion on structuring your GitOps repositories, refer to our blog regarding GitOps Repository Structure.
Benefits of Using GitOps with Kubernetes
There are several benefits to using GitOps for secrets management in Kubernetes. These include:
Version control: With GitOps, all changes to secrets – including additions, updates, and deletions – are tracked and versioned in the Git repository. This enables rollbacks and auditing of secrets management
Collaboration: GitOps enables teams to collaborate on secrets management through the use of PRs. This allows multiple team members to review and approve changes before they are applied to the Kubernetes cluster
Auditability: Because all changes to secrets are tracked and versioned in the Git repository, it is easy to audit the history of secrets management. This is useful for compliance.
Best practices for Secret Management with GitOps and Kubernetes
When it comes to secret management in a GitOps workflow, there are a few best practices you should follow:
Use a secret management tool: There are a variety of tools available for storing and managing secrets, such as Hashicorp Vault, AWS Secrets Manager, or Google Cloud KMS. These tools provide secure storage for your secrets and can be integrated with your GitOps workflow
Store secrets in a separate repository: It is generally not a good idea to store secrets in the same repository as your application or infrastructure code. Instead, you should store secrets in a separate repository that is only accessible to the necessary team members
Use encrypted secrets: You should always encrypt your secrets before storing them in a repository. This can be done using a tool like GPG.
Use environment variables: In Kubernetes, it is generally a good idea to use environment variables to pass secrets to your application. This allows you to avoid storing secrets in plaintext in your configuration files and makes it easier to rotate secrets
Use RBAC to control access to secrets: If you are using Kubernetes, you should use Role-Based Access Control (RBAC) to control which team members have access to which secrets. This helps to ensure that secrets are only accessed by the people who need them
By following these best practices, you can effectively manage secrets in a GitOps workflow and ensure the security and integrity of your infrastructure and applications.
For more on setting up Kubernetes test environments, visit Stakater’s blog on why you need a Kubernetes test environment and how to set it up.
Conclusion
By integrating secret management into the GitOps workflow, organizations can ensure that their secrets are handled in a secure and compliant manner. Overall, GitOps and secret management can help organizations improve their security posture, reduce the risk of breaches, and maintain the integrity of their infrastructure and applications.
If you're looking for comprehensive solutions on Kubernetes consultancy, you can explore Stakater’s Kubernetes Consultancy services. You can also check out our Kubernetes Platform Assessment for a detailed evaluation of your Kubernetes setup.