Whether you are a GitOps practitioner trying to find the best practices for setting up your GitOps repository or if you are looking to adopt GitOps and looking for ideas for the GitOps repository structure, we present Stakater’s opinion on organizing the files in a repository for practicing GitOps. We follow the repository structure described in this article for most of our customers, and the simplicity of the structure has rewarded us with many benefits, some of which we share below. The current structure is a result of more than two years of learning and using the structure in production on multiple clusters and customers.
For a detailed comparison between DevOps and GitOps, you can check out our blog on DevOps vs. GitOps.
Cluster and application configuration
Before we look into the structure of the repository, it is essential to understand our cluster and application that is referenced throughout the article.
Some terminology that is used in the article:
Cluster - A cluster can be either a Kubernetes, OpenShift, Tanzu, or Rancher cluster where the application is deployed through GitOps. To explore our Kubernetes solutions, check out our Kubernetes Consultancy and Openshift-as-a-Service.
Tenant - A tenant refers to a team or product that shares one or more clusters. Each tenant is isolated from the others in a multi-tenant cluster.
Application (app) - An application or app refers to a service/deployment that constitutes the workload that is deployed on the cluster.
Environment (env) - Environment or env refers to the deployment environment such as dev, staging, and production.
DTE (dynamic test environment) - Dynamic Test Environments are temporary environments(namespaces) that are created for each Pull Request for testing the changes of the Pull Request.
To summarize, a cluster can hold multiple tenants; each tenant can hold multiple applications, and each application can be deployed in multiple environments.
Cluster
Our approach for the GitOps repository structure can be applied to any Kubernetes cluster. The cluster-scoped infrastructural and administrative configurations are deployed through a separate infra repository. The infrastructure repository contains the ArgoCD configuration for deploying cluster-wide configuration of various operators. The infrastructure repository contains a directory at the root level for each cluster, allowing the use of the same git repository for multiple clusters.
Application
Applications are packaged as Helm charts, which are stored externally and not in the GitOps repository. Refer to Using an external chart in a Helm chart repository for the usage of external Helm charts in a GitOps repository. The generic Helm chart for our applications, which is stored externally, is also open-sourced at Stakater Charts.
Structure of the GitOps repository
This repository structure is a result of several iterations of the structure. Stakater’s opinion on the GitOps repository structure is as follows:
Two repositories: Use one repository for application and the other for admin/cluster-wide across multiple clusters, applications, and environments.
Single branch: All manifests are in a single git branch, i.e., main and then separate directories per environment
ArgoCD for application deployment: ArgoCD is our choice for implementing GitOps
Helm for templatization: Use Helm for substituting values during deployment
This GitOps structure supports:
Multiple clusters
Multiple tenants/teams/products
Multiple apps
Multiple environments (both static and dynamic(DTE))
Multiple clusters
Each cluster has a separate directory for storing the cluster’s ArgoCD Application definitions inside the 00-argocd-apps directory, which is at the repository’s root. Each cluster’s directory has separate directories for the environments in that cluster and each environment directory contains separate YAML files for each tenant describing the ArgoCD Application. These Applications are defined using the App Of Apps pattern of ArgoCD. One of the advantages of organizing applications in such a way is that we can create one root application per cluster that points to the clusters specific directory and all the cluster applications will be applied.
App of apps
App of Apps is an ArgoCD pattern that allows the definition of a parent ArgoCD Application and these Applications point to child Applications. The Application definitions in the 00-argocd-apps directory at the repository's root have the path to the actual ArgoCD Application definition for the respective tenant and environment. Consider these as the primary or root Application that is defined for ArgoCD or the index to the other Application definitions defined in the respective tenant directories.
Multiple tenants
Each tenant contains a directory at the root of the repository and is prefixed with a number starting with 01-, which increases for each tenant. Each tenant directory contains the 00-argocd-apps directory, which contains the ArgoCD Application definitions. Each of these 00-argocd-apps directories contains a separate directory for the environments in the tenant. The environment directory contains the ArgoCD Application definition for each application in that environment.
Multiple apps
Each tenant directory that is present in the root of the repository contains individual directories for the applications of the tenant. These application directories are prefixed with a number starting with 01-, which increases for each application.
Multiple environments
Each application directory contains directories for each environment of the tenant. Environment names are prefixed by a number that determines the order in which the application should be promoted between the different environments; e.g. DTE -> dev -> stage -> prod
Workflow of CICD
Apart from understanding the repository structure described above, it is essential to understand how CICD is performed with such a repository.
Stakater uses Tekton for performing Continuous Integration, although any CI tool can be used. The Tekton pipelines are defined as Kubernetes manifests and are deployed through GitOps using the same repository. A dedicated namespace called build is used for a tenant which holds Tekton pipelines. All Tekton pipelines for all of the applications of a given tenant run in this particular namespace. For more information on choosing the best CI/CD tool, you can read our comparison of GitHub Actions, Bitbucket Pipelines, GitLab CI, and Tekton.
The CI pipeline performs the application's build along with other usual CI activities. The pipeline updates the values.yaml of the Helm chart for the lower environment in the repository with the new image version. The lower environment can be any environment where the newer changes are tested before pushing to higher environments. Higher environments are generally the production to pre-production environments and the lower environments are the development and testing environments. Once the changes are tested and validated in the lower environment, the values.yaml of the Helm chart for the higher environments are changed manually in the repository through PRs from developers.
Managing secrets effectively is crucial in GitOps practices. Ensuring that sensitive information such as API keys, passwords, and tokens are securely stored and accessed is a key part of maintaining a secure infrastructure. For best practices on handling secrets within a GitOps workflow, refer to our blog on Secrets Management with GitOps and Kubernetes.
Stakater is building a CLI tool to manage and support this structure. The CLI tool helps with the promotion of the application from a lower environment to a higher environment, reducing the manual effort required. The tool will be open source and open to contribution once it is made public.
The above image summarizes the CICD workflow for three environments - dev, stage, and prod, where the dev environment is the lower environment.
Using an external chart in a Helm chart repository
Helm charts in GitOps repositories can be defined in two ways:
Option # 1: Locally, with a templates folder that includes all the Kubernetes manifests
Option # 2: Using a Helm dependency, and this dependency points to an external Helm Chart Repository where the chart is present.
We prefer the latter approach (option # 2) with this setup. The only configuration present in the GitOps repository is the environment-specific configuration and dependencies, based on versions. This solution scales better and has better versioning capabilities, but requires some more Helm expertise from developers.
The resources are structured as follows:
A Chart.yaml is defined for each environment with an external dependency to a Chart Version. The Helm chart is present in a remote Helm Chart repository
The values.yaml contains all the environment-specific configurations for the particular environment along with an image tag that points to a tag in the Container registry.
Conclusion
We present Stakater’s opinion of structuring a git repository for GitOps, and it works well for multiple clusters/tenants/applications/environments. Adopting GitOps has its set of challenges but brings in a lot of advantages. We hope the presented structure can simplify your approach to GitOps and help follow the best practices. For a comprehensive assessment of your Kubernetes platform, consider our Kubernetes Platform Assessment.
We hope you found this useful. Contact us if you'd like to get a free demo of this setup or to have your current approach assessed by one of our experts for free.
nice blog, is work for me
Nice blog, could you share me the diagram tool name you are using?😀
i like the article , how do you guys intend to handle feature requests/bugs/hot fixes with git branches ?