top of page

Stakater Blog

Follow our blog for the latest updates in the world of DevSecOps, Cloud and Kubernetes

Muneeb Aijaz
Aug 269 min read

MTO on AKS

In this blog, we’ll walk you through running the Multi-Tenant Operator (MTO) on an Azure Kubernetes Service (AKS) cluster provided by Microsoft Azure. We’ll also show how users can interact with the cluster.


Multi Tenant Operator

MTO is a product by Stakater that’s been delivering robust multi-tenancy solutions with integrations to ArgoCD and Vault on OpenShift for several years. Since March 2022, MTO has been a RedHat Certified Operator, and we’ve recently broadened our focus to include native Kubernetes and cloud-provided Kubernetes distributions.

Our MTO ensures effective tenancy through its RBAC model and targeted admission policies. This setup guarantees safe cluster usage, with segregated partitions within the cluster and across its integrations. For a detailed list of features, check out Simplify Multitenancy with Stakater's Multi Tenant Operator.


Walk-through

We can integrate AKS clusters with Microsoft Entra ID to manage user authentication within the clusters. In this example, we’ll set up a small test cluster to demonstrate how different Entra ID users, such as developers and Site Reliability Engineers (SREs), can access the AKS cluster. We’ll also explore how tenancy under the Multi-Tenant Operator affects their operational scope.


We’ll use this Microsoft guide on Azure role-based access control for Kubernetes Authorization as our reference.


Prerequisites

  • Azure CLI: Ensure you have Azure CLI version 2.0.61 or later installed and configured. If you need to install or upgrade, follow Install Azure CLI. To check your version, run:

az --version

  • Kubectl: You’ll also need kubectl, version 1.18.3 or later. For installation instructions, visit Install Kubectl.

  • Helm CLI: To install MTO, you’ll need Helm CLI. Find instructions at Installing Helm.

  • Azure Portal User: Have a user set up in Azure Portal with the necessary permissions to create clusters, users, and groups.


Create an Admin Group

Let’s start by creating an admin group, which we’ll use to manage our AKS cluster. Log in to Azure Portal from CLI:

$ az login

A browser window will open, directing us to the Azure Login page. After logging in, we’ll see our user details:

$ az ad signed-in-user show

{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users/$entity",
  "businessPhones": [],
  "displayName": "test-admin-user",
  "givenName": null,
  "id": "<user-id>",
  "jobTitle": null,
  "mail": null,
  "mobilePhone": null,
  "officeLocation": null,
  "preferredLanguage": null,
  "surname": null,
  "userPrincipalName": "<upn>"
}

The user-id will be used later to add our user in the admin group for MTO.


Create a group called mto-admins:

$ az ad group create --display-name mto-admins --mail-nickname mto-admins

Using the above user-id, link user to the newly created group:

$ az ad group member add --group mto-admins --member-id <user-id>

Use this command to get admin-group-id, this will later be used while provisioning the cluster:

$ az ad group show --group mto-admins

{
  **********
  "description": null,
  "displayName": "mto-admins",
  "expirationDateTime": null,
  "groupTypes": [],
  "id": "<admin-group-id>",
  "isAssignableToRole": null,
  **********
}

Create an AKS Cluster

Let’s create a Resource Group using the az group create command in our preferred Azure location:

$ az group create --name myResourceGroup --location westus2

Create a small cluster:

$ az aks create --resource-group myResourceGroup --name myAKSCluster --node-count 1 --vm-set-type VirtualMachineScaleSets --enable-cluster-autoscaler --min-count 1 --max-count 3 --enable-aad --aad-admin-group-object-ids <admin-group-id>

Create test groups in Entra ID

First, let’s store the ID of our AKS cluster in a variable named AKS_ID:

$ AKS_ID=$(az aks show --resource-group myResourceGroup --name myAKSCluster --query id -o tsv)

Let’s create our first test group named appdev using the group command and assign its ID to APPDEV_ID variable:

$ APPDEV_ID=$(az ad group create --display-name appdev --mail-nickname appdev --query id -o tsv)

Allow the appdev group to interact with the AKS cluster using kubectl by assigning them the Azure Kubernetes Service Cluster User Role:

$ az role assignment create --assignee $APPDEV_ID --role "Azure Kubernetes Service Cluster User Role" --scope $AKS_ID

Let’s create our second test group named opssre using the command and assign its ID to the OPSSRE_ID variable:

$ OPSSRE_ID=$(az ad group create --display-name opssre --mail-nickname opssre --query id -o tsv)

Allow the opssre group to interact with the AKS cluster using Kubectl by assigning them the Azure Kubernetes Service Cluster User Role:

$ az role assignment create --assignee $OPSSRE_ID --role "Azure Kubernetes Service Cluster User Role" --scope $AKS_ID

Create test users in Entra ID

Set the User Principal Name (UPN) and password for our users. The UPN should include the verified domain name of our tenant, for example user@company.com.


Following command reads the UPN for the appdev group and stores it in the AAD_DEV_UPN variable:

$ echo "Please enter the UPN for application developers: " && read AAD_DEV_UPN

For this scope of this blog, we will assume that the entered UPN was aksdev@company.com.


Following command reads the password for our user and stores it in the AAD_DEV_PW variable:

$ echo "Please enter the secure password for application developers: " && read AAD_DEV_PW

Create the user AKS Dev using the previously created variables:

$ AKSDEV_ID=$(az ad user create --display-name "AKS Dev" --user-principal-name $AAD_DEV_UPN --password $AAD_DEV_PW --query id -o tsv)

Add this user to the appdev group that was previously created:

$ az ad group member add --group appdev --member-id $AKSDEV_ID

Repeat the steps for OPS SRE user.


The following command reads the UPN for our user and stores it in the AAD_SRE_UPN variable:

$ echo "Please enter the UPN for SREs: " && read AAD_SRE_UPN

For this scope of this blog, we will assume that the entered UPN was opssre@company.com.


The following command reads the password for our user and stores it in the AAD_SRE_PW variable:

$ echo "Please enter the secure password for SREs: " && read AAD_SRE_PW

Create the user AKS SRE using above variables

# Create a user for the SRE role
$ AKSSRE_ID=$(az ad user create --display-name "AKS SRE" --user-principal-name $AAD_SRE_UPN --password $AAD_SRE_PW --query id -o tsv)

Add this user to the opssre group that was previously created:

$ az ad group member add --group opssre --member-id $AKSSRE_ID

Installing Cert Manager and MTO

In this section, we’ll install the Multi-Tenant Operator (MTO) to manage tenancy between different users and groups. MTO requires several webhooks that need certificates. To handle these certificates automatically, we’ll first install Cert Manager as a prerequisite.


Start by logging in to Azure from CLI by running the following command:

$ kubectl get pods

Running the command will open a browser window where we can log in using our test-admin-user.


Running kubectl auth whoami will show us the user info:

kubectl auth whoami

ATTRIBUTE    VALUE
Username     test-admin-user
Groups       [<mto-admins-id> system:authenticated]
Extra: oid   [<oid>]

You will notice that the mto-admins group ID is attached with our test-admin-user user. This user will be used for all the cluster admin level operations.


Install Cert Manager

Install Cert Manager in the cluster for automated handling of operator webhook certs:

Let's wait for the pods to be up:

$ kubectl get pods -n cert-manager --watch

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7fb948f468-wgcbx              1/1     Running   0          7m18s
cert-manager-cainjector-75c5fc965c-wxtkp   1/1     Running   0          7m18s
cert-manager-webhook-757c9d4bb7-wd9g8      1/1     Running   0          7m18s

Install MTO using Helm

We’ll use Helm to install MTO since it’s the only available method for installing it on Kubernetes clusters.


Use helm install command to install MTO helm chart. Here, `bypassedGroups` has to be set as `system:masters` as it is used by masterclient of AKS and `<mto-admins-id>` as it is used by test-admin-user:

$ helm install tenant-operator oci://ghcr.io/stakater/public/charts/multi-tenant-operator --version 0.12.62 --namespace multi-tenant-operator --create-namespace --set bypassedGroups='system:masters\,<mto-admins-id>'

Wait for the pods to come to a running state:

$ kubectl get pods -n multi-tenant-operator --watch

NAME                                                              READY   STATUS    RESTARTS   AGE
tenant-operator-namespace-controller-768f9459c4-758kb             2/2     Running   0          2m
tenant-operator-pilot-controller-7c96f6589c-d979f                 2/2     Running   0          2m
tenant-operator-resourcesupervisor-controller-566f59d57b-xbkws    2/2     Running   0          2m
tenant-operator-template-quota-intconfig-controller-7fc99462dz6   2/2     Running   0          2m
tenant-operator-templategroupinstance-controller-75cf68c872pljv   2/2     Running   0          2m
tenant-operator-templateinstance-controller-d996b6fd-cx2dz        2/2     Running   0          2m
tenant-operator-tenant-controller-57fb885c84-7ps92                2/2     Running   0          2m
tenant-operator-webhook-5f8f675549-jv9n8                          2/2     Running   0          2m

Setting up Tenant for Users

Start by getting IDs for opssre and appdev groups by running az ad group show command:

$ az ad group show --group appdev

{
  ***********
  "displayName": "opssre",
  "expirationDateTime": null,
  "groupTypes": [],
  "id": "<opssre-group-id>",
  "isAssignableToRole": null,
  ************
}
$ az ad group show --group appdev

{
  ***********
  "displayName": "appdev",
  "expirationDateTime": null,
  "groupTypes": [],
  "id": "<appdev-group-id>",
  "isAssignableToRole": null,
  ************
}

Create a `Quota CR` with some resource limits:

$ kubectl apply -f - <<EOF
apiVersion: tenantoperator.stakater.com/v1beta1
kind: Quota
metadata:
  name: small
spec:
  limitrange:
    limits:
    - max:
        cpu: 800m
      min:
        cpu: 200m
      type: Container
  resourcequota:
    hard:
      configmaps: "10"
EOF

Now, mention this `Quota` in two `Tenant` CRs, with opssre and appdev group IDs in the groups section of spec:

$ kubectl apply -f - <<EOF
apiVersion: tenantoperator.stakater.com/v1beta3
kind: Tenant
metadata:
  name: tenant-a
spec:
  namespaces:
    withTenantPrefix:
    - dev
    - build
  accessControl:
    owners:
      groups:
    - <opssre-group-id>
  quota: small
EOF
$ kubectl apply -f - <<EOF
apiVersion: tenantoperator.stakater.com/v1beta3
kind: Tenant
metadata:
  name: tenant-b
spec:
  namespaces:
    withTenantPrefix:
    - dev
    - build
  accessControl:
    owners:
      groups:
    - <appdev-group-id>
  quota: small
EOF

Notice that the only difference in both tenant specs are the groups.


Check if the tenant namespaces have been created:

$ kubectl get namespaces

NAME                    STATUS   AGE
cert-manager            Active   4h26m
default                 Active   5h25m
kube-node-lease         Active   5h25m
kube-public             Active   5h25m
kube-system             Active   5h25m
multi-tenant-operator   Active   3h55m
tenant-a-build          Active   10m
tenant-a-dev            Active   10m
tenant-b-build          Active   10m
tenant-b-dev            Active   10m

Notice that MTO has created two namespaces under each tenant.


Users Interaction with the Cluster

AppDev group

AppDev is one of the groups we created earlier, and its scope is limited to Tenant A namespaces, as we mentioned its group ID in Tenant A. Let’s start by clearing the token of our test-admin-user:

$ kubelogin remove-tokens

Use the aksdev user from appdev group to login to the cluster:

$ kubectl get pods

This will take us to the devicelogin page. After entering the correct code, we’ll be redirected to the Microsoft Login page, where we’ll enter the email and password for the aksdev user created at the start of the article.


After a successful login, we’ll see the output of our kubectl command:

Error from server (Forbidden): pods is forbidden: User "aksdev@company.com" cannot list resource "pods" in API group "" in the namespace "default"

This user does not have access to default namespace.


Now. let’s try accessing the resources in the tenant namespaces under Tenant A:

$ kubectl get pods -n tenant-a-dev

No resources found in tenant-a-dev namespace.

Create an nginx pod in the same namespace

$ kubectl run nginx --image=nginx -n tenant-a-dev

pod/nginx created

Now, let’s try the same operation in another namespace of Tenant B:

$ kubectl run nginx --image=nginx -n tenant-b-dev

Error from server (Forbidden): pods is forbidden: User "aksdev@company.com" cannot create resource "pods" in API group "" in the namespace "tenant-b-dev"

This operation fails with an error showing strict controls in their Tenants.


OpsSre group

OpsSre is the second group we created at the start of this article, and its scope is limited to Tenant B namespaces, as we mentioned its group ID in Tenant B.


Let’s start by clearing the token of the appdev user.

$ kubelogin remove-tokens

Then, use the opssre user from the Opssre group to log in to the cluster:

$ kubectl get pods

This will take us to the devicelogin page. After entering the correct code, we’ll be redirected to the Microsoft Login page, where we’ll enter the email and password for the opssre user created at the start of the article.


After a successful login, we’ll see the output of our kubectl command:

Error from server (Forbidden): pods is forbidden: User "opssre@company.com" cannot list resource "pods" in API group "" in the namespace "default"

This user doesn’t have access to the default namespace.


Now, let’s try accessing the resources in the tenant namespaces under Tenant B:

$ kubectl get pods -n tenant-b-dev

No resources found in tenant-b-dev namespace.

Create an nginx pod in the same namespace:

$ kubectl run nginx --image=nginx -n tenant-b-dev

pod/nginx created

Now, let’s try the same operation in another namespace of Tenant A:

$ kubectl run nginx --image=nginx -n tenant-a-dev

Error from server (Forbidden): pods is forbidden: User "opssre@company.com" cannot create resource "pods" in API group "" in the namespace "tenant-a-dev"

This operation fails with an error indicating strict controls in their tenants.


Cleanup Resources

We need to clean up the users, groups, AKS Cluster, and Resource Group created for this blog.


Run the following set of commands to remove the resources created in the sections above:

# Delete the Azure AD user accounts for aksdev and akssre.

$ az ad user delete --id $AKSDEV_ID
$ az ad user delete --id $AKSSRE_ID

# Delete the Azure AD groups for appdev,opssre and mto-admins. This also deletes the Azure role assignments.

$ az ad group delete --group appdev
$ az ad group delete --group opssre
$ az ad group delete --group mto-admins

# Delete the Resource Group which will also delete the underlying AKS test cluster and related resources

$ az group delete --name myResourceGroup

Conclusion

In this blog, we explored how to achieve multi-tenancy in AKS clusters using the Multi-Tenant Operator by adding different Microsoft Entra ID groups to tenants. Managing large clusters with multiple teams becomes crucial to avoid resource overlap and prevent issues caused by shared access. The Multi-Tenant Operator is ideal for these scenarios, as it restricts users to their specific tenant namespaces, providing fine-grained control over the cluster. This setup is easily accomplished by integrating the Multi-Tenant Operator with AKS and Microsoft Entra ID.

32 views0 comments

Recent Posts

See All

MTO on EKS

0 comments

Comments

bottom of page