Mutual TLS with Istio

Service mesh is a decentralized application-networking infrastructure that allows applications to be secure, resilient, observable and controllable. The service mesh exists to make your distributed applications behave reliably in any environment e.g. production. There are multiple open-source products available like linkerd, istio, Conduit etc.

Istio is an open-source implementation of a service mesh that provides several capabilities for traffic monitoring, access control, discovery, security, resiliency, and other useful things to a bundle of services. Istio does all that at a platform level so it requires minimal, if any, code changes to the applications.

Istio deploys envoy proxy as a sidecar alongside each application instance. They are responsible for handling all incoming and outgoing traffic from the application. This approach makes it possible to have consistent network policies throughout the service mesh. These sidecars can take the responsibility of encrypting the traffic without the application having to do anything or even know.

Transport Layer Security(TLS) ensures that communication between services is encrypted. In mTLS the client and server both verify each other’s certificates and use them to encrypt traffic using TLS. Istio takes care of certificate generation and maintenance using Citadel and propagates these certificates to all the istio envoy proxies.

Istio requires us to use a Policy object to instruct a service, namespace, or mesh to receive mTLS traffic. We then use DestinationRule to define a policy that ensures that all traffic intended for the service(s) uses mTLS i.e. client services make mTLS connection with target services using appropriate certificates.

mTLS types:

1. Permissive: Allow both http and mutual TLS traffic

2. Strict: Allow only mutual TLS traffic i.e. it won’t accept plain text data

3. Disabled: Plain text data will be sent in and out of the mesh

To explore how Stakater can assist you with Kubernetes and Istio implementation, consider our Kubernetes Consultancy and Kubernetes Platform Assessment services.

To enable mTLS first create mTLS authentication policy:

1. Mesh-wide:

apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
  name: default
spec:
  peers:
  - mtls: {}

2. Namespace-wide:

apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
  name: default
spec:
  peers:
  - mtls: {}

3. For a specific service:

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: default
  namespace: namespace-name
spec:
  targets:
  - name: service-name
  peers:
  - mtls: {}

Create destination rule:

The “Policy” informs the running services to expect any incoming traffic to use mTLS. But for “Mutual” TLS, we need to inform the clients to use mTLS when they call a service. DestinationRule serves this purpose:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: default
  namespace: namespace-name
spec:
  host: "service-name.namespace-name.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Use*.namespace-name.svc.cluster.local for forcing mTLS on all services in that particular namespace and *.local for mesh-wide mTLS.

Health checks:

Requests for health checks, against liveness and readiness probes, are sent from Kubelet which does not have Istio-issued certificates. So when mutual TLS is enabled, the health check requests will fail. An easy fix for this is to let istio handle it, istio can rewrite the application PodSpec readiness/liveness probe, such that the probe request will be sent to Pilot agent. Pilot agent then redirects the request to the application, and strips the response body only returning the response code.

To use this, use the following value with istio helm chart(or deployment): values.sidecarInjectorWebhook.rewriteAppHTTPProbe=true

Or, annotate the pod with sidecar.istio.io/rewriteAppHTTPProbers: "true"

Demo:

Estimated time: 5 mins

Now let’s go through a short practical demonstration for mTLS. We’ll use bookinfo which is an application based on micro-services architecture.

1. Deploy BookInfo

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.4/samples/bookinfo/platform/kube/bookinfo.yaml

2. Verify that the application is accepting plain-text requests:

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://reviews:9080/health -o /dev/null -s -w '%{http_code}\n'

Output: 200

3. Apply policy and destination rule(namespace-wide for demo):

cat <<EOF | kubectl apply -f -
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: default
  namespace: default
spec:
  peers:
  - mtls: {}
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: default
  namespace: default
spec:
  host: "*.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

4. Plain-text requests will fail now with:

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://reviews:9080/health -o /dev/null -s -w '%{http_code}\n'

Output: 000 command terminated with exit code 56

5. Verify that the application is accepting TLS requests:

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://reviews:9080/health -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k

Output: 200

Relevant links:

https://github.com/stakater/til/tree/master/istio

https://istio.io/docs/tasks/security/authentication/mutual-tls/

https://istio.io/docs/concepts/

https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.3.x/8mTLS.html