How does Istio service mesh deal with security?

In this blog, I will explain how Istio can help to solve issues such as encrypting traffic, provide flexible service access control, configure mutual TLS and fine-grained access policies and auditing.

Istio Security Architecture

The following Istio components are involved in providing security features in Istio:

• Certificate authority (CA) for managing keys and certificates
• Sidecar and perimeter proxies: implement secure communication between clients and servers (they work as Policy Enforcement Points (PEPs)
• Envoy proxy extensions: manage telemetry and auditing
• Configuration API server: distributes authentication, authorization policies and secure naming information

Policy Enforcement Point (PEP) is a component that serves as a gatekeeper to a resource.

Let’s look at the architecture diagram in the figure below for different components and their responsibilities.

Authentication

Based on the definition, authentication is a process or action of verifying the identity of a user or a process. This means Istio needs to extract credentials from requests and prove they are authentic. Envoy proxies in Istio are using a certificate for their credentials when communicating with each other. These certificates are tied to service accounts in Kubernetes.

When two services start communicating, they need to exchange the credentials with identity information to mutually authenticate themselves. The client checks the server’s identity against the secure naming information to see if it is an authorized runner of the service. On the server-side, the server determines what information the client can access based on the authorization policies. Additionally, the server can audit who accessed what at what time, and make decisions whether to approve or reject clients from making calls to the server. The secure naming information contains mappings from service identities to the service names. The server identities are encoded in certificates, and the service names are names used by the discovery service or DNS. A single mapping from an identity A to a service name B means that “A is allowed and authorized service B”. Secure naming information gets generated by the Pilot and then distributed to all sidecar Envoys.

Identity

For issuing identities, Istio uses Secure Production Identity Framework for Everyone, or SPIFFE (pronounced spiffy). SPIFFE is a specification for a framework that can bootstrap and issue identities. Citadel implements the SPIFFE spec; another implementation of SPIFFE is called SPIRE (SPIFFE Runtime Framework).

There are three concepts to the SPIFFE standard:

• SPIFFE ID: identity namespace that defines how service identify themselves
• SPIFFE Verifiable Identity Document (SVID): dictates how an issued identity is presented and verified. It encodes the SPIFFE ID.
• Workload API: specifies an API for a workload issuing and/or retrieving antoher workload’s SVID

In Kubernetes, service accounts are used for service identity. The URI that represents the SPIFFE ID is formatted like this: spiffe://cluster-name/ns/namespace/sa/service-account-name. By default, any pods that don’t set a service account explicitly will use the default service account that’s deployed in a namespace.

You can take look at the service account and the corresponding secret like this:

$ kubectl describe sa default
Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   default-token-pjqr9
Tokens:              default-token-pjqr9
Events:              <none>

The mountable secret/token name is the name of the secret in the same namespace that contains the certificate and token.

$ kubectl describe secret default-token-pjqr9

Name:         default-token-pjqr9
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: fe107ed9-8707-11e9-9803-025000000001

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  7 bytes
token:      ey....

The SPIFFE ID for the default service account would therefore be encoded like this: spiffe://cluster.local/ns/default/sa/default. The specification also describes how to encode this identity into a certificate that can be used to prove the identity. The SPIFFE says that the identity (the URI) needs to be encoded in the certificate’s subject alternative name (SAN).

Finally, the workload API for issuing and retrieving SVIDs in Istio is implemented using ACME (Automatic Certificate Management Environment) protocol.

The Citadel component automatically creates the certificate for existing and new service accounts, then stores them as Kubernetes secrets. If you create a deployment and look at the pod spec, you will notice something like this:

...
 volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-pjqr9
      readOnly: true
...

Using this snippet, Kubernetes mounts the certificate and other information from the service account to the pod. Because issued certificates are short-lived for security purposes (i.e. even if the attacker can get the SVID, they can only use it for a short time), Citadel ensures that certificates get rotated automatically.

Mutual TLS authentication

Transport authentication, also known as service-to-service authentication is one of the authentication types supported by Istio. Istio implements mutual TLS as a solution for transport authentication.

TLS stands for Transport Layer Security. TLS is used each time you try to access a secure endpoint. For example, visiting https://learnistio.com over HTTPS leverages TLS to secure the communication between the server where the website is running, and your browser. It doesn’t even matter if sensitive or private information is being transferred - the connection is secured regardless.

Using TLS requires a certificate authority (CA) to issue a digital certificate to the server, and this server then hands it over to the browser for validation with the CA.

mTLS takes the same idea but applies it to applications or services. This means that instead of the client only verifying the servers' certificate, the server also verifies the clients certificate.

An example of TLS would be crossing a border where you need to present your passport (a certificate) to the customs officer. Customs officer ensures your passport is valid, hasn’t expired, etc. In the mTLS case, you would also ask for a passport from the customs officer, and you would validate it.

Once both parties have validated the certificates with their respective CAs, the communication between parties can happen securely.

In the case of Istio, all communication between services goes through the Envoy proxies. Here are the steps that happen when the call gets made from service A to service B:

1. Traffic gets routed from service A to the Envoy proxy in the same pod
2. Service A proxy starts an mTLS handshake with the Service B proxy (secure naming check happens as well)
3. mTLS connection gets established
4. Traffic gets forwarded to the Service B proxy
5. Service B proxy forwards traffic to the service B in the same pod

Mutual TLS in Istio supports a permissive mode. This mode allows a service to accept both plain text traffic and mTLS traffic at the same time. This can help you gradually migrate your services to mTLS, without breaking existing plain text traffic. Once all services have the proxy, you can configure mTLS only mode instead.

To configure mTLS between services, the traffic policy field in the destination rule is used. For example, to require a client to use mTLS when communcating with the service-b, you’d use the ISTIO_MUTUAL mode:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: service-b-istio-mtls
spec:
  host: service-b.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

You could also provide your own certificates and set mode to MUTUAL like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: service-b-mtls
spec:
  host: service-b.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: MUTUAL
      clientCertificate: /etc/certs/cert.pem
      privateKey: /etc/certs/pkey.pem
      caCertificates: /etc/certs/cacerts.pem

Finally, you can set the mode field to SIMPLE to configure the client to use TLS:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: service-b-tls
spec:
  host: service-b.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: SIMPLE

Origin authentication

Origin authentication, known as end-user authentication, is used for verifying original clients requesting as an end-user or device. Istio enables original authentication with JSON Web Token (JWT) validation and open-source OpenID connect providers (e.g. Googe Auth, Auth0 or Firebase Auth).

In the case of origin authentication (JWT), the application itself is responsible for acquiring and attaching the JWT token to the request.

Authentication policies

Authentication policies are used to specify authentication requirements for services within the mesh. Similarly, as with traffic routing, Pilot watches for changes in the policy resources and then translates and pushes the configuration to the Envoy proxies.

These policies define what authentication methods can be accepted (i.e. requests being received). While for the outgoing requests, you would use the destination rule as explained earlier in this blog. The figure below illustrates this:

Authentication policies can be defined in two scopes that are explained next.

Namespace-scoped policy

The policies in the namespace scope can only affect services running in the same namespace. Additionally, you need to specify the namespace name, otherwise, the default namespace is used. Here’s an example of a namespace policy for the prod namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT

Mesh-scoped policy

Mesh-scoped policies can apply to all services in the mesh. You can only define one mesh-scope policy with the name default and an empty targets section. One difference from the namespace-scoped policy is the resource name. While namespace-scope policy resource is called “Policy” the mesh-scoped policy resource is called “MeshPolicy”.

Target selectors

To define which services are affected by the policies, target selectors are used. Target selectors are a list of rules to selected services that the policy should be applied. If a target selector is not provided, the policy is used on all services in the same namespace.

For example, a namespace-scope policy below would apply for the service-a (regardless of the ports) and service-b on port 8080:

apiVersion: authentication.istio.io/v1alpha1
kind: PeerAuthentication
metadata:
  name: sample-policy
  namespace: prod
spec:
  target:
  - name: service-a
  - name: service-b
    ports:
    - number: 8080

In the case of multiple policies, they get evaluated from the narrowest matching policy (e.g. service-specific), to namespace and the mesh wide. If more than one policies apply to a service, one is randomly chosen.

Transport authentication

The field called peers defines the authentication methods and any parameters for the method. At the time of writing this, the only supported authentication method is mTLS. To enable it, use the mtls key like this (using the previous example):

apiVersion: authentication.istio.io/v1alpha1
kind: PeerAuthentication
metadata:
  name: sample-policy
  namespace: prod
spec:
  target:
  - name: service-a
  - name: service-b
    ports:
    - number: 8080
  peers:
    - mtls:
...

Origin authentication

The only origin authentication currently supported by Istio is JWT. Using the origins field, you can define the method and parameters, such as allowed JWT issuers and enable or disable JWT authentication for a specific path. Here’s a sample snippet that shows how to define origin authentication that accepts JWTs issued by Google. Additionally, we are excluding the /health path from JWT authentication:

origins:
- jwt:
    issuer: https://accounts.google.com
    jwksUri: https://www.googleapis.com/oauth2/v3/certs
    trigger_rules:
    - excluded_paths:
      - exact: /health

Authorization

Authorization feature can be used to enable access control on workloads in the mesh. The policy supports both ALLOW and DENY policies. In case when you’re using both allow and deny policies at the same time, the deny policies get evaluated first. Each Envoy proxy uses an authorization engine that decides at runtime if requests should be allowed or denied.

When requests reach the proxy, the authorization engine evaluates the request and returns the authorization result - either ALLOW or DENY. The policies are evaluated in the following order:

1. If any DENY policy matches the request → deny the request
2. If there no ALLOW policies for the workload → allow the request
3. If any of the ALLOW policies match the request → allow the request
4. Deny the request

There is no need to separately enable any authorization features. It’s enough to create and apply an authorization policy for your workloads. By default, if there are no authorization policies defined, no access control is enforced and all requests are allowed.

Authorization policy is configured using the AuthorizationPolicy resource. This resource includes a selector (target workloads), action (allow or deny) and the list of rules that specify when to trigger the action.

For example, with the snippet below you can apply an authorization policy to any workloads with labels app=greeter-service and version=v2 set. Once the request comes to the Envoy proxy of the workload, the authorization engine checks if the traffic is coming from the principal with the provided service account (helloweb) and if the operation is a GET and the x-user header is set to user-1 - if all these are satisfied, the request is allowed, otherwise, the request gets denied.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
 action: ALLOW
 selector:
   matchLabels:
     app: greeter-service
     version: v2
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/helloweb"]
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.headers[x-user]
     values: ["user-1"]

We are specifically applying the authorization policy to workloads labelled with app: greeter-service and version: v2. If we wanted to apply the policy to all workloads in the default namespace, we could simply omit the selector field like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/helloweb"]
   to:
   - operation:
       methods: ["GET"]
   when:
   - key: request.headers[x-user]
     values: ["user-1"]

You can also define an authorization policy that applies to all workloads in your service mesh, regardless of the namespace. To do that, you need to create an AuthorizationPolicy in the root namespace. By default, the root namespace is istio-system. If you need to change it, you will have to update the rootNamespace field in the MeshConfig.

Value matching

You can use the following matching schemes for most fields in the authorization policy:

• Exact: matches an exact string
• Prefix: matches strings that start with the specified value ([prefix]*). For example: “hello.world” matches “hello.world.blah”, but not “blah.hello.world”
• Suffix: matches strings that end with the specified value (*[suffix]). For example: “hello.world” matches “blah.hello.world”, but not “hell.world.blah”
• Presence: matches any value, except empty (i.e. value must be provided, but we don’t care what it is as long as it’s not empty)

A couple of fields are exempted and only support exact matching:

• key field under the when section
• ipBlocks field under the source section
• ports field under the to section

Here’s an example of how to allow access to any path under /api as long as it’s a GET operation:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
  selector:
    matchLabels:
      app: greeter-service
  action: ALLOW
  rules:
   - to:
    - operation:
        methods: ["GET"]
        paths: ["/api/*"]

Exclusions

In addition to inclusion matching, Istio also support matching exclusions. This means you can match negative conditions like notValues, notPorts or notIpBlocks. The following snippet allows requests that are not under the /private path:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
  selector:
    matchLabels:
      app: greeter-service
  action: ALLOW
  rules:
   - to:
    - operation:
        notPaths: ["/private"]

Deny all and allow all

To create an allow all authorization policy that allows full access to all workloads in the specified namespace, you can create a policy with an empty rules section like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all
  namespace: default
spec:
  action: ALLOW
  rules:
  - {}

Similarly, you can deny access to all workloads by using an empty spec field:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  {}

Examples

To demonstrate the security features, we will deploy the Hello Web, Greeter service, and corresponding virtual service.

Start with the greeter deployment and service:

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: greeter-service-v1
  labels:
    app: greeter-service
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: greeter-service
      version: v1
  template:
    metadata:
      labels:
        app: greeter-service
        version: v1
    spec:
      containers:
        - image: learnistio/greeter-service:1.0.0
          imagePullPolicy: Always
          name: svc
          ports:
            - containerPort: 3000
---
kind: Service
apiVersion: v1
metadata:
  name: greeter-service
  labels:
    app: greeter-service
spec:
  selector:
    app: greeter-service
  ports:
    - port: 3000
      name: http
EOF

Then create the Hello web deployment and service:

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloweb
  labels:
    app: helloweb
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloweb
      version: v1
  template:
    metadata:
      labels:
        app: helloweb
        version: v1
    spec:
      containers:
        - image: learnistio/hello-web:1.0.0
          imagePullPolicy: Always
          name: web
          ports:
            - containerPort: 3000
          env:
            - name: GREETER_SERVICE_URL
              value: 'http://greeter-service.default.svc.cluster.local:3000'
---
kind: Service
apiVersion: v1
metadata:
  name: helloweb
  labels:
    app: helloweb
spec:
  selector:
    app: helloweb
  ports:
    - port: 3000
      name: http
EOF

Finally, create the Virtual service for the Hello web, so that we can expose it through the gateway. Don’t forget to deploy the gateway as well - check blog 3 for the snippet.

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloweb
spec:
  hosts:
    - '*'
  gateways:
    - gateway
  http:
    - route:
      - destination:
          host: helloweb.default.svc.cluster.local
          port:
            number: 3000
EOF

If you open http://$GATEWAY you should see the familiar Hello web with the response from the greeter service.

Let’s use an authorization policy that denies access to all workloads:

cat <<EOF | kubectl apply -f - 
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: default
spec: {}
EOF

With this config, we are denying access to all workloads in the default namespace.

Try refreshing the http://$GATEWAY or running curl http://$GATEWAY. This time, it won’t work and you will see the following error:

RBAC: access denied

To allow Hello Web service to call to the Greeter service we can update the authorization policy that explicitly allows Hello Web making requests to the Greeter service.

Let’s delete the previous policy first by running:

kubectl delete authorizationpolicy deny-all

Now we can create a new policy:

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
 selector:
  matchLabels:
    app: greeter-service
 rules:
  - to: 
    - operation:
        methods: ["GET"]
EOF

If you try to reaccess the site, you should be able to see the responses again. Note that there might be some delays due to caching.

Let’s tighten up the service role a bit more and update the authorization policy, so we can only call the /hello endpoint on it:

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: greeter-service
 namespace: default
spec:
 selector:
  matchLabels:
    app: greeter-service
 rules:
  - to: 
    - operation:
        methods: ["GET"]
        paths: ["*/hello"]
EOF

To test this, we will get a shell inside the Hello web container and use curl to make requests to the greeter service. First, let’s figure out the Hello web pod name by running kubectl get pod and then run the exec command:

kubectl exec -it [podname] /bin/sh

With the shell inside the container, let’s install curl first:

apk add curl

We can test out the service role now. If you run curl against the /hello endpoint, everything works as expected:

curl greeter-service.default.svc.cluster.local:3000/hello
{"message":"hello 👋 ","version":"1.0.0"}

However, if you make a request against the /version endpoint, you will see the familiar error message:

curl greeter-service.default.svc.cluster.local:3000/version
RBAC: access denied

To clean everything, simply delete the authorization policy.

Conclusion

In this blog, you learned about how Istio service mesh deals with the security and different features you can use to define authorization policies through a couple of examples.