Out of the box, the Kubernetes authentication is not very user-friendly for end users. In this lab, we will see how to integrate Active Directory with Kubernetes to give the easiest authentication experience to the end users.
For this, we will use a project called Dex. Dex is an OpenID Connect provider done by CoreOS. It take care of the translation between Kubernetes tokens and Active Directory users. We will also use Heptio Gangway to generate kubectl configuration files for us, and Bitly OAuth2 Proxy to forward the OpenID token to the Kubernetes dashboard.
Requirements
You will need an ISO of Windows Server 2016 and an IP on your network for the Active Directory server. In my case, this IP will be 10.10.40.5/24. You will also need a working Kubernetes cluster, and the nodes of this cluster should be able to communicate with the Active Directory IP.
As we will use the awesome Let's Encrypt service to sign the certificates for the different components of our authentication mechanism, you will also need a way to NAT external traffic to the Kubernetes cluster. I will use OPNSense for this and I will also use OPNSense as a local DNS server. My OPNSense machine has the IP 10.10.40.1/24.
You will also need a domain name that supports wildcard DNS entry. I will use the wildcard DNS "*.k8s.inkubate.io" to route external traffic to my Kubernetes cluster.
If your Kubernetes cluster is on-prem, like mine, you will need a load balancer to route the external traffic to your Kubernetes services. I suggest that you install MetalLB on your cluster for this. You can refer to the Install and configure MetalLB as a load balancer for Kubernetes article.
Your Kubernetes cluster should have a working certificate manager to automatically sign SSL certificates via Let's Encrypt. If you don't have one yet, you can refer to the Automatically generate signed SSL certificates for your Kubernetes web applications article.
Install an Active Directory server
Create a Windows Server 2016 machine
1- Create a new virtual machine.
2- Choose a name for your virtual machine.
3- Select the location of your virtual machine.
4- Select the compatibility of your virtual machine.
5- Select the type of the guest OS.
6- Attach the Windows Server 2016 ISO to the virtual machine.
7- Create the virtual machine.
8- Power-on the virtual machine.
9- Open a remote console.
10- Choose to boot normally and press a key to boot on the Windows Server 2016 ISO.
11- Select your language and time zone.
12- Start the installation of Windows Server 2016.
13- Choose the type of installation and accept the license terms.
14- Choose to do a custom installation.
15- Select the disk on which to install Windows Server 2016.
16- Choose a password for the Administrator user.
17- Login to your new Windows Server 2016.
18- Right-click on the network card and "open the network and sharing center".
19- Click on the "Ethernet0" network card.
20- Open the properties of the network card.
21- Edit the IPV4 properties.
22- Open the "Server Manager"
23- Enable the remote desktop connection for the local server.
24- You should now be able to access your Windows Server 2016 with a remote desktop connection.
25- Go back to the VMware vSphere client and install the VMware tools.
26- Go back to your remote desktop connection and launch the VMware tools installer.
27- Install the VMware tools.
28- Reboot the machine.
29- When the machine is up, add your Windows Server 2016 license.
Add the Active Directory role to the Windows Server 2016
1- In the "Server Manager", select "Add roles and features"
2- Select the installation type.
3- Select your server.
4- Add the role "Active Directory Domain Services".
5- Promote your Windows Server 2016 to domain controller.
6- Choose your root domain name.
7- Enter a password for the restore mode.
8- Don't create a DNS delegation.
9- Choose the NetBIOS domain name.
10- Leave the default path.
11- Start the installation.
12- Reboot the server.
Create an Active Directory user
1- Launch the Active Directory Administrative Center.
2- Select your domain.
3- Right-click on Users and select New User.
4- Fill the user information, select the password policy and click on OK.
Create a role binding for your user
We are going to create a Kubernetes role binding for your Active Directory user. This will give permissions to your user on the Kubernetes cluster. In this example, the user will be administrator of the default namespace.
1- Create the role binding.
$ kubectl create rolebinding sguyennet-default-admin --clusterrole=admin --user=sguyennet
Deploy Dex
Dex is an OpenID Connect provider that will be in charge of our authentication. We will use Active Directory as a backend for Dex, but there are many other backend solutions to choose from.
1- Create a dex-namespace.yaml file.
$ vim dex-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: auth-system
2- Create the namespace for Dex.
$ kubectl apply -f dex-namespace.yaml
3- Create a dex-rbac.yaml file.
$ vim dex-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: dex
namespace: auth-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: dex
namespace: auth-system
rules:
- apiGroups: ["dex.coreos.com"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: dex
namespace: auth-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dex
subjects:
- kind: ServiceAccount
name: dex
namespace: auth-system
4- Create the permissions for Dex.
$ kubectl apply -f dex-rbac.yaml
5- Create a dex-configmap.yaml file. Modify the issuer URL, the redirect URIs, the client secret and the Active Directory configuration accordingly.
$ vim dex-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: dex
namespace: auth-system
data:
config.yaml: |
issuer: https://auth.k8s.inkubate.io/
web:
http: 0.0.0.0:5556
frontend:
theme: custom
telemetry:
http: 0.0.0.0:5558
staticClients:
- id: oidc-auth-client
redirectURIs:
- 'https://kubectl.k8s.inkubate.io/callback'
- 'http://dashboard.k8s.inkubate.io/oauth2/callback'
name: 'oidc-auth-client'
secret: ***********
connectors:
- type: ldap
id: ldap
name: LDAP
config:
host: ad.inkubate.io:389
insecureNoSSL: true
insecureSkipVerify: true
bindDN: cn=Administrator,cn=Users,dc=inkubate,dc=io
bindPW: '***********'
userSearch:
baseDN: cn=Users,dc=inkubate,dc=io
filter: "(objectClass=user)"
username: sAMAccountName
idAttr: sAMAccountName
emailAttr: sAMAccountName
nameAttr: displayName
oauth2:
skipApprovalScreen: true
storage:
type: kubernetes
config:
inCluster: true
6- Configure Dex.
$ kubectl apply -f dex-configmap.yaml
7- Create the dex-deployment.yaml file.
$ vim dex-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: dex
name: dex
namespace: auth-system
spec:
replicas: 1
selector:
matchLabels:
app: dex
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: dex
revision: "1"
spec:
initContainers:
- name: dl-theme
image: alpine/git
command:
- git
- clone
- "https://github.com/sguyennet/dex-inkubate-branding.git"
- /theme
volumeMounts:
- name: theme
mountPath: /theme/
containers:
- command:
- /usr/local/bin/dex
- serve
- /etc/dex/cfg/config.yaml
image: quay.io/dexidp/dex:v2.17.0
imagePullPolicy: IfNotPresent
name: dex
ports:
- containerPort: 5556
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/dex/cfg
name: config
- mountPath: /web/themes/custom/
name: theme
dnsPolicy: ClusterFirst
serviceAccountName: dex
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
items:
- key: config.yaml
path: config.yaml
name: dex
name: config
- name: theme
emptyDir: {}
8- Deploy Dex.
$ kubectl apply -f dex-deployment.yaml
9- Create a dex-service.yaml file.
$ vim dex-service.yaml
apiVersion: v1
kind: Service
metadata:
name: dex
namespace: auth-system
spec:
selector:
app: dex
ports:
- name: dex
port: 5556
protocol: TCP
targetPort: 5556
10- Create a service for the Dex deployment.
$ kubectl apply -f dex-service.yaml
11- Create a dex-ingress.yaml file. Change the host parameters and your certificate issuer name accordingly.
$ vim dex-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dex
namespace: auth-system
annotations:
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- secretName: dex
hosts:
- auth.k8s.inkubate.io
rules:
- host: auth.k8s.inkubate.io
http:
paths:
- backend:
serviceName: dex
servicePort: 5556
12- Create the ingress for the Dex service.
$ kubectl apply -f dex-ingress.yaml
13- Wait a couple of minutes until the cert manager generates a certificate for Dex and check that Dex is deployed properly by browsing to https://auth.k8s.inkubate.io/.well-known/openid-configuration.
Configure the Kubernetes API to access Dex as OpenID connect provider
These steps have to be done on each of your Kubernetes master nodes.
1- SSH to your node.
$ ssh sguyennet@10.10.40.30
2- Edit the Kubernetes API configuration. Add the OIDC parameters and modify the issuer URL accordingly.
$ sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
...
command:
- /hyperkube
- apiserver
- --advertise-address=10.10.40.30
...
- --oidc-issuer-url=https://auth.k8s.inkubate.io/
- --oidc-client-id=oidc-auth-client
- --oidc-username-claim=email
- --oidc-groups-claim=groups
...
3- The Kubernetes API will restart by itself.
Deploy Gangway
Gangway is a web interface made by Heptio. It will allow us to configure kubectl with our user settings.
1- Generate a secret key for Gangway.
$ kubectl -n auth-system create secret generic gangway-key \
--from-literal=sesssionkey=$(openssl rand -base64 32)
2- Create a gangway-configmap.yaml file. Modify the cluster name, the URLs, and the client secret accordingly. For the client secret, use the same secret that you specified in the Dex configmap during the previous step.
$ vim gangway-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: gangway
namespace: auth-system
data:
gangway.yaml: |
clusterName: "Inkubate"
apiServerURL: "https://10.10.40.33:6443"
authorizeURL: "https://auth.k8s.inkubate.io/auth"
tokenURL: "https://auth.k8s.inkubate.io/token"
clientID: "oidc-auth-client"
clientSecret: "***********"
redirectURL: "https://kubectl.k8s.inkubate.io/callback"
scopes: ["openid", "profile", "email", "offline_access"]
usernameClaim: "email"
emailClaim: "email"
3- Configure Gangway.
$ kubectl apply -f gangway-configmap.yaml
4- Create a gangway-deployment.yaml file.
$ vim gangway-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gangway
namespace: auth-system
labels:
app: gangway
spec:
replicas: 1
selector:
matchLabels:
app: gangway
strategy:
template:
metadata:
labels:
app: gangway
revision: "1"
spec:
containers:
- name: gangway
image: gcr.io/heptio-images/gangway:v2.0.0
imagePullPolicy: Always
command: ["gangway", "-config", "/gangway/gangway.yaml"]
env:
- name: GANGWAY_SESSION_SECURITY_KEY
valueFrom:
secretKeyRef:
name: gangway-key
key: sesssionkey
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "100m"
memory: "100Mi"
volumeMounts:
- name: gangway
mountPath: /gangway/
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 20
timeoutSeconds: 1
periodSeconds: 60
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 1
periodSeconds: 10
failureThreshold: 3
volumes:
- name: gangway
configMap:
name: gangway
5- Create the Gangway deployment.
$ kubectl apply -f gangway-deployment.yaml
6- Create a gangway-service.yaml file.
$ vim gangway-service.yaml
kind: Service
apiVersion: v1
metadata:
name: gangway-svc
namespace: auth-system
labels:
app: gangway
spec:
type: ClusterIP
ports:
- name: "http"
protocol: TCP
port: 80
targetPort: "http"
selector:
app: gangway
7- Create the service for the Gangway deployment.
$ kubectl apply -f gangway-service.yaml
8- Create a gangway-ingress.yaml file. Modify the host parameter and the certificate manager issuer accordingly.
$ vim gangway-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gangway
namespace: auth-system
annotations:
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- secretName: gangway
hosts:
- kubectl.k8s.inkubate.io
rules:
- host: kubectl.k8s.inkubate.io
http:
paths:
- backend:
serviceName: gangway-svc
servicePort: http
9- Create the ingress for the Gangway service.
$ kubectl apply -f gangway-ingress.yaml
10- Wait a couple of minutes while the certificate manager generates a SSL certificate for Gangway and browse to https://kubectl.k8s.inkubate.io
11- Click on "Sign In".
12- Login with your Active Directory user.
13- Copy your cluster administrator configuration.
$ cd ~/.kube
$ cp config admin-config
14- Follow the steps to generate your kubectl configuration file.
15- You should now be logged in with your Active Directory user and you should be able to list the pods in the default namespace, but not in the kube-system namespace.
$ kubectl get pods
$ kubectl get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "sguyennet" cannot
list pods in the namespace "kube-system"
16- Log back with your cluster administrator user.
$ export KUBECONFIG=~/.kube/admin-config
$ kubectl get pods -n kube-system
Deploy the Kubernetes dashboard
1- Create the Kubernetes dashboard manifest.
$ vim kubernetes-dashboard.yaml
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.8.
#
# Example usage: kubectl create -f
# ------------------- Dashboard Secret ------------------- #
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kube-system
type: Opaque
---
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1beta2
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
ports:
- containerPort: 8443
protocol: TCP
args:
- --auto-generate-certificates
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
2- Deploy the dashboard.
$ kubectl create -f kubernetes-dashboard.yaml
Deploy the Oauth2 proxy
We are going to use a modified version of the Bitly Oauth2 proxy to pass the authentication token to the Kubernetes dashboard. Thanks to Joel Speed for the modification. You can find the source code on his GitHub.
1- Generate a secret for the Oauth2 proxy.
python -c 'import os,base64; print base64.urlsafe_b64encode(os.urandom(16))'
2- Copy the generated secret and use it for the OAUTH2_PROXY_COOKIE_SECRET value in the next step.
3- Create an oauth2-proxy-deployment.yaml file. Modify the OIDC client secret, the OIDC issuer URL, and the Oauth2 proxy cookie secret accordingly.
$ vim oauth2-proxy-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: auth-system
spec:
replicas: 1
selector:
matchLabels:
k8s-app: oauth2-proxy
template:
metadata:
labels:
k8s-app: oauth2-proxy
spec:
containers:
- args:
- --cookie-secure=false
- --provider=oidc
- --client-id=oidc-auth-client
- --client-secret=***********
- --oidc-issuer-url=https://auth.k8s.inkubate.io/
- --http-address=0.0.0.0:8080
- --upstream=file:///dev/null
- --email-domain=*
- --set-authorization-header=true
env:
# docker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'
- name: OAUTH2_PROXY_COOKIE_SECRET
value: ***********
image: sguyennet/oauth2-proxy:header-2.2
imagePullPolicy: Always
name: oauth2-proxy
ports:
- containerPort: 8080
protocol: TCP
4- Deploy the Oauth2 proxy.
$ kubectl apply -f oauth2-proxy-deployment.yaml
5- Create an oauth2-proxy-service.yaml file.
$ vim oauth2-proxy-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: auth-system
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
k8s-app: oauth2-proxy
6- Create a service for the Oauth2 proxy deployment.
$ kubectl apply -f oauth2-proxy-service.yaml
7- Create a dashboard-ingress.yaml file. Modify the dashboard URLs and the host parameter accordingly.
$ vim dashboard-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: kube-system
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://dashboard.k8s.inkubate.io/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://dashboard.k8s.inkubate.io/oauth2/start?rd=https://$host$request_uri$is_args$args"
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
auth_request_set $token $upstream_http_authorization;
proxy_set_header Authorization $token;
spec:
rules:
- host: dashboard.k8s.inkubate.io
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 443
path: /
8- Create the ingress for the dashboard service.
$ kubectl apply -f dashboard-ingress.yaml
9- Create an oauth2-proxy-ingress.yaml file. Modify the certificate manager issuer and the host parameters accordingly.
$ vim oauth2-proxy-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/cluster-issuer: "letsencrypt-production"
ingress.kubernetes.io/force-ssl-redirect: "true"
name: oauth-proxy
namespace: auth-system
spec:
rules:
- host: dashboard.k8s.inkubate.io
http:
paths:
- backend:
serviceName: oauth2-proxy
servicePort: 8080
path: /oauth2
tls:
- hosts:
- dashboard.k8s.inkubate.io
secretName: kubernetes-dashboard-external-tls
10- Create the ingress for the Oauth2 proxy service.
$ kubectl apply -f oauth2-proxy-ingress.yaml
11- Wait a couple of minutes and browse to https://dashboard.k8s.inkubate.io.
12- Login with your Active Directory user.
13- You should be able to see and modify the default namespace, but not the other ones.
Conclusion
Success! You are now able to access your Kubernetes cluster, in a user-friendly fashion, both with the kubectl command line and with the dashboard.