Let’s prepare for GKE cluster versions >= 1.19

Google cloud has taken two significant changes in the GKE cluster version from 1.18 to 1.19 that we need to pay more attention to.

1. Google Kubernetes Engine doesn’t support the basic authentication for accessing the Kubernetes APIs.

2. Kubernetes controller manager image change from Debian to distroless.

Considering the above changes, here we are talking about the 1st change because that is the change most of the applications used for building the application coordination in the Kubernetes environment.

First, you have to identify your application components that use the Kubernetes APIs. Most of the applications which built early stage of the Kubernetes versions probably that application used the basic authentication for calling the Kubernetes APIs.

“Basic Authentication means that you have to enter the username and password when making the API request to get authenticated.”

Below API call can get all the endpoint details from your Kubernetes cluster using the basic authentication.

curl -u username:password https://<cluster_endpoint>/api/v1/namespaces/default/endpoints

After upgrading your cluster to the GKE version 1.19 or above, this API call wasn’t work because upgraded GKE versions do not support basic authentication. You can get the below error for the above API call,

{
   "kind":"Status",
   "apiVersion":"v1",
   "metadata":{
   },
   "status":"Failure",
   "message":"endpoints is forbidden: User \"system:anonymous\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"default\"",
   "reason":"Forbidden",
   "details":{
      "kind":"endpoints"
   },
   "code":403

The solution is to use the Token-based authentication instead of the basic authentication. You have to follow the below steps to generate the token.

Step 01

You have to create the service account for your namespace in the Kubernetes cluster.

Service account name = my-service-account

Namespace = my-namespace

kubectl create serviceaccount my-service-account -n my-namespace

You can verify by list downing the service account,

kubectl get sa -n my-namespace

Step 2

You have to create the cluster role by allowing the allowed accesses.

Create the cluster-role.yaml file using the below content.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-cluster-role
  namespace: my-namespace
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

You can apply the changes by executing the below command,

kubectl apply -f cluster-role.yaml

my-cluster-role has full access to all Kubernetes cluster resources. In your scenarios, you can choose the access level and the resource that your cluster role should access. Based on that, you can adjust the able yaml file.

Step 3

We have to map our service account and the cluster role by creating the CluserRole Binding.

Create the cluster-binding.yaml file using the below content.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-cluster-role-binding
subjects:
- namespace: my-namespace 
  kind: ServiceAccount
  name: my-service-account 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: my-cluster-role

You can apply the changes by executing the below command,

kubectl apply -f cluster-binding.yaml

Step 4

Getting the token which is created for the service account.

kubectl get sa -n my-namespace
kubectl describe sa my-service-account -n my-namespace

Get token name from the above command results.

kubectl get secrets -n my-namespace

Get the correct token similar to the token name that is returned from the description of the service account.

kubectl describe secrets <token_name> -n my-namespace

Now, let’s call the API call again with the token –

curl https://<cluster_endpoint>/api/v1/namespaces/default/endpoints -H “Authorization: Bearer <token>”

You can get the correct results, and the curl is authenticated through the bearer token.

Try to migrate your GKE version to the new GKE version. Don’t wait until GCP does the force upgrades.