Fun with RBAC, or How I Hacked Into My Own Kubernetes Cluster

This is an exploratory post, where I describe a problem I ran into and the workaround I found. I make no promises of my reasoning being correct. I'm just sharing in the hopes it might help someone, start a discussion, or maybe encourage someone to explain in a comment.

Last week, I ran into a very fun problem.

I was supposed to deploy a custom helm chart for an app on a new Kubernetes cluster.
Basic steps:

1. spin up a Kubernetes cluster on GKE
2. deploy a chart to it from our chart repo on Quay
3. expose the deployment

For Step One, I followed the Quickstart Guide from Google Cloud and ran from my terminal

gcloud config list

to make sure I was in the correct project space. Then, I ran

gcloud container clusters create guinscluster

and waited for the spinner to stop turning.

To check that I had my nodes up, I ran

gcloud container clusters get-credentials guinscluster

so that I could use kubectl to mess with my cluster. A quick run of

kubectl get nodes

confirmed that I had a working cluster with three nodes of Kubernetes Version 1.8.8-gke-0. Yay!

For the next step, I knew I needed to refresh my knowledge of helm a bit. Again, I used their Quickstart guide for help. The chart repo I had to work with used a script to generate a Chart.yaml in CI, which would then be deployed to our app registry on Quay. I had to generate my own local Chart.yaml from a Chart.yaml.in file, which wound up being confusing.

Anyhow, I went ahead and followed the guide and ran

helm init

What this does is it will deploy a thing called Tiller on a pod into your cluster. Tiller is what enables Helm to do its thing. To check whether I successfully started Tiller, I ran

kubectl get pods --all-namespaces

You must run this check with the --all-namespaces flag, because Tiller gets deployed into the kube-system namespace.

Here's what I saw:

Yay, tiller works!

That was to be the end of my success for the day.

I followed the guide and attempted to run

helm install ./myawesomechart

and got the following fun error:

Error: release chart-technical-on-boarding failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "default": Unknown user "system:serviceaccount:kube-system:default"

So that was cool. No matter what I did, I could not install that chart. My coworkers tried to help me out by suggesting I deploy it with a name, or into a namespace, or running helm package but no matter what I tried, I could not get this to work. Nothing in the helm user guide had prepared me for this.
I decided clearly the problem was I didn't know enough about Helm and went to this tutorial to create my own helm chart from scratch. 

helm install --name example mychart-0.1.0.tgz
Error: release example failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "default": Unknown user "system:serviceaccount:kube-system:default"

Same problem!

So here is where I don't quite remember how I suspected what the problem was. I Googled around a bit for the error messages and I decided that maybe the problem wasn't with helm, but with my cluster and the access I had to it. RBAC is a fascinating thing that I have run into a couple times, so I went with my hunch.

I found this article on Google Cloud (and by "found", I mean a coworker sent it to me - teamwork is great, everyone!) and it seemed to indicate that GKE clusters running Kubernetes version 1.6 or later already came with RBAC enabled. This did not mean what I thought it did. However, I ran

gcloud container clusters update --no-enable-legacy-authorization guinscluster

anyway, and followed through with the suggested

gcloud components update

No change to my ability to install a helm chart.

Okay.

So I looked around some more and I found this Github issue. Okay. So it seemed the problem was that for clusters with RBAC enabled one needed to create the correct permissions to actually use helm. I'm still puzzled why it's okay to install tiller, but not charts? Oh well! So I tried it:

kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin

 which resulted in

Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "guin@fake-email.com" cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope: Required "container.clusterRoleBindings.create" permission.

You know the feeling when you get a new, different error? It's this frustration mixed with guilt because getting a new error is supposed to be a good thing, it means you're finding out more? But also it could mean you're falling further behind since maybe you're taking the wrong track?

Anyway. So my question is, if I bring up a shiny new GKE cluster, version 1.9, why on earth would I not have user permissions on it? I'm sure there's a good reason; I just don't know why, and I wasn't expecting it.

StackOverflow to the rescue. If I could get the "admin" user's password for the cluster, I could pass user and password as flags for the clusterrolebinding and make it happen.

gcloud container clusters describe guinscluster

After that it was easy to copy the password from the result and pass it to the above command via flag like so:

kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin --username=admin --password=<adminpw>

After this, helm install worked like a charm, both on the tutorial example chart, and on guinsawesomechart. I have no idea why this song and dance was necessary, or how I should have spun up my cluster so that my identity had user permissions. This is something I hope to learn more about and bring it back here.

I also, as always, welcome comments.

Guin out. Time for KubeCon.