A Couple of Terraform Gotchas

I am putting together a talk All the Cool Kids Use Terraform to Deploy Kubernetes …So Can You!, for Boston Code Camp. My talk centers around deploying an Azure Kubernetes Service, with a running application, using Terraform. I want to show a couple of different Terraform providers to deploy different pods to the cluster, so I am using the Helm provider to install an Nginx ingress controller and the Kubernetes Provider to install a web application. I ran into a couple of little gotchas that I thought I would share in case someone else is having the same issues.

kubectl config

For my demo, I create an AKS cluster; the HCL (HashiCorp Configuration Language) was pretty straight forward. So I went to use Helm to install the Nginx ingress controller and got the following error:

Error: invalid configuration: more than one authentication method found for userid; found [token basicAuth], only one is allowed

So I went and checked the documentation for the Helm Provider and ensure that I was configuring the correct authentication for Kubernetes

My code matched the example provided by the documentation. I tried setting the values in a few different ways. Always with the same results. I did what any stuck developer would do….go to the Googles. I didn’t find much. I went back to the provider documentation and saw this right below the example configuration

If you have both valid configuration in a config file and static configuration, the static one is used as override. i.e. any static field will override its counterpart loaded from the config.

The problem I was running into was my kubectl current-context had a token field that wasn’t being overridden, and Helm was confused. The solution was to change my current-context to a context that contained the fields I was configuring. I have since run into this quite often in my testing. After I run my script, I would switch my context to be that of the AKS instance. When I go and run terraform destroy, I get the same error because I did not switch my context. Eventually, I will remember.

Know Your Load Balancers

Once I got past my config issue, I was excited. I was finally going to get to see Helm deploy my ingress controller. I fired up terraform apply, waited the few minutes for the AKS instance to be provision the I get to the where the Helm is making its call….and I wait….and I wait (the Helm providers default timeout is set to 300 seconds). I see the following:

The first thing I tried was to increase the timeout to no avail. Ok, back to the Googles. The answers I was seeing were "the resource never finished creating before the timeout". Not helpful. What was the controller trying to do? I went to the Azure Management portal and saw that my cluster had been created. So I did what any Kubernetes developer would do, change my context to that cluster, and start running kubectl commands.

kubectl get all

Can you see what could be causing the issue? I’ll give you a hint: check out the External IP column. Why is it pending? My HCL creates an Azure public IP address in the node namespace like I am supposed to do. When I manually set up ingress controllers it is set right away. I never see pending. So I run:

kubectl describe service nginx-ingress-controller

Under the events, I see the following:

I have never run into this issue when I manually installed ingress controllers, so why was I running into it when I use Terraform. It seems that the Azure AKS resource in Terraform does not create a Standard SKU load balancer by default. I had to add the following block of code to my resource:

The creation of the Ingress controller no longer times out with this change. It now takes a few seconds to be created.

Summary

Working with Terraform can be frustrating at times. But once all the issues are worked out, it is nice to be able to deploy an application, it’s infrastructure, and all the dependencies, with a single command. I hope that this will help someone who is running to the same or similar issues as I did.