While managing my kubernetes cluster, I needed to install several helm charts to do things like:
- Installing ingress controller in order to connect my cluster's ingress with my cloud-provider's load balancer
- Installing
metrics-server to be able to get basic metrics about my cluster
In doing so, I found myself writing commands like
helm repo add bitnami https://charts.bitnami.com/bitnami
helm upgrade --install metrics-server bitnami/metrics-server \
--create-namespace --namespace metrics-server \
--set apiService.create=true \
--set extraArgs.kubelet-insecure-tls=true \
--set extraArgs.kubelet-preferred-address-types=InternalIP \
--set hostNetwork=true
in the command line to install the required resources then forget about it.
I didn't like this, as it does not document what I have installed, and does not document how I installed it, and if I needed to install the same infrastructure on another cluster (say from another provider) I will need to scan through my command line history to find the commands I used to install the needed resources.
I began experimenting with creating my own helm chart for my setup, specifying all the required dependencies and so. But as I learned more about it I felt it was too much for what I wanted.
Enters helmfile
Searching about a simpler solution, I found helmfile. The whole idea is that you create a yaml file consisting of your dependencies charts, and you use helmfile to install them.
It was exactly what I wanted.
The configuration file
First, we need to write a yaml file that describes the helm charts that needs to be installed, and specify some details about the related repo, namespace, and other things.
Let's take another look at the commands I ran earlier
helm repo add bitnami https://charts.bitnami.com/bitnami
helm upgrade --install metrics-server bitnami/metrics-server \
--create-namespace --namespace metrics-server \
--set apiService.create=true \
--set extraArgs.kubelet-insecure-tls=true \
--set extraArgs.kubelet-preferred-address-types=InternalIP \
--set hostNetwork=true
We can extract some values from those 2 commands, like repo URL (https://charts.bitnami.com/bitnami), chart name (bitnami/metrics-server), release name (metrics-server), installation namespace (metrics-server), and some flags (--create-namespace flag), and some values that we set using --set flag.
From the above information, and with some help from the documentation of helmfile we can create the following schema
repositories:
- name: bitnami
url: https://charts.bitnami.com/bitnami
releases:
- name: metrics-server
chart: bitnami/metrics-server
namespace: metrics-server
createNamespace: true
set:
- name: apiService.create
value: true
- name: extraArgs.kubelet-insecure-tls
value: true
- name: extraArgs.kubelet-preferred-address-types
value: InternalIP
- name: hostNetwork
value: true
Now we are only missing 1 required field for the release, and that is the version of the chart. We can get the version number by running
helm repo add bitnami https://charts.bitnami.com/bitnami
helm show chart bitnami/metrics-server
I.e. we add the repo, then call helm show chart {repo}/{chart}.
As of the time of writing this article, the latest version of the metrics-server is 6.0.7, but it has a bug so I am using an older version of 5.11.3
Digging through the documentation of helmfile, we can also make use of atomic option that restores previous state in case of failed release, so that the cluster does not change if the installation failed.
On last thing that I like to do is to set the cluster context in the configuration file, I manage several clusters, and I don't want one cluster's configurations to be applied onto another one, so in the root of the configuration file I added:
helmDefaults:
kubeContext: my-cluster-context
Now, the final configuration file looks like:
helmDefaults:
kubeContext: my-cluster-context
repositories:
- name: bitnami
url: https://charts.bitnami.com/bitnami
releases:
- name: metrics-server
chart: bitnami/metrics-server
version: 5.11.3
namespace: metrics-server
createNamespace: true
atomic: true
set:
- name: apiService.create
value: true
- name: extraArgs.kubelet-insecure-tls
value: true
- name: extraArgs.kubelet-preferred-address-types
value: InternalIP
- name: hostNetwork
value: true
Applying The Configurations
We could download the binaries from Github repo, then install some dependencies, then use that binary to install our configuration, but that's not my favorite method to do it.
My favorite method is by using Docker.
In their installation section they have a section for "Running as a container", so let's use that.
I created a small bash file based on the command in the documentation, and I put it in the same directory as the configuration file we made earlier. Here are the contents of the bash file...
current_context=$(kubectl config current-context)
kubectl config view --flatten > ~/.kube/config
kubectl config use-context $current_context
docker run \
--rm \
--net=host \
-v "${HOME}/.kube:/root/.kube" \
-v "${HOME}/.config/helm:/root/.config/helm" \
-v "${PWD}:/wd" --workdir /wd \
quay.io/roboll/helmfile:helm3-v0.135.0 helmfile sync
The command kubectl config view --flatten > ~/.kube/config is used to write all my kubernetes contexts into the file ~/.kube/config as the docker image reads the contexts from that file.
The first and the third commands are to keep the state of the current context, as it tends to change when I write the contexts into the config file.
All what's left is to run the bash file. It will take a minute, then print something like the following...
Adding repo bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
Affected releases are:
metrics-server (bitnami/metrics-server) UPDATED
Upgrading release=metrics-server, chart=bitnami/metrics-server
Listing releases matching ^metrics-server$
Release "metrics-server" has been upgraded. Happy Helming!
NAME: metrics-server
LAST DEPLOYED: Sat Jun 18 04:38:44 2022
NAMESPACE: metrics-server
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
CHART NAME: metrics-server
CHART VERSION: 5.11.3
APP VERSION: 0.6.1
** Please be patient while the chart is being deployed **
The metric server has been deployed.
In a few minutes you should be able to list metrics using the following
command:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
metrics-server metrics-server 5 2022-06-18 04:38:44.662123115 +0000 UTC deployed metrics-server-5.11.3 0.6.1
And that's it, metrics server should now be installed in a namespace called "metrics-server" and we should be able to get some basic metrics about the cluster by running kubectl top nodes.
We can now maintain that configuration file to include any dependency that we need to install using helm. The configurations can, and should be, pushed to version control; we can also setup some simple CI/CD to auto deploy the file on any changes.
