Kubernetes Cluster Add-On Bootstrapping: Part 2, TKG
In the first part, I illustrated a simple and flexible way to automate the add-on deployment process to TKGI clusters using a container that clones a Git repo and applies those manifests. In this part, I'm going to show how you can use that same method with Tanzu Kubernetes Grid (TKG) clusters.
Just to levelset since there may be confusion on the differences between TKGI and TKG, Tanzu Kubernetes Grid Integrated Edition (TKGI) is a rename of what was once Enterprise PKS (AKA VMware PKS AKA just "PKS" if you will). This is an offering jointly developed by Pivotal and VMware which uses BOSH as the lifecycle manager and, most commonly, uses NSX-T for its CNI. Tanzu Kubernetes Grid (TKG), although sharing most of the same letters in the acronym, is a completely separate product which is based on Cluster API rather than BOSH and is in its nascent stages presently. The TKG offering I'm covering here is the so-called "multi-cloud" or "standalone" offering which does not ship with vSphere with Kubernetes (either Standard, requiring VCF, or Basic, without). The steps I outline here should also work with the TKG "service" flavor that ships with vSphere with Kubernetes, but only with the tkg CLI tool used in addition.
In the TKGI scenario in part one, you'll recall we leveraged Ops Manager's add-ons functionality found within a plan to store our bootstrap scaffolding manifests. TKG has no Ops Manager, but it does have "plans" per se. Specifically, these plans define the Cluster API resources and other files needed to build a K8s cluster. In order for us to insert our add-ons bootstrapper, we will need to create and modify some of these files.
If, after creating a TKG management cluster, you check your home directory, you'll find a file tree structure like the following below.
NOTE: You can expand and collapse long code blocks like these in our Clarity theme for Hugo.
1$ tree ~/.tkg/
2/home/chip/.tkg/
3├── bom
4│ ├── bom-1.1.0+vmware.1.yaml
5│ ├── bom-1.1.2+vmware.1.yaml
6│ ├── bom-1.1.3+vmware.1.yaml
7│ ├── bom-1.17.11+vmware.1.yaml
8│ ├── bom-1.17.6+vmware.1.yaml
9│ ├── bom-1.17.9+vmware.1.yaml
10│ ├── bom-1.18.8+vmware.1.yaml
11│ ├── bom-1.2.0+vmware.1.yaml
12│ └── bom-tkg-1.0.0.yaml
13├── config.yaml
14├── features.json
15└── providers
16 ├── bootstrap-kubeadm
17 │ └── v0.3.10
18 │ └── bootstrap-components.yaml
19 ├── cluster-api
20 │ └── v0.3.10
21 │ └── core-components.yaml
22 ├── config.yaml
23 ├── config_default.yaml
24 ├── control-plane-kubeadm
25 │ └── v0.3.10
26 │ └── control-plane-components.yaml
27 ├── infrastructure-aws
28 │ ├── v0.5.5
29 │ │ ├── cluster-template-definition-dev.yaml
30 │ │ ├── cluster-template-definition-prod.yaml
31 │ │ ├── infrastructure-components.yaml
32 │ │ └── ytt
33 │ │ ├── base-template.yaml
34 │ │ └── overlay.yaml
35 │ └── ytt
36 │ └── aws-overlay.yaml
37 ├── infrastructure-azure
38 │ ├── v0.4.8
39 │ │ ├── cluster-template-definition-dev.yaml
40 │ │ ├── cluster-template-definition-prod.yaml
41 │ │ ├── infrastructure-components.yaml
42 │ │ └── ytt
43 │ │ ├── base-template.yaml
44 │ │ └── overlay.yaml
45 │ └── ytt
46 │ └── azure-overlay.yaml
47 ├── infrastructure-docker
48 │ ├── v0.3.10
49 │ │ ├── cluster-template-definition-dev.yaml
50 │ │ ├── cluster-template-definition-prod.yaml
51 │ │ ├── infrastructure-components.yaml
52 │ │ ├── metadata.yaml
53 │ │ └── ytt
54 │ │ ├── base-template.yaml
55 │ │ └── overlay.yaml
56 │ └── ytt
57 │ └── docker-overlay.yaml
58 ├── infrastructure-tkg-service-vsphere
59 │ ├── v1.0.0
60 │ │ ├── cluster-template-definition-dev.yaml
61 │ │ ├── cluster-template-definition-prod.yaml
62 │ │ └── ytt
63 │ │ ├── base-template.yaml
64 │ │ └── overlay.yaml
65 │ └── ytt
66 │ └── tkg-service-vsphere-overlay.yaml
67 ├── infrastructure-vsphere
68 │ ├── v0.7.1
69 │ │ ├── cluster-template-definition-dev.yaml
70 │ │ ├── cluster-template-definition-prod.yaml
71 │ │ ├── infrastructure-components.yaml
72 │ │ └── ytt
73 │ │ ├── add_csi.yaml
74 │ │ ├── base-template.yaml
75 │ │ ├── csi-vsphere.lib.txt
76 │ │ ├── csi.lib.yaml
77 │ │ └── overlay.yaml
78 │ └── ytt
79 │ └── vsphere-overlay.yaml
80 ├── providers.md5sum
81 └── ytt
82 ├── 01_plans
83 │ ├── dev.yaml
84 │ ├── oidc.yaml
85 │ └── prod.yaml
86 ├── 02_addons
87 │ ├── cni
88 │ │ ├── add_cni.yaml
89 │ │ ├── antrea
90 │ │ │ ├── antrea.lib.yaml
91 │ │ │ └── antrea_overlay.lib.yaml
92 │ │ └── calico
93 │ │ ├── calico.lib.yaml
94 │ │ └── calico_overlay.lib.yaml
95 │ └── metadata
96 │ └── add_cluster_metadata.yaml
97 ├── 03_customizations
98 │ ├── add_sc.yaml
99 │ ├── filter.yaml
100 │ ├── registry_skip_tls_verify.yaml
101 │ └── remove_mhc.yaml
102 │
103 └── lib
104 ├── helpers.star
105 └── validate.star
We need to do a couple of things if we want to run arbitrary manifests after the cluster is created, similar to TKGI's ability to run an add-ons job through BOSH. First, depending on the infrastructure provider you're using (I'm using just simple vSphere here), we need to create a folder. This should be called 04_user_customizations and created at ~/.tkg/providers/ytt/. Inside this directory, we'll also create a file called cluster_addons.yaml which will contain not only the bootstrapping manifests, but also directives to the ytt tool on how and when to insert these YAML documents into our cluster. Create the file now, and paste in the following content.
1#@ load("@ytt:overlay", "overlay")
2#@ load("@ytt:data", "data")
3#@ load("@ytt:yaml", "yaml")
4
5#@ def cluster_addons():
6---
7apiVersion: v1
8kind: ServiceAccount
9metadata:
10 name: cluster-build
11 namespace: default
12---
13apiVersion: rbac.authorization.k8s.io/v1
14kind: ClusterRoleBinding
15metadata:
16 name: cluster-build
17subjects:
18- kind: ServiceAccount
19 name: cluster-build
20 namespace: default
21roleRef:
22 kind: ClusterRole
23 name: cluster-admin
24 apiGroup: rbac.authorization.k8s.io
25---
26apiVersion: v1
27kind: Secret
28metadata:
29 name: cluster-build
30 namespace: default
31type: Opaque
32data:
33 gitrepo: <base64_encoded_git_repo_URL_goes_here>
34---
35apiVersion: batch/v1
36kind: Job
37metadata:
38 name: cluster-build
39 namespace: default
40spec:
41 template:
42 spec:
43 initContainers:
44 - name: cluster-build-apply
45 image: chipzoller/k8s-addon-bootstrap:latest
46 command: ["/bin/sh"]
47 args: ["-c", "git clone $(GIT_REPO) manifests/ && kubectl apply -f manifests/ -R"]
48 env:
49 - name: GIT_REPO
50 valueFrom:
51 secretKeyRef:
52 name: cluster-build
53 key: gitrepo
54 containers:
55 - name: cluster-build-cleanup
56 image: chipzoller/k8s-addon-bootstrap:latest
57 command: ["kubectl", "delete", "secrets/cluster-build"]
58 restartPolicy: Never
59 serviceAccountName: cluster-build
60#@ end
61
62#@ if data.values.TKG_CLUSTER_ROLE == "workload" and data.values.CLUSTER_PLAN == "dev":
63
64---
65apiVersion: addons.cluster.x-k8s.io/v1alpha3
66kind: ClusterResourceSet
67metadata:
68 name: #@ "{}-cluster-addons".format(data.values.CLUSTER_NAME)
69 labels:
70 cluster.x-k8s.io/cluster-name: #@ data.values.CLUSTER_NAME
71spec:
72 strategy: "ApplyOnce"
73 clusterSelector:
74 matchLabels:
75 tkg.tanzu.vmware.com/cluster-name: #@ data.values.CLUSTER_NAME
76 resources:
77 - name: #@ "{}-cluster-addons".format(data.values.CLUSTER_NAME)
78 kind: ConfigMap
79---
80apiVersion: v1
81kind: ConfigMap
82metadata:
83 name: #@ "{}-cluster-addons".format(data.values.CLUSTER_NAME)
84data:
85 value: #@ yaml.encode(cluster_addons())
86
87#@ end
Without going into too much detail, here's what's essentially happening. When creating a cluster with the tkg CLI, you specify a plan. By default, TKG ships with plans named dev and prod. Each plan is a template which gathers multiple files from different places, each with their own directives, that then results in a fully-defined cluster. Using the ytt tool and its powerful templating and overlaying abilities, pieces from multiple files are inserted at the appropriate places and at the correct times. What we're doing here is making use of a primitive in the Cluster API project known as a ClusterResourceSet. This resource type defines additional user-defined add-ons that should be deployed once a cluster is built. It's almost identical in purpose to the function of BOSH add-ons in TKGI.
The top of the manifest creates a definition called cluster_addons which will be referenced later similar to a variable. The bottom portion actually defines the Kubernetes manifests for a ClusterResourceSet and its source of data using a ConfigMap in this case, which has as its data the cluster_addons definition created above. The if statement sandwiched in between them instructs ytt and thereby the tkg CLI when to apply these manifests. Those two conditions are when the K8s cluster is a workload cluster (i.e., not a management cluster) and what plan is being called. In this case, the only plan which will receive these bootstrapper add-ons is the dev plan.
Save the manifest ensuring the YAML indentation is correct and don't forget to replace the gitrepo key in the Secret resource with your base64-encoded URL to your own Git repo. I've pre-populated the images in this manifest with the image I built and pushed to Docker Hub, but for production use you should strongly consider building your own from upstream. If/when you do build and push to your own registry, one of my previous articles on how to securely trust an internal registry should help tremendously, and it complements this workflow nicely. In fact, I'm using both here in this article.
Once you've got everything situated, create a new cluster from your dev plan.
1$ tkg create cluster pickles02 --plan=dev --vsphere-controlplane-endpoint-ip=192.168.1.223
2Logs of the command execution can also be found at: /tmp/tkg-20201107T091123523747106.log
3Validating configuration...
4Creating workload cluster 'pickles02'...
5Waiting for cluster to be initialized...
6Waiting for cluster nodes to be available...
7Waiting for addons installation...
8
9Workload cluster 'pickles02' created
Notice the highlighted line in the code block above. If the new add-ons file was taken, they should be installed at the final step here (along with default add-ons such as CNI). If there was a syntax error somewhere in your file, ytt will fail to parse it and you'll see an error message of some sort shortly after you issue the tkg create cluster command.
Check your cluster and see if your add-on bootstrapper ran.
1$ k get po
2NAME READY STATUS RESTARTS AGE
3cluster-build-z9kzz 0/1 Completed 0 18m
Here we see the Job for cluster-build completed. You should now be able to see the results of all the manifests you had defined in your Git repo.
One last thing. Why don't we create a new plan so we can test these updates safely and not impact the default dev or prod plans. The process is fairly simple: In your infrastructure-vsphere folder, go into the v0.7.1 subfolder (or whatever the provider version happens to be when you read this) and copy the cluster-template-definition-dev.yaml file to a new one and call it cluster-template-definition-test.yaml. The tkg CLI will parse anything after the cluster-template-definition- string as the name of a plan, so in this case our plan is simply called test. Once the new file exists, it doesn't need to be modified.
Next, up in our cluster_addons.yaml file we created earlier, find the if statement line and change dev to test like so.
1#@ if data.values.TKG_CLUSTER_ROLE == "workload" and data.values.CLUSTER_PLAN == "test":
Alternatively, let's say you wanted to enable these cluster bootstrap add-ons for multiple plans, you can add multiple the following way.
1#@ if data.values.TKG_CLUSTER_ROLE == "workload" and data.values.CLUSTER_PLAN in ["test", "dev"]:
Depending on how you modified this line, your bootstrap add-ons should either get deployed when rolling out a test plan, or for both the test and dev plans.
And I think that's it. I hope this has been a useful article and two-part series in showing you one way you can use across different installments in the Tanzu portfolio to easily install your custom add-ons to Kubernetes clusters. Using this as a base, you can extend the capabilities many different ways, all easily by checking in new manifests to your Git repo. As always, feel free to hit me up on Twitter or LinkedIn with your feedback.