I set myself a new challenge for the New Year: Stand Up a Kubernetes cluster of my own. At home. I have reasons (long term costs being [potentially] lower than cloud-hosted alternatives being just one of them) but honestly since Kubernetes is a dominant presence in my day-to-day work these days, the main reason was that I was curious to learn more about it, and this seemed like a good way to do that.
Multi-Tasking
First of all, I should explain that this was a project that I was interleaving with another, higher priority project that also kicked-off in the New Year.
Her name is Zoya. She arrived around mid-day on 3rd January and is absolutely gorgeous. 🙂
The Kubernetes side of things took me 3 weeks [elapsed] which included an initial fully manual install, time spent exploring automation options and then the final green-field deployment, together with a couple of mis-steps along the way.
I likely would have completed my Kubernetes challenge much sooner if it hadn’t been conducted only while Zoya slept (and had I not been quite so sleep-deprived!).
This is not a complaint, merely an observation. 🙂
With my automation solutions in place, I’m confident I could now rebuild my cluster, from scratch, in a couple of hours at most (and could get that down to minutes if I invested in some further automation).
First: What Is Kubernetes?
If you already know the answer, feel free to skip ahead to the Project Hardware. For those still reading at this point…
The pithy answer is that Kubernetes is a container orchestration solution, which may not help much if you don’t already work with such things.
A more detailed answer, in very much simplified terms, is that Kubernetes provides an infrastructure into which containerised services can be deployed declaratively. You provide a description of the services you wish to run and submit that description to Kubernetes. Kubernetes constantly monitors the current state of the system compared to the desired (described) state and makes whatever changes are required, adding or removing containers (and other objects) as required. That includes replacing containers if/when they crash or if the described state changes.
For example, if you deploy a service with 2 containers and then update the description of the service such that it now requires an additional container and a change to one of the existing services, you do not “uninstall” the old service and re-install the new one, you simply update the description. Kubernetes takes care of making the required changes to bring the system state into line with the new state as described.
A key consideration with services deployed into Kubernetes is what is often referred to as the “cattle, not pets” mindset that it employs in respect of the servers on which the services run (and indeed the services themselves).
Pets vs Cattle
In a traditional/legacy model where an application is deployed onto a specific server, those applications/services and the servers running them are treated like Pets, demanding loving care and attention such as OS updates, hardware replacements/upgrades etc. If a problem develops on a server or an application, a team typically springs into action to resolve that problem and bring the server and the services on it back into an operational state. Such servers have names (even if they are more like catalogue numbers in an enterprise setting) and we care very much about the state and the health of those servers. If a server fails, someone has to carefully redeploy all the services that were running on that server onto some other server or perform fail-over to a redundant backup server, following a carefully crafted – but often not well-rehearsed – fail-over runbook.
Servers in Kubernetes are more like Cattle. The servers provide a pool of resources. If a problem develops with a machine in the pool it is removed, usually to be replaced by another. When the server is removed, Kubernetes will notice that the services on it are no longer running (causing a discrepancy with the described state of the system) and will automatically take action to restore that state, i.e. by starting new instances of those services elsewhere in the pool. The servers may still have names for inventory management, but we don’t care about them as individual servers.
We also don’t know (nor care) which server is running any particular service or services. All we care about is that our pool of servers provides sufficient resources for the totality of services running in that pool, which we ensure by monitoring the load on the servers, adding additional capacity by adding more servers as required (or indeed removing excess capacity) and letting Kubernetes take care of scheduling resources across the revised resource pool.
It’s not quite server-less – though honestly there really is no such thing, only degrees to which you care about individual servers, and by that metric Kubernetes comes pretty close, especially in an enterprise setting where even an on-premise Kubernetes infrastructure is typically provided as a service to application developers.
Kubernetes is an incredibly deep rabbit hole if you want to go down it, and there’s a lot to learn. But I think this covers the basics and should be enough to understand and appreciate what I am trying to achieve with the project I am describing here.
Kubernetes Terminology and “The Project”
Time for some Kubernetes terminology:
| Node | A machine (physical or virtual) with Kubernetes installed |
| Cluster | A collection of Nodes. A Cluster requires at least one Master Node and one or more Worker Nodes (though a single node can be both Master and Worker). |
| Pod | A container for running containerised services on a Node. |
| Kubelet | Software that runs on a node, facilitating communications with and between other nodes. |
| Kubeadm | Administration command-line tool to perform tasks on a node such as initialising a cluster or joining a node to an existing cluster. |
| Kubectl | Kubernetes command-line tool, used to run commands on a workstation to work with a Kubernetes cluster on the network, such as adding, removing or querying the state of services and other objects in a cluster etc. |
| Bare Metal | Kubernetes installations provisioned on your own hardware, as opposed to using managed services such as Azure Kubernetes Service (AKS) or Google Kubernetes Engine (GKE). |
The goal of my project was to establish a Bare Metal Kubernetes Cluster at home that I could deploy services into for dev/test purposes, more-or-less identically to how those services would then be deployed into production using cloud-hosted Kubernetes.
Why go to all the trouble?
- To enable me to build and deploy GoLang micro-services at home and to experiment with other technologies relevant in the micro-services/cloud-native field
- To learn and to challenge myself
Also: ‘k8s’
One other term that you will quickly run into in any discussion of Kubernetes is k8s.
This is a contractionym (I think I just made that term up), similar to i18n and l10n for internationalisation and localisation, respectively. i.e. a word beginning with ‘k’ and ending in ‘s’, with 8 other letters in between: kubernetes.
You may hear “k8s” pronounced kay-eights or simply kates.
The Project Hardware
Although mainly a Mac user these days, I’ve also fallen in love with Intel NUC machines. I run a few of these for various purposes including hosting build agents for my Azure DevOps pipelines as well as more mundane tasks such as hosting a PLEX Media Server.
But it’s the form factor that appeals, more than the specific hardware, and there are other options in this space that can provide better cost/performance propositions. One such, that I chose to host my Kubernetes Cluster is the Asus PN50-E1 Mini PC:
Specifically, I opted for a Ryzen 7 model with an 8-core (16-thread) CPU barebones system to which I added 64GB of DDR4 RAM, 500GB SSD system drive and 1TB SSD for storage.
My goal was a multi-node cluster but rather than multiple physical machines I planned to run multiple VM’s on this single box. I know this is possible with Linux but as I am still more familiar with Windows chose to put Windows 10 Pro on the system, using Hyper-V to provide the VM’s.
Kubernetes can manage Windows containers, but I intend using this cluster primarily for GoLang microservices and so my goal was a cluster of Linux machines. The Windows host OS plays no part in the Kubernetes system itself, other than providing the hypervisor under which the VM’s are running.
The VM’s themselves will all be running Ubuntu Server 20.04.
Although not a major consideration, I run Ubiquiti UniFi network gear. This provides me with some fairly sophisticated network management facilities which I intended to use (more on that later). Specifically, I intended my Kubernetes nodes to run in their own domain, on a dedicated VLAN with static IP’s.
This will be a factor in one of the “missteps” I stumbled into.
Sizing The Cluster and Nodes
After researching recommended cluster and node specs for sizing my cluster and the nodes within it, and finding a whole lot of not very helpful ‘it depends’ advice, I resorted to a careful and considered thumb-suck, settling on 4 nodes for my cluster: 1 Master Node and 3 additional Worker Nodes.
The 4 VM’s would be specified thus:
| Virtual CPUs | 4 |
| RAM | 8GB (fixed) |
| HDD (dynamic) | Dynamic capacity to max. 120GB |
| Network | VLAN ID 2 / Static MAC Address |
| Other | Secure Boot Disabled |
Allocating 8GB of RAM to each VM would leave 32GB for the host OS whilst 120GB of storage would consume only half the capacity of the SSD set aside for the purpose (and then only once each dynamic disk was full). This provides me with some headroom if I need to increase the size of my VM’s or experiment with adding further VM nodes to my cluster.
For each VM I decided to configure 4 virtual CPU’s, resulting in the 8-core physical CPU servicing 16 ‘virtual cores’. This might seem counter-intuitive at first, but just as physical core count doesn’t dictate the number of threads or processes that can run ‘simultaneously’ on a CPU, neither does physical core count dictate the number of virtual cores that can be hosted simultaneously (at least not on a 1:1 basis). Over time, monitoring would tell me whether I had this balance right.
With the rather non-scientific nature of the sizing specification process, I was aware that I may need to resize (and rebuild) my cluster at some point in the future as I came to understand my needs better, so identifying opportunities to automate the process were front of mind from the outset.
I would also need some means of monitoring resource utilization within the cluster so that I could right-size things in the future.
Network Preparation
The first step was to enable Hyper-V in the Windows OS and configure it with an External network switch for my VM’s to use. This would enable them to communicate with each other but also with the wider internet (which they would need in order to pull container images from container registries, among other things).
Kubernetes runs its own networking stack within a cluster by which the nodes communicate with each other but the nodes themselves would also appear on my network and I had some ideas about how I wanted that to appear.
Static IP’s were required for reasons that will become apparent later and whilst I could have simply configured 4 static IP’s on my existing 192.168.1.1/24 network, I wanted to make my Kubernetes nodes distinct and to isolate traffic between the nodes from the rest of the network.
For the Kubernetes nodes, I opted to define a new VLAN network in my UniFi Security Gateway Pro, using the 10.10.0.1/24 subnet with its own VLAN ID of 2.
I then chose 4 consecutive MAC addresses from the range that Hyper-V would otherwise have used, as indicated in the Global Network Settings under the Hyper-V Switch Manager.
Like most routers, the UniFi Security Gateway Pro will register clients automatically which can then be assigned static IP’s etc. It also allows clients to be pre-registered by MAC address, so I added 4 new clients to my network with static IP’s allocated to the MAC addresses, with hostnames of master, worker1, worker2 and worker3.
VLAN’s and static IP configuration in this manner are all very straightforward and something I had already become familiar with when establishing separate VLAN’s for our household IoT devices and gaming consoles, for example, allowing UPnP to be enabled for gaming consoles without the potential security issue of having UPnP enabled across the entire network.
For the Kubernetes VLAN I also configured a domain of “k8s” so that the nodes could be addressed via a domain qualified hostname such as master.k8s, worker1.k8s, etc
The UniFi networking eco-system has provided me with great learning opportunities in the networking space and I would discover that there was yet more to learn in this area which we’ll get to later.
I called this switch “Kubernetes Switch“.
Next Steps…
With all of this decided, and the network suitably prepared to receive the 4 VM’s that would form my Kubernetes cluster, I was ready to set about establishing the VM’s themselves.
This would prove to be the single biggest job in the whole exercise and the subject of the next post in this series.