One of my clients insists on using bare metal Windows Server in their infra. Meanwhile, I insists on using Kubernetes to deploy the stack to make it reliable and less maintenance burden. This weird situation is why I have to set up Kubernetes on top of Windows Server to run my workload
By the time I wrote this article, we have Windows 10 Desktop Pro. Installing Kubernetes stack with Linux worker nodes is trivial for development purposes. There are many ways to do that with everything preconfigured. Here are some of them:
- Install Docker for Desktop for WSL 2 then activate Kubernetes support (it will use minikube)
- Install Ubuntu WLS 2 then install Microk8s
- Install Ubuntu Multipass, then create Ubuntu Hyper-V (via Multipass) then install Microk8s
- Install Ubuntu Microk8s from Ubuntu’s provided installer on Windows (it will use Multipass anyway)
- Install Rancher Desktop (somewhat experimental at the moment)
- Install minikube directly and choose your driver (can be KVM2, Docker, Virtual Box, etc)
Now for some reasons and the other, above approach don’t work nicely if we are talking about self-hosted production deployment using Windows Server on prem.
The cause is:
- Windows Server Standard Edition (2019) don’t have WSL2 support
- Ubuntu Multipass works for the recent Windows 10 Pro Hyper-V manager but not Windows Server Hyper-V manager
- Minikube is not production ready and to my experience contains many wierd bug unsuitable for production environment
- Rancher Desktop needs WSL 2
- Docker for Desktop and Virtual Box license prevent us to use this for production deployment (I think)
There is also Docker Enterprise Edition, but we won’t use that because it only deals with Windows container. We need Linux containers.
Given these constraint, then my chosen approach would be:
- Install Ubuntu manually using Hyper-V manager bundled with Windows Server
- Install K3s on top of Ubuntu (for some reason Microk8s have poor performance)
- Setup extra things such as service management, port-forward, and networking.
Hyper-V is basically a Windows Hypervisor. You can use Hyper-V to share your bare metal computer resources to be used by your VM. The VM can be a Linux VM. Windows Server Standard Edition have this feature but you need to enable it. Just follow this instruction
The important settings from the role installation:
- In the step, make sure you do not select your physical network device, so that Windows will configure a Default Switch for the VM to use.
Hyper-V > Virtual Switches
- Restart after installing
Once you enable Hyper-V, you can open Hyper-V manager. Just search the program from the search bar.
From the Hyper-V manager window, choose your server then click menu
Action > New > Virtual Machine
- Choose Generation 2
- Allocate a decent chunk of memory because we are going to use it as Kubernetes nodes.
- For the networking, choose because we want the VM to have internet access. If for some reason the
Default Switchis missing. Then we need to set up NAT later. I will talk about it later.
- Download your Ubuntu server ISO and select the ISO for the media installation option.
Once you created your VM profile, you need to tweak more settings. Settings page is available after you click your VM. The options is in your right panel.
- Disable Secure Boot
- Add more resources if necessary, like CPU and RAM
- If you have problem with missing, then go to NAT networking section first, then go back here.
- In the Network Adapter section, choose or
Default Switchvirtual adapter that you set up with NAT in step 3.
Then start your VM and proceed with a standard Ubuntu installation. In my current case, I installed minimal Ubuntu Server LTS 20.04 version. I had to setup networking and user password within the installation step in order to get access to the machine.
Now, assuming you already have either
If Windows create
If Windows don’t have
After configuring the network, I recommend testing these things to check connection:
- Ping from inside the VM to the host with the or
Default Switchstatic IP as the target
- Ping from the host to the VM using VM eth0 static IP as the target
- Ping from the VM to your network DNS (or public DNS)
- Check if any website in the internet is reachable from inside the VM.
Skip this step if you already have
Windows Hyper-V manager gives you the
From the Server Manager interface, Add Role, and choose
Remote and Routing
Once the role is added, you will have new tools called
Routing and Remote
Key point to look out for:
- In Hyper-V manager, there is a Virtual Switch Manager action. Click that and create a new Virtual Switch with type (not External, not Private). Give it a name
- In the Routing and Remote Access manager, right click on your server and choose . Choose NAT.
Configure and Enable Routing and Remote Access
- In the interface selection, there will be at least two interface. Your physical interface connected with internet, in my case it is called Ethernet,
internet public IPv4 address. Then the second one is our virtual interface . The selection asked you to choose which interface is connected with public internet access, so choose that interface. Then let the process finished.
Note that a misconfiguration will likely cause you to not be able to access your Windows Server remotely. In my case, I set it up in Hetzner and whenever I misconfigured my network, I had to clean install the server (I tried different things before settling on this approach). So, make sure you choose the correct interface.
Next, we are going to configure the network your VM will use, which is the
- In your host machine (the Windows Server), Open Network and Sharing Center. Choose virtual interface, it is likely to be named as with type Hyper-V Virtual Ethernet Adapter.
- Right click adapter > Properties > IPv4 > Properties. Configure your subnet. I am using 192.168.100.1 as IP address with netmask 255.255.255.0. This is because Kubernetes usually uses 10.x.x.x IP adresses. I just want these two subnets can be easily recognized at a glance. I mean, if I saw 192.x.x.x somewhere in the guest VM, that means it is the Hyper-V network interface and not K8s Cluster Network interface. I can easily recognized it.
- Also setup DNS if you use non-default one.
Now once you completed this step, go back to the previous step of Ubuntu installation.
Depending on which k8s distro you choose, install it yourself. If you use microk8s in Ubuntu server, then usually the option is presented in the installation steps.
In my case, I’m using k3s. Simply download the binary and run it.
curl -sfL https://get.k3s.io | sh -
This will add single node k3s server.
You may want to guard the controlplane to be accessible only by you, from the host. Here’s the plan: Enable SSH, but disable password authentication. Put strong password for the normal user and disallow sudo for this user. Instead of accessing the VM via Hyper-V manager, I recommend to access the VM via SSH as root to reduce ways of access, because after all the controlplane is only used as controlplane. This is how I do it. Each step has indicator on where the task should be executed.
host Download Visual Studio Code and Install
host In Visual Studio Code, enable Remote Development Extension Pack from Microsoft
guest Enable SSH with password authentication (temporarily): editas root and add
/etc/ssh/sshd_configin the last line. Restart ssh using
sudo systemctl restart sshd
host Connect to the VM using Visual Studio Code “connect via SSH” as described here
guest Once you are inside the VM. Open VSCode terminal window. Generate new ssh key pair and use it as credentials to access the VM as root.
ssh-keygen sudo mkdir -p /root/.ssh cat ~/.ssh/id_rsa.pub | sudo tee -a /root/.ssh/authorized_keys # Open the private key in VSCode tab code ~/.ssh/id_rsa
From the newly opened window offile, copy the content of the file into your host/Windows user directory. This is your SSH private key. Keep it safe and delete it from the guest VM.
guest Configure your SSHD config (in) to disable password authentication by setting
/etc/ssh/sshd_configfor the line we set in previous step. Restart SSHD:
sudo systemctl restart sshd
host Redo connection to the VM using Visual Studio Code, this time, make sure that it doesn’t ask for password, and you log in as.
guest From your VSCode window that is remote connecting to the guest VM, install Kubernetes extension made by Microsoft. This will allow you to do a quick K8s operation via VSCode GUI.
By now, whenever you want to access the guest VM (let’s call this controlplane), use VSCode from the host machine. A more sophisticated method (if you are familiar) is to use Visual Studio Code in your local computer to access the controlplane via your Windows Server host as the bastion host. This requires you to do SSH tunnel/forward in your config file.
With k8s operations, you want to have at least these ports exposed:
- Port 22 of the controlplane to Windows Server public IP for SSH access to controlplane (or for more security)
- Port 6443 (depending on the distro) of the controlplane to Windows Server public IP 6443 for Kubernetes API Server access
- Port 80/443 of the Load Balancer VM to be exposed to the port 80/443 of the Windows Server public IP so that we can expose HTTP/S service
It depends if your Kubernetes distro is using another layer of virtualization or not. With k3s, the controlplane is accessible in the guest VM. So the controlplane internal IP address is just the VM internal IP. You can just get it via
With k3s, since the port is exposed in the VM. We can refer to the
The network topology we uses is that the Windows Server host has the interface that has public IP (internet addressable). The VM uses network interface called the
We are forwarding the port using
For each port we want to forward, do:
netsh interface portproxy add v4tov4 listenport=<port in host> connectport=<port in VM> connectaddress=<VM_IP> listenaddress=<host IP>
For some reason, the forwarding rules doesn’t work after machine restart. Even though, the rules appears when I do
netsh interface portproxy show all
# in a file called `port-proxy.ps1` netsh interface portproxy reset # For http netsh interface portproxy add v4tov4 listenport=80 connectport=80 connectaddres=192.168.100.2 listenaddress=192.168.1.2 # For https netsh interface portproxy add v4tov4 listenport=443 connectport=443 connectaddres=192.168.100.2 listenaddress=192.168.1.2 # For k8s controlplane netsh interface portproxy add v4tov4 listenport=6443 connectport=6443 connectaddres=192.168.100.2 listenaddress=192.168.1.2 # For SSH netsh interface portproxy add v4tov4 listenport=22 connectport=22 connectaddres=192.168.100.2 listenaddress=192.168.1.2 netsh interface show all
You can check by doing SSH or access the HTTP/S endpoint using the public host IP interface.
For the port to be able to be accessed from outside the Windows Server host, you need to declare inbound rules in the Windows Firewall settings. Just allow port 80,443,6443,22.
You want the forwarding rule to be automatically applied if the server restarts. Create a scheduled task and schedule it for every after system startup to execute the previous
This depends on your k8s distro.
In the case of K3s, the certificate was setup with the default SAN: 127.0.0.1 and internal IP address, which in this case the
# Save our current tls config cd /var/lib/rancher/k3s/server mkdir -p .tls-backup k3s kubectl -n kube-system get secret k3s-serving -o yaml > .tls-backup/k3s-serving.yaml mv tls .tls-backup/ # Delete current certificate k3s kubectl -n kube-system delete secret k3s-serving rm -rf tls # restart k3s systemctl restart k3s
For production instance, we want our k8s cluster to be up immediately when the machine is up.
Check your Hyper-V VM Settings. Under the
automatically start if it was running when the service stopped