Setup Kubernetes Cluster Ubuntu 18.04 Pakai Kubeadm
Posted on
Ini adalah catatan bagaimana aku setup kubernetes cluster pakai kubeadm. Aku setup pakai VM (KVM) 1 master dan 1 node worker. Aku pakai vm ubuntu 18.04. Selama ini aku setup kubernetes cluster di debian 10. Sekarang belajar setup di ubuntu 18.04. Meskipun ubuntu berbasis debian, tapi ada beberapa yang berbeda ketika setup kubernetes cluster, yg paling kentara di Debian 10 adalah iptables-legacy.
Rangkuman singkatnya begini:
- install docker dan kubeadm di master
- setup kubeadm di master
- install docker dan kubeadm di node
- join node ke cluster pakai kubeadm
- Cluster selesai, sisanya pasang aplikasi sesua kebutuhan
Log install kubernetes di Ubuntu 18.04
A. Setup Master
- Node IP addr: 192.168.100.198
- Node Hostname: master
- OS: Ubuntu 18.04
- RAM: 2 GB
- CPU: 2 core
A.1 Install docker
Ikuti panduan https://docs.docker.com/engine/install/ubuntu/
apt-get update
apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io
A.2 Ganti driver docker ke systemd, awalnya cgroupfs
bash
Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF
bash
mkdir -p /etc/systemd/system/docker.service.d
bash
Restart Docker
systemctl daemon-reload systemctl restart docker systemctl enable docker
## A.3 Install kubeadm
Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
```bash
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
A.4 Init cluster pakai kubeadm
Matikan swap
bash swapoff -a
Matikan swap pas reboot pakai crontab
@reboot swapoff -a
Disini mau pakai flannel untuk networknya, oleh karena itu pakai parameter `--pod-network-cidr=10.244.0.0/16`
bash
kubeadm init --pod-network-cidr=10.244.0.0/16
bash root@master:/home/sumar# kubectl get po -o wide -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-66bff467f8-jghgf 0/1 Pending 0 4m22s Setelah install flannel, pod coredns jadi jalan bash root@master:/home/sumar# kubectl get po -o wide -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-66bff467f8-jghgf 1/1 Running 0 8m19s 10.244.0.3 master apt-get update apt-get install curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - add-apt-repository apt-get update apt-get install docker-ce docker-ce-cli containerd.io bash mkdir -p /etc/systemd/system/docker.service.d Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ bash kubeadm join 192.168.100.198:6443 --token 98vdpp.r7tziuipa5ebwpg8 Dari data diatas, semua pod running. Aku asumsikan semua beres. Setelah cluster berjalan, aku lanjutkan install aplikasi di atas cluster kubernetes. Kita kembali ke master, dan install-install dari master. Helm v3 ini akan membatu banget untuk install-install aplikasi diatas cluster kubernetes. bash curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 chmod 700 get_helm.sh ./get_helm.sh helm repo add stable https://kubernetes-charts.storage.googleapis.com/ helm repo update Bikin configmap untuk metallb, silakan merujuk ke https://metallb.universe.tf/configuration/#layer-2-configuration Yang perlu diperhatikan adalah bash kubectl apply -f metallb-configmap.yaml Ikuti https://kubernetes.github.io/ingress-nginx/deploy/#using-helm Ingress controller ini membantu untuk expose service ke luar cluster, misalnya expose web service via domain. Nginx Ingress Controller hanyalah satu dari sekian banyak Ingress Controller. Selain nginx ada Traefik, HAProxy, dll. Aku sendiri selama ini pakai Traefik v1.7.24 di production. Dah kali ini aku akan pakai Nginx bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm install my-release ingress-nginx/ingress-nginx Berkat bantuan metallb, EXTERNAL-IP dari Untuk memastikan ingress controllernya berfungsi, aku akan test dari luar cluster yakni dari PC. bash ➜ ~ curl 192.168.100.30 Kita coba deploy aplikasi Isi dari bash kubectl apply -f whoareyou.yaml Kita test hasil deploynya. Anggaplah bash ➜ ~ curl -H "Host: whoareyou.kvm.local" 192.168.100.30 Hostname: whoareyou-deployment-74bff9c468-mlnjd IP: 127.0.0.1 IP: 10.244.1.7 RemoteAddr: 10.244.1.4:33058 GET / HTTP/1.1 Host: whoareyou.kvm.local User-Agent: curl/7.70.0 Accept: / X-Forwarded-For: 10.244.1.1 X-Forwarded-Host: whoareyou.kvm.local X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Real-Ip: 10.244.1.1 X-Request-Id: bce405ea511051351247e01746104180 X-Scheme: http Outputnya adalah 404, karena ingress rule untuk domain sembarangan.com tidak ada. Kesimpulannya adalah, kubeadm ini sangat memudahkan kita untuk setup kubernetes cluster. Banyak magic-nya. Aku senang dengan adanya kubeadm
pod coredns status pending krn belum install flannel,
bash
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# B. Setup Nodesatu
- Node IP addr: 192.168.100.199
- Node Hostname: nodesatu
- OS: Ubuntu 18.04
- RAM: 2GB
- CPU: 2 core
## B.1 Install docker
Ikuti panduan https://docs.docker.com/engine/install/ubuntu/
apt-transport-https
ca-certificates
curl
gnupg-agent
software-properties-common
"deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"
## B.2 Ganti driver docker ke systemd, awalnya cgroupfs
bash
# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
bash
# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
B.3 Install kubeadm
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
B.4 Join nodesatu ke cluster
--discovery-token-ca-cert-hash sha256:d6d44f1588470756cd52f70b5576aa7ddd6b5b9a57559b025dbfa21ecee2c0eb
Langkah A dan B adalah langkah untuk setup cluster. Setelah setup master selesai dan setup nodesatu selesai, kita verifikasi:
bash
root@master:~# kubectl cluster-info
Kubernetes master is running at https://192.168.100.198:6443
KubeDNS is running at https://192.168.100.198:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@master:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master Ready master 74m v1.18.5 192.168.100.198 <none> Ubuntu 18.04.4 LTS 4.15.0-108-generic docker://19.3.12
nodesatu Ready <none> 41m v1.18.5 192.168.100.199 <none> Ubuntu 18.04.4 LTS 4.15.0-108-generic docker://19.3.12
root@master:~# kubectl -n kube-system get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-66bff467f8-jghgf 1/1 Running 0 77m 10.244.0.3 master <none> <none>
coredns-66bff467f8-zkcdd 1/1 Running 0 77m 10.244.0.2 master <none> <none>
etcd-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-apiserver-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-controller-manager-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-flannel-ds-amd64-rb4wc 1/1 Running 0 69m 192.168.100.198 master <none> <none>
kube-flannel-ds-amd64-rmdw7 1/1 Running 1 44m 192.168.100.199 nodesatu <none> <none>
kube-proxy-hgkn9 1/1 Running 0 44m 192.168.100.199 nodesatu <none> <none>
kube-proxy-lc5ps 1/1 Running 0 77m 192.168.100.198 master <none> <none>
kube-scheduler-master 1/1 Running 0 77m 192.168.100.198 master <none> <none>
C. Pasca Setup Cluster
C.1 Install Helm3
## C.2 Install metallb
Ikuti https://metallb.universe.tf/installation/
Krn cluster yg kubuat di vm local, dan gak punya dedicated LB, maka aku butuh metallb untuk assign IP ke service bertipe LoadBalancer
bash
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
nano metallb-configmap.yaml
# isinya sbb:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.100.30-192.168.100.40
192.168.100.30-192.168.100.40, IP ini aku kasih range dari router, ya krn ini VM di local. Kalau di cloud/datacenter range ip tersebut harus ip public supaya service bisa dijangkau dari luar.
Verifikasi hasil install metallb
bash
root@master:~# kubectl -n metallb-system get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
controller-57f648cb96-b5llj 1/1 Running 0 33m 10.244.1.2 nodesatu <none> <none>
speaker-csgx5 1/1 Running 0 33m 192.168.100.199 nodesatu <none> <none>
speaker-mrv7x 1/1 Running 0 33m 192.168.100.198 master <none> <none>
C.3 Install Nginx Ingress Controller
Perintah diatas akan install nginx di namespace default, kita check:
bash
@master:~# kubectl -n default get po,svc -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs 1/1 Running 0 34m 10.244.1.4 nodesatu <none> <none>
NAME TYPE CLUSTER-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 85m <none>
service/my-release-ingress-nginx-controller LoadBalancer 10.110.208.65 192.168.100.30 80:30509/TCP,443:30954/TCP 34m app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
service/my-release-ingress-nginx-controller-admission ClusterIP 10.104.254.133 <none> 443/TCP 34m app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
service/my-release-ingress-nginx-controller tidak pending. Dalam hal ini dia mendapatkan IP 192.168.100.30. IP tsb kita asumsikan IP public yg mana akan dimasukan ke DNS A Record404 Not Found
whoareyouwhoareyou.yamlapiVersion: apps/v1
kind: Deployment
metadata:
name: whoareyou-deployment
spec:
replicas: 2
selector:
matchLabels:
app: whoareyou
template:
metadata:
labels:
app: whoareyou
spec:
containers:
- name: whoareyou-container
image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
name: whoareyou-service
spec:
ports:
- name: http
targetPort: 80
port: 80
selector:
app: whoareyou
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: whoareyou-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: whoareyou.kvm.local
http:
paths:
- path: /
backend:
serviceName: whoareyou-service
servicePort: http
Verifikasi hasil deploy:
bash
root@master:~# kubectl -n default get po,svc,ingress
NAME READY STATUS RESTARTS AGE
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs 1/1 Running 0 45m
pod/whoareyou-deployment-74bff9c468-mlnjd 1/1 Running 0 110s
pod/whoareyou-deployment-74bff9c468-p5bkp 1/1 Running 0 110s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 97m
service/my-release-ingress-nginx-controller LoadBalancer 10.110.208.65 192.168.100.30 80:30509/TCP,443:30954/TCP 45m
service/my-release-ingress-nginx-controller-admission ClusterIP 10.104.254.133 <none> 443/TCP 45m
service/whoareyou-service ClusterIP 10.108.79.85 <none> 80/TCP 110s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.extensions/whoareyou-ingress <none> whoareyou.kvm.local 192.168.100.30 80 110s
whoareyou.kvm.local nama domain yg kita set dan 192.168.100.30 adalah public ip. Krn ini di local, cara testnya bisa kita masukin ke /etc/hosts komputer di luar cluster. Atau pakai curl dengan param header host. Disini aku test pakai curl
Sip, sukses. Yg keluar adalah output dari pod whoareyou.
Bagaimana kalau kita pakai nama domain yg lain?
bash
➜ ~ curl -H "Host: sembarangan.com" 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>