sumarsono.com
Take it with a grain of salt


Setup Kubernetes Cluster Ubuntu 18.04 Pakai Kubeadm

Posted on

Ini adalah catatan bagaimana aku setup kubernetes cluster pakai kubeadm. Aku setup pakai VM (KVM) 1 master dan 1 node worker. Aku pakai vm ubuntu 18.04. Selama ini aku setup kubernetes cluster di debian 10. Sekarang belajar setup di ubuntu 18.04. Meskipun ubuntu berbasis debian, tapi ada beberapa yang berbeda ketika setup kubernetes cluster, yg paling kentara di Debian 10 adalah iptables-legacy.

Rangkuman singkatnya begini:

Log install kubernetes di Ubuntu 18.04

A. Setup Master

A.1 Install docker

Ikuti panduan https://docs.docker.com/engine/install/ubuntu/

apt-get update

apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
 apt-get update
 apt-get install docker-ce docker-ce-cli containerd.io

A.2 Ganti driver docker ke systemd, awalnya cgroupfs

bash

Set up the Docker daemon

cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF


bash
mkdir -p /etc/systemd/system/docker.service.d

bash

Restart Docker

systemctl daemon-reload systemctl restart docker systemctl enable docker


## A.3 Install kubeadm

Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

```bash
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

A.4 Init cluster pakai kubeadm

Matikan swap

bash swapoff -a


Matikan swap pas reboot pakai crontab

@reboot swapoff -a




Disini mau pakai flannel untuk networknya, oleh karena itu pakai parameter `--pod-network-cidr=10.244.0.0/16`

bash
kubeadm init --pod-network-cidr=10.244.0.0/16

bash root@master:/home/sumar# kubectl get po -o wide -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-66bff467f8-jghgf 0/1 Pending 0 4m22s kube-system coredns-66bff467f8-zkcdd 0/1 Pending 0 4m22s kube-system etcd-master 1/1 Running 0 4m32s 192.168.100.198 master kube-system kube-apiserver-master 1/1 Running 0 4m32s 192.168.100.198 master kube-system kube-controller-manager-master 1/1 Running 0 4m32s 192.168.100.198 master kube-system kube-proxy-lc5ps 1/1 Running 0 4m22s 192.168.100.198 master kube-system kube-scheduler-master 1/1 Running 0 4m32s 192.168.100.198 master root@master:/home/sumar#


pod coredns status pending krn belum install flannel,

bash
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Setelah install flannel, pod coredns jadi jalan

bash root@master:/home/sumar# kubectl get po -o wide -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-66bff467f8-jghgf 1/1 Running 0 8m19s 10.244.0.3 master kube-system coredns-66bff467f8-zkcdd 1/1 Running 0 8m19s 10.244.0.2 master kube-system etcd-master 1/1 Running 0 8m29s 192.168.100.198 master kube-system kube-apiserver-master 1/1 Running 0 8m29s 192.168.100.198 master kube-system kube-controller-manager-master 1/1 Running 0 8m29s 192.168.100.198 master kube-system kube-flannel-ds-amd64-rb4wc 1/1 Running 0 46s 192.168.100.198 master kube-system kube-proxy-lc5ps 1/1 Running 0 8m19s 192.168.100.198 master kube-system kube-scheduler-master 1/1 Running 0 8m29s 192.168.100.198 master


# B. Setup Nodesatu

- Node IP addr: 192.168.100.199
- Node Hostname: nodesatu
- OS: Ubuntu 18.04
- RAM: 2GB
- CPU: 2 core

## B.1 Install docker

Ikuti panduan https://docs.docker.com/engine/install/ubuntu/

apt-get update

apt-get install
apt-transport-https
ca-certificates
curl
gnupg-agent
software-properties-common


curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -


add-apt-repository
"deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"


apt-get update apt-get install docker-ce docker-ce-cli containerd.io


## B.2  Ganti driver docker ke systemd, awalnya cgroupfs

bash
# Set up the Docker daemon
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

bash mkdir -p /etc/systemd/system/docker.service.d


bash
# Restart Docker
systemctl daemon-reload
systemctl restart docker
systemctl enable docker

B.3 Install kubeadm

Ikuti panduan https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF | tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

B.4 Join nodesatu ke cluster

bash kubeadm join 192.168.100.198:6443 --token 98vdpp.r7tziuipa5ebwpg8
--discovery-token-ca-cert-hash sha256:d6d44f1588470756cd52f70b5576aa7ddd6b5b9a57559b025dbfa21ecee2c0eb




Langkah A dan B adalah langkah untuk setup cluster. Setelah setup master selesai dan setup nodesatu selesai, kita verifikasi:

bash
root@master:~# kubectl cluster-info
Kubernetes master is running at https://192.168.100.198:6443
KubeDNS is running at https://192.168.100.198:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@master:~# kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master     Ready    master   74m   v1.18.5   192.168.100.198   <none>        Ubuntu 18.04.4 LTS   4.15.0-108-generic   docker://19.3.12
nodesatu   Ready    <none>   41m   v1.18.5   192.168.100.199   <none>        Ubuntu 18.04.4 LTS   4.15.0-108-generic   docker://19.3.12


root@master:~# kubectl -n kube-system get po -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
coredns-66bff467f8-jghgf         1/1     Running   0          77m   10.244.0.3        master     <none>           <none>
coredns-66bff467f8-zkcdd         1/1     Running   0          77m   10.244.0.2        master     <none>           <none>
etcd-master                      1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-apiserver-master            1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-controller-manager-master   1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-flannel-ds-amd64-rb4wc      1/1     Running   0          69m   192.168.100.198   master     <none>           <none>
kube-flannel-ds-amd64-rmdw7      1/1     Running   1          44m   192.168.100.199   nodesatu   <none>           <none>
kube-proxy-hgkn9                 1/1     Running   0          44m   192.168.100.199   nodesatu   <none>           <none>
kube-proxy-lc5ps                 1/1     Running   0          77m   192.168.100.198   master     <none>           <none>
kube-scheduler-master            1/1     Running   0          77m   192.168.100.198   master     <none>           <none>

Dari data diatas, semua pod running. Aku asumsikan semua beres.

C. Pasca Setup Cluster

Setelah cluster berjalan, aku lanjutkan install aplikasi di atas cluster kubernetes. Kita kembali ke master, dan install-install dari master.

C.1 Install Helm3

Helm v3 ini akan membatu banget untuk install-install aplikasi diatas cluster kubernetes.

bash curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh

./get_helm.sh

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

helm repo update




## C.2 Install metallb

Ikuti https://metallb.universe.tf/installation/

Krn cluster yg kubuat di vm local, dan gak punya dedicated LB, maka aku butuh metallb untuk assign IP ke service bertipe LoadBalancer

bash
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Bikin configmap untuk metallb, silakan merujuk ke https://metallb.universe.tf/configuration/#layer-2-configuration

nano metallb-configmap.yaml
# isinya sbb:
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.100.30-192.168.100.40

Yang perlu diperhatikan adalah 192.168.100.30-192.168.100.40, IP ini aku kasih range dari router, ya krn ini VM di local. Kalau di cloud/datacenter range ip tersebut harus ip public supaya service bisa dijangkau dari luar.

bash kubectl apply -f metallb-configmap.yaml


Verifikasi hasil install metallb

bash
root@master:~# kubectl -n metallb-system get po -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
controller-57f648cb96-b5llj   1/1     Running   0          33m   10.244.1.2        nodesatu   <none>           <none>
speaker-csgx5                 1/1     Running   0          33m   192.168.100.199   nodesatu   <none>           <none>
speaker-mrv7x                 1/1     Running   0          33m   192.168.100.198   master     <none>           <none>

C.3 Install Nginx Ingress Controller

Ikuti https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

Ingress controller ini membantu untuk expose service ke luar cluster, misalnya expose web service via domain.

Nginx Ingress Controller hanyalah satu dari sekian banyak Ingress Controller. Selain nginx ada Traefik, HAProxy, dll. Aku sendiri selama ini pakai Traefik v1.7.24 di production. Dah kali ini aku akan pakai Nginx

bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm install my-release ingress-nginx/ingress-nginx


Perintah diatas akan install nginx di namespace default, kita check:

bash
@master:~# kubectl -n default get po,svc -o wide
NAME                                                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs   1/1     Running   0          34m   10.244.1.4   nodesatu   <none>           <none>

NAME                                                    TYPE           CLUSTER-IP             PORT(S)                      AGE   SELECTOR
service/kubernetes                                      ClusterIP      10.96.0.1        <none>           443/TCP                      85m   <none>
service/my-release-ingress-nginx-controller             LoadBalancer   10.110.208.65    192.168.100.30   80:30509/TCP,443:30954/TCP   34m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx
service/my-release-ingress-nginx-controller-admission   ClusterIP      10.104.254.133   <none>           443/TCP                      34m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=ingress-nginx

Berkat bantuan metallb, EXTERNAL-IP dari service/my-release-ingress-nginx-controller tidak pending. Dalam hal ini dia mendapatkan IP 192.168.100.30. IP tsb kita asumsikan IP public yg mana akan dimasukan ke DNS A Record

Untuk memastikan ingress controllernya berfungsi, aku akan test dari luar cluster yakni dari PC.

bash โžœ ~ curl 192.168.100.30

404 Not Found

404 Not Found


nginx/1.19.0
```

Kita coba deploy aplikasi whoareyou

Isi dari whoareyou.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoareyou-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoareyou
  template:
    metadata:
      labels:
        app: whoareyou
    spec:
      containers:
      - name: whoareyou-container
        image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoareyou-service
spec:
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoareyou
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: whoareyou-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: whoareyou.kvm.local
    http:
      paths:
      - path: /
        backend:
          serviceName: whoareyou-service
          servicePort: http

bash kubectl apply -f whoareyou.yaml


Verifikasi hasil deploy:

bash
root@master:~# kubectl -n default get po,svc,ingress
NAME                                                       READY   STATUS    RESTARTS   AGE
pod/my-release-ingress-nginx-controller-7fcf9d598b-k68gs   1/1     Running   0          45m
pod/whoareyou-deployment-74bff9c468-mlnjd                  1/1     Running   0          110s
pod/whoareyou-deployment-74bff9c468-p5bkp                  1/1     Running   0          110s

NAME                                                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/kubernetes                                      ClusterIP      10.96.0.1        <none>           443/TCP                      97m
service/my-release-ingress-nginx-controller             LoadBalancer   10.110.208.65    192.168.100.30   80:30509/TCP,443:30954/TCP   45m
service/my-release-ingress-nginx-controller-admission   ClusterIP      10.104.254.133   <none>           443/TCP                      45m
service/whoareyou-service                               ClusterIP      10.108.79.85     <none>           80/TCP                       110s

NAME                                   CLASS    HOSTS                 ADDRESS          PORTS   AGE
ingress.extensions/whoareyou-ingress   <none>   whoareyou.kvm.local   192.168.100.30   80      110s

Kita test hasil deploynya. Anggaplah whoareyou.kvm.local nama domain yg kita set dan 192.168.100.30 adalah public ip. Krn ini di local, cara testnya bisa kita masukin ke /etc/hosts komputer di luar cluster. Atau pakai curl dengan param header host. Disini aku test pakai curl

bash โžœ ~ curl -H "Host: whoareyou.kvm.local" 192.168.100.30 Hostname: whoareyou-deployment-74bff9c468-mlnjd IP: 127.0.0.1 IP: 10.244.1.7 RemoteAddr: 10.244.1.4:33058 GET / HTTP/1.1 Host: whoareyou.kvm.local User-Agent: curl/7.70.0 Accept: / X-Forwarded-For: 10.244.1.1 X-Forwarded-Host: whoareyou.kvm.local X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Real-Ip: 10.244.1.1 X-Request-Id: bce405ea511051351247e01746104180 X-Scheme: http


Sip, sukses. Yg keluar adalah output dari pod whoareyou.

Bagaimana kalau kita pakai nama domain yg lain?

bash
โžœ  ~ curl -H "Host: sembarangan.com" 192.168.100.30
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.0</center>
</body>
</html>

Outputnya adalah 404, karena ingress rule untuk domain sembarangan.com tidak ada.

Kesimpulannya adalah, kubeadm ini sangat memudahkan kita untuk setup kubernetes cluster. Banyak magic-nya. Aku senang dengan adanya kubeadm