UBUNTU18+KVM+VAGRANT+KUBERNETES 9)nfs server and nfs provisioning from k8s

UBUNTU18+KVM+VAGRANT+KUBERNETES 9)nfs server and nfs provisioning from k8s



This short article explains how to install kubectl PersistentVolume and PersistentVolumeClaim using outer nfs server.
We should prepare nfs node.(10.1.0.7 is my nfs server installed kvm env.


Prerequisites)
Vagrant knowledge. Beforhand articles about Ubuntu18+k8s+kvm+vagrant on my blog. Linux Command.

I refered below link. That is very awesome github. Thank you.
https://github.com/kubernetes-incubator/external-storage/blob/master/nfs-client/deploy/
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client

1.Nfs sERver install on centos7. Adding vagrant node centos7 for nfs server. - 10.1.0.7 is nfs server


   config.vm.define "nfserver" do |nfserver|
   nfserver.vm.box = "centos/7"
   nfserver.vm.provision "shell", path: "check_key.sh"
   nfserver.vm.provision "file", source: "id_rsa",destination: "/home/vagrant/.ssh/id_rsa"
   nfserver.vm.provision "shell", :path => "k8s_docker_install.sh"
   nfserver.vm.network "private_network", ip:"10.1.0.7"
   nfserver.vm.provision "annfs", type:"ansible" do |ansible|
      ansible.playbook = "nfsFileSystem.yaml"
   end
   nfserver.vm.provision "nfsdocker", type:"shell", :path => "nfsDocker.sh"
   nfserver.vm.host_name = "nfserver"
     nfserver.vm.provider :libvirt do |lv|
      lv.storage_pool_name = "data"
      lv.cpus = 1
      lv.memory = 512
      lv.storage :file, :size => '5G', :type => 'raw'
     end
   end

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ vagrant up nfserver
Bringing machine 'nfserver' up with 'libvirt' provider...
==> nfserver: Checking if box 'centos/7' version '1902.01' is up to date...
==> nfserver: A newer version of the box 'centos/7' for provider 'libvirt' is
==> nfserver: available! You currently have version '1902.01'. The latest is version
==> nfserver: '1905.1'. Run `vagrant box update` to update.
==> nfserver: Creating image (snapshot of base box volume).
==> nfserver: Creating domain with the following settings...
==> nfserver:  -- Name:              k8s_nfserver
==> nfserver:  -- Domain type:       kvm
==> nfserver:  -- Cpus:              1
==> nfserver:  -- Feature:           acpi
==> nfserver:  -- Feature:           apic
==> nfserver:  -- Feature:           pae
==> nfserver:  -- Memory:            512M
==> nfserver:  -- Management MAC:  
==> nfserver:  -- Loader:          
==> nfserver:  -- Nvram:           
==> nfserver:  -- Base box:          centos/7
==> nfserver:  -- Storage pool:      data
==> nfserver:  -- Image:             /data/k8s_nfserver.img (41G)
==> nfserver:  -- Volume Cache:      default
==> nfserver:  -- Kernel:          
==> nfserver:  -- Initrd:          
==> nfserver:  -- Graphics Type:     vnc
==> nfserver:  -- Graphics Port:     -1
==> nfserver:  -- Graphics IP:       127.0.0.1
==> nfserver:  -- Graphics Password: Not defined
==> nfserver:  -- Video Type:        cirrus
==> nfserver:  -- Video VRAM:        9216
==> nfserver:  -- Sound Type:  
==> nfserver:  -- Keymap:            en-us
==> nfserver:  -- TPM Path:        
==> nfserver:  -- Disks:         vdb(raw,5G)
==> nfserver:  -- Disk(vdb):     /data/k8s_nfserver-vdb.raw
==> nfserver:  -- INPUT:             type=mouse, bus=ps2
==> nfserver: Creating shared folders metadata...
==> nfserver: Starting domain.
==> nfserver: Waiting for domain to get an IP address...







#On below link, you can find related sources such as whole Vagrantfile above.
https://github.com/ohyoungjooung2/u18kvk8s/tree/master/k8s



2. deployment. Create deployment.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ vi ext-nfs-pro.yaml
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat ext-nfs-pro.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 10.1.0.7
            - name: NFS_PATH
              value: /
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.1.0.7
            path: /

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb create -f ext-nfs-pro.yaml
serviceaccount/nfs-client-provisioner created
deployment.extensions/nfs-client-provisioner created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep nfs
nfs-client-provisioner-78665db465-xp2wr   0/1     ContainerCreating   0          5s


oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep nfs
nfs-client-provisioner-78665db465-xp2wr   1/1     Running   0          31s
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$



3. Create class.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat storageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false" # When set
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb create -f storageClass.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get sc | grep nfs
managed-nfs-storage   fuseim.pri/ifs                 7s




4. Authorization. In my case I installed kubernets 1.15, by default RBAC is configured.
So I need to configure authorization process.

By default RBAC is on.
[vagrant@kubemaster ~]$  ps -ef | grep RBAC
root      4573  4518  4 17:27 ?        00:09:36 kube-apiserver --advertise-address=10.1.0.2 \
--allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt \
 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
 --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
--etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\
 --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\
 --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
--requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub
 --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key


Create authoriztion.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ vi clusterRoleAuth.yaml
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat clusterRoleAuth.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  #name: nfs-client-provisioner-runner
  name: nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "create", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "watch", "list", "get", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb create -f clusterRoleAuth.yaml
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get clusterRole | grep nfs
nfs-client-provisioner                                                 8s


oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ vi clusterRoleBinding.yaml
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat clusterRoleBinding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb create -f clusterRoleBinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/nfs-client-provisioner created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kubectl get clusterRoleBinding | grep nfs
nfs-client-provisioner                                 10s




5.Storage claim(persistentVolumeClaim)
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat storageClaimForWp.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: storage-claim-for-wp
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb create -f storageClaimForWp.yaml
persistentvolumeclaim/storage-claim-for-wp created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get pvc | grep for-wp
storage-claim-for-wp                 Bound    pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c   1Gi        RWX            managed-nfs-storage   12s



!Congratulations! on us!No errors.
[vagrant@kubemaster pvc]$ kubectl logs nfs-client-provisioner-74bc458c8b-rqwqv

I0707 20:49:05.579562       1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-78665db465-xp2wr_b45597ab-a0f7-11e9-b54f-62e8ccc05947!
I0707 20:49:05.579664       1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"fuseim.pri-ifs", UID:"af67b9e0-1056-4f78-af42-2604ee734c0b", APIVersion:"v1", ResourceVersion:"190065", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' nfs-client-provisioner-78665db465-xp2wr_b45597ab-a0f7-11e9-b54f-62e8ccc05947 became leader
I0707 20:49:05.696694       1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-78665db465-xp2wr_b45597ab-a0f7-11e9-b54f-62e8ccc05947!
I0707 20:51:52.219703       1 controller.go:987] provision "default/storage-claim-for-wp" class "managed-nfs-storage": started
I0707 20:51:52.226059       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"storage-claim-for-wp", UID:"8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c", APIVersion:"v1", ResourceVersion:"190411", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/storage-claim-for-wp"
I0707 20:51:52.230764       1 controller.go:1087] provision "default/storage-claim-for-wp" class "managed-nfs-storage": volume "pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c" provisioned
I0707 20:51:52.230839       1 controller.go:1101] provision "default/storage-claim-for-wp" class "managed-nfs-storage": trying to save persistentvvolume "pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c"
I0707 20:51:52.249339       1 controller.go:1108] provision "default/storage-claim-for-wp" class "managed-nfs-storage": persistentvolume "pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c" saved
I0707 20:51:52.249413       1 controller.go:1149] provision "default/storage-claim-for-wp" class "managed-nfs-storage": succeeded
I0707 20:51:52.249516       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"storage-claim-for-wp", UID:"8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c", APIVersion:"v1", ResourceVersion:"190411", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c

Now like Linux volume manager, I claim some storage from external nfs server. Just for dev test, 1Gi.


I already deploy wordpress with below ...configuration.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get svc | grep word
wordpress        NodePort    10.98.100.201   <none>        8070:32333/TCP   35h
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe svc wordpress
Name:                     wordpress
Namespace:                default
Labels:                   app=wordpress
Annotations:              <none>
Selector:                 app=wordpress
Type:                     NodePort
IP:                       10.98.100.201
Port:                     http  8070/TCP
TargetPort:               80/TCP
NodePort:                 http  32333/TCP
Endpoints:                10.244.2.37:80,10.244.2.38:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep wordpress
wordpress-749f7f984-pk6hb                 1/1     Running   3          35h
wordpress-749f7f984-t85gw                 1/1     Running   3          35h
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$



oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe deploy wordpress
Name:               wordpress
Namespace:          default
CreationTimestamp:  Sat, 06 Jul 2019 18:57:44 +0900
Labels:             app=wordpress
Annotations:        deployment.kubernetes.io/revision: 1
                    kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"wordpress"},"name":"wordpress","namespace":"defa...
Selector:           app=wordpress
Replicas:           2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:  app=wordpress
  Containers:
   wordpress:
    Image:      wordpress:4.8-apache
    Port:       80/TCP
    Host Port:  0/TCP
    Environment:
      WORDPRESS_DB_HOST:      mariadb-master
      WORDPRESS_DB_PASSWORD:  <set to the key 'password' in secret 'mariadb-pass-hmt2hb8m6g'>  Optional: false
    Mounts:
      /var/www/html from wordpress-persistent-storage (rw)
  Volumes:
   wordpress-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wp-pv-claim
    ReadOnly:   false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   wordpress-749f7f984 (2/2 replicas created)
Events:          <none>


!!!But these two pod is running on same nodes kubeworker1 that has local-disks provisioned.!!!..
 
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe po wordpress-749f7f984-pk6hb | grep kube
Node:           kubeworker1/192.168.121.6
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7pn58 (ro)
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe po wordpress-749f7f984-t85gw | grep kube
Node:           kubeworker1/192.168.121.6
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7pn58 (ro)
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

So I am going to change this two pod to be running in different nodes to balance.

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat wp-dp.yaml
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wordpress
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          #Previsous mariadb-master svc, I already created.!
          value: mariadb-master
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mariadb-pass
              key: password
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          #claimName: wp-pv-claim => this line (using local-disk provisiong) is changed into below that using nfs.
          claimName: storage-claim-for-wp
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ cat kustomization.yaml
secretGenerator:
- name: mariadb-pass
  literals:
  - password=StrongPass$^^$
resources:
  - wp-dp.yaml

#Existiing pod is being replaced with new pod(wordpress of course) like below.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po
NAME                                      READY   STATUS        RESTARTS   AGE
mariadb-master-0                          1/1     Running       18         3d2h
mariadb-slave-0                           1/1     Running       7          2d3h
mariadb-slave-1                           1/1     Running       7          2d3h
nfs-client-provisioner-78665db465-xp2wr   1/1     Running       0          23m
wordpress-749f7f984-pk6hb                 1/1     Terminating   3          35h
wordpress-749f7f984-t85gw                 1/1     Terminating   3          35h
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep wordpress
NAME                                      READY   STATUS              RESTARTS   AGE
mariadb-master-0                          1/1     Running             18         3d2h
mariadb-slave-0                           1/1     Running             7          2d3h
mariadb-slave-1                           1/1     Running             7          2d3h
nfs-client-provisioner-78665db465-xp2wr   1/1     Running             0          23m
wordpress-6bb56544f9-hbzkr                1/1     Running             0          5s

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep word

wordpress-6bb56544f9-hbzkr                1/1     Running             0          10s
wordpress-6bb56544f9-zxmw5                0/1     ContainerCreating   0          10s

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb get po | grep wordpress
wordpress-6bb56544f9-hbzkr                1/1     Running   0          72s
wordpress-6bb56544f9-zxmw5                1/1     Running   0          72s

oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe po wordpress-6bb56544f9-hbzkr | grep kube
Node:           kubeworker1/192.168.121.6
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7pn58 (ro)
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
  Normal  Scheduled  3m20s  default-scheduler     Successfully assigned default/wordpress-6bb56544f9-hbzkr to kubeworker1
  Normal  Pulled     3m18s  kubelet, kubeworker1  Container image "wordpress:4.8-apache" already present on machine
  Normal  Created    3m18s  kubelet, kubeworker1  Created container wordpress
  Normal  Started    3m17s  kubelet, kubeworker1  Started container wordpress
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/nfsProvisioning$ kb describe po wordpress-6bb56544f9-zxmw5 | grep kube
Node:           kubeworker3/192.168.121.232
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7pn58 (ro)
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
  Normal  Scheduled  3m42s  default-scheduler     Successfully assigned default/wordpress-6bb56544f9-zxmw5 to kubeworker3
  Normal  Pulling    3m40s  kubelet, kubeworker3  Pulling image "wordpress:4.8-apache"
  Normal  Pulled     2m58s  kubelet, kubeworker3  Successfully pulled image "wordpress:4.8-apache"
  Normal  Created    2m58s  kubelet, kubeworker3  Created container wordpress
  Normal  Started    2m57s  kubelet, kubeworker3  Started container wordpress



With browser connect below link, will show wordpress install page.
http://10.1.0.4:32333/wp-admin/install.php




ssh tunneling for convenient connection to wordpress that is being run in kubernet pod..etc.
[vagrant@kubemaster storage_test]$ ssh -L 9000:10.106.208.28:80 vagrant@10.1.0.2

oyj@oyj-ThinkPad-E465:~/kuber$ ssh -L 9000:localhost:9000 vagrant@10.1.0.2


http://localhost:9000 will suffice.



#Nfs servers directory...
[vagrant@nfserver ~]$ ls /nfs/
default-storage-claim-for-wp-pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c  test
[vagrant@nfserver ~]$ ls /nfs/default-storage-claim-for-wp-pvc-8c82beed-6a8a-4de9-b7de-a8d6bcecbc1c/
index.php    wp-activate.php     wp-comments-post.php  wp-content   wp-links-opml.php  wp-mail.php      wp-trackback.php
license.txt  wp-admin            wp-config-sample.php  wp-cron.php  wp-load.php        wp-settings.php  xmlrpc.php
readme.html  wp-blog-header.php  wp-config.php         wp-includes  wp-login.php       wp-signup.php
[vagrant@nfserver ~]$

UBUNTU18+KVM+VAGRANT+KUBERNETES 4)HOW TO ADD WORKER NODE ON A CLUSTER

#This docu explain how to add worker node on a K8S cluster.

[vagrant@kubemaster ~]$ kubeadm token create
tfj754.wlscygdbg7bay497
[vagrant@kubemaster ~]$ kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
tfj754.wlscygdbg7bay497 23h 2018-08-26T10:06:48Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token


[vagrant@kubeworker3 ~]$ sudo -i
[root@kubeworker3 ~]# kubeadm join --token tfj754.wlscygdbg7bay497 10.1.0.2:6443
discovery: Invalid value: "": using token-based discovery without discoveryTokenCACertHashes can be unsafe. set --discovery-token-unsafe-skip-ca-verification to continue
[root@kubeworker3 ~]#


[vagrant@kubemaster ~]$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
> openssl dgst -sha256 -hex | sed 's/^.* //'
42e4571e6081c102d202ee6e3a2a2753a13f33ac1a463734fdff200cfb40d413
[vagrant@kubemaster ~]$


[root@kubeworker3 ~]# kubeadm join --token tfj754.wlscygdbg7bay497 10.1.0.2:6443 --discovery-token-ca-cert-hash sha256:42e4571e6081c102d202ee6e3a2a2753a13f33ac1a463734fdff200cfb40d413
[preflight] running pre-flight checks
 [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0825 10:20:22.768168 4841 kernel_validator.go:81] Validating kernel version
I0825 10:20:22.768442 4841 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "10.1.0.2:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.1.0.2:6443"
[discovery] Requesting info from "https://10.1.0.2:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.1.0.2:6443"
[discovery] Successfully established connection with API Server "10.1.0.2:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubeworker3" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
 was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.


[vagrant@kubemaster ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubemaster Ready master 12d v1.11.2
kubeworker1 Ready <none> 12d v1.11.2
kubeworker2 Ready <none> 12d v1.11.2
kubeworker3 NotReady <none> 56s v1.11.2



[vagrant@kubemaster ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubemaster Ready master 12d v1.11.2
kubeworker1 Ready <none> 12d v1.11.2
kubeworker2 Ready <none> 12d v1.11.2
kubeworker3 Ready <none> 2m v1.11.2

UBUNTU18+KVM+VAGRANT+K8S 3)INSTALL K8S

UBUNTU18+KVM+VAGRANT+K8S 3)INSTALL K8S

UBUNTU18+KVM+VAGRANT+K8S 2)INSTALL VAGRANT

UBUNTU18+KVM+VAGRANT+K8S 2)INSTALL VAGRANT

UBUNTU18+KVM+VAGRANT+K8S 1)INSTALL KVM

UBUNTU18+KVM+VAGRANT+K8S 1)INSTALL KVM

UBUNTU18+KVM+VAGRANT+KUBERNETES 8 Provisioning local disk and wordpress ...

HI.!
This short article shows how to provision and install wordpress  using local-disk provisioning.
In this case, pod shall be running the server(node) that has local-disk(physically).

!!!If you want to know more about local-disk provisioning visit below site!!!
https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/

Prerequisites) Knowledge about Linux command, docker,k8s and some articles.
  If you don't mind, conveniently you can see below youtubu. Sorry for not recording good quality though.^^'
https://www.youtube.com/channel/UCRGmm0EOwn8qRXigPLuPQig
github)
https://github.com/ohyoungjooung2/u18kvk8s/tree/master/k8s/wordpress

Creating wordpress cms.(storage is local-disk provision)
1)Creating storage class(sc) and persistent volumes for wordpress
2) Create service for wordpress(node port)
3) Create deployment wordpress
4) Connection test


1)Creating storage class(sc) and persistent volumes for wordpress
[vagrant@kubemaster wordpress]$ vi wordpress-sc.yaml
[vagrant@kubemaster wordpress]$ kb create -f wordpress-sc.yaml
storageclass.storage.k8s.io/wordpress-sc created
[vagrant@kubemaster wordpress]$ alias | grep kb
alias kb='kubectl'
[vagrant@kubemaster wordpress]$ cat wordpress-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: wordpress-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

[vagrant@kubemaster wordpress]$ kb describe sc  wordpress-sc
Name:                  wordpress-sc
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           kubernetes.io/no-provisioner
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>




[vagrant@kubemaster wordpress]$ vi wordpress-pv.yaml
[vagrant@kubemaster wordpress]$ cat wordpress-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: wordpress-pv
spec:
  capacity:
    storage: 1Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: wordpress-sc
  local:
    path: /web
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubeworker1
[vagrant@kubemaster wordpress]$ kb create -f wordpress-pv.yaml
persistentvolume/wordpress-pv created
[vagrant@kubemaster wordpress]$ kb get pv | grep wordpress
wordpress-pv          1Gi        RWO            Recycle           Available                                                wordpress-sc                 11

2) Create service for wordpress
[vagrant@kubemaster wordpress]$ vi wordpress-svc.yaml
[vagrant@kubemaster wordpress]$ cat wordpress-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress

  type: NodePort
  ports:
  - port: 8070
    nodePort: 32333
    targetPort: 80
    protocol: TCP
    name: http
[vagrant@kubemaster wordpress]$ kb create -f wordpress-svc.yaml
service/wordpress created

 

3) Create  wordpress deployment
[vagrant@kubemaster wordpress]$ cat wp-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: wordpress-sc
[vagrant@kubemaster wordpress]$ kb create -f wp-pvc.yaml
persistentvolumeclaim/wp-pv-claim created
[vagrant@kubemaster wordpress]$ kb get pvc | grep wp-pv-claim
wp-pv-claim                          Pending                                                   wordpress-sc        9s



[vagrant@kubemaster wordpress]$ vi kustomization.yaml
[vagrant@kubemaster wordpress]$ cat kustomization.yaml
secretGenerator:
- name: mariadb-pass
  literals:
  - password=StrongPass$^^$
resources:
  - wp-dp.yaml

[vagrant@kubemaster wordpress]$ vi wp-dp.yaml
[vagrant@kubemaster wordpress]$ cat wp-dp.yaml
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - image: wordpress:4.8-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: mariadb-master
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mariadb-pass
              key: password
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim

[vagrant@kubemaster wordpress]$ kb apply -k .
secret/mariadb-pass-hmt2hb8m6g unchanged
deployment.apps/wordpress created
[vagrant@kubemaster wordpress]$ kb get po | grep wordpress
wordpress-57646c5f5f-f7g7l   1/1     Running   0          8s
[vagrant@kubemaster wordpress]$


[vagrant@kubemaster wordpress]$ kb describe svc wordpress
Name:                     wordpress
Namespace:                default
Labels:                   app=wordpress
Annotations:              <none>
Selector:                 app=wordpress
Type:                     NodePort
IP:                       10.97.157.133
Port:                     http  8070/TCP
TargetPort:               80/TCP
NodePort:                 http  32333/TCP
Endpoints:                10.244.2.19:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>


4) Connection test
#Check node ip and port with telnet
[vagrant@kubemaster wordpress]$ telnet 10.1.0.4 32333
Trying 10.1.0.4...
Connected to 10.1.0.4.
Escape character is '^]'.
^C^]
telnet> quit
Connection closed.
[vagrant@kubemaster wordpress]$ telnet 10.1.0.3 32333
Trying 10.1.0.3...
Connected to 10.1.0.3.
Escape character is '^]'.
^]
telnet> q
Connection closed.
[vagrant@kubemaster wordpress]$ telnet 10.1.0.5 32333
Trying 10.1.0.5...
Connected to 10.1.0.5.
Escape character is '^]'.
^]
telnet> q
Connection closed.

[vagrant@kubemaster wordpress]$ kb get svc | grep word
wordpress        NodePort    10.97.157.133   <none>        8070:32333/TCP   92m

[vagrant@kubemaster wordpress]$ curl -v http://10.97.157.133:8070
* About to connect() to 10.97.157.133 port 8070 (#0)
*   Trying 10.97.157.133...
* Connected to 10.97.157.133 (10.97.157.133) port 8070 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.97.157.133:8070
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Sat, 06 Jul 2019 09:28:01 GMT
< Server: Apache/2.4.10 (Debian)
< X-Powered-By: PHP/5.6.32
< Expires: Wed, 11 Jan 1984 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0
< Location: http://10.97.157.133:8070/wp-admin/install.php
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host 10.97.157.133 left intact




Conclusion)
In this  article, I introduce how to provisioing wordpress deployment and and storages.
In real production, maybe there are lots of other services related with wordpress such as cache servers and so on.


If you know more about deployment, please visit https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Thanks for reading.

UBUNTU18+KVM+VAGRANT+KUBERNETES 7 Provisioning local disk and mariadb sl...

HI.!
This short article shows how to provision and install mariadb-slave  using local-disk provisioning.
In this case, pod must be running the server(node) that has local-disk(physically).

!!!If you want to know more about local-disk provisioning visit below site!!!
https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/

Prerequisites) Knowledge about Linux command, docker,k8s and some articles.
  If you don't mind, conveniently you can see below youtubu. Sorry for not recording good quality though.^^'
https://www.youtube.com/channel/UCRGmm0EOwn8qRXigPLuPQig
github)
https://github.com/ohyoungjooung2/u18kvk8s/tree/master/k8s/mariadb-slave

Creating mariadb-slave server.(storage is local-disk provision)
1)Creating storage class(sc) and persistent volumes for mariadb-slave
2) Create service for mariadb-slave
3) Create Stateful mariadb-slave
4) Replication test.


1)Creating storage class(sc) and persistent volumes for mariadb-slave
[vagrant@kubemaster mariadb-slave]$ cat mariadb-slave-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mariadb-slave-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer


vagrant@kubemaster mariadb-slave]$ kb create -f mariadb-slave-sc.yaml
storageclass.storage.k8s.io/mariadb-slave-sc created

[vagrant@kubemaster mariadb-slave]$ kb describe sc  mariadb-slave-sc
Name:                  mariadb-slave-sc
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           kubernetes.io/no-provisioner
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>



[vagrant@kubemaster mariadb-slave]$ vi mariadb-slave-pv.yaml
[vagrant@kubemaster mariadb-slave]$ cat mariadb-slave-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mariadb-slave-pv0
spec:
  capacity:
    storage: 1Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mariadb-slave-sc
  local:
    path: /db1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubeworker2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mariadb-slave-pv1
spec:
  capacity:
    storage: 1Gi
  # volumeMode field requires BlockVolume Alpha feature gate to be enabled.
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: mariadb-slave-sc
  local:
    path: /db1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubeworker3

[vagrant@kubemaster mariadb-slave]$ kb create -f mariadb-slave-pv.yaml
persistentvolume/mariadb-slave-pv0 created
persistentvolume/mariadb-slave-pv1 created
[vagrant@kubemaster mariadb-slave]$ kb get pv
NAME                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                        STORAGECLASS        REASON   AGE
mariadb-master-disk   1Gi        RWO            Recycle          Bound       default/mariadb-master-st-mariadb-master-0   mariadb-master-sc            21h
mariadb-slave-pv0     1Gi        RWO            Delete           Available                                                mariadb-slave-sc             3s
mariadb-slave-pv1     1Gi        RWO            Delete           Available                                                mariadb-slave-sc             3s
[vagrant@kubemaster mariadb-slave]$




2) Create service for mariadb-slave
[vagrant@kubemaster mariadb-slave]$ vi mariadb-slave-svc.yaml
[vagrant@kubemaster mariadb-slave]$ cat mariadb-slave-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: mariadb-slave
  labels:
    app: mariadb-slave
spec:
  ports:
  - name: mariadb-slave
    port: 3306
  clusterIP: None
  selector:
    app: mariadb-slave
 

[

3) Create Stateful mariadb-slave
[vagrant@kubemaster mariadb-slave]$ vi kustomization.yaml
[vagrant@kubemaster mariadb-slave]$ cat kustomization.yaml
secretGenerator:
- name: mariadb-pass
  literals:
  - password=StrongPass$^^$
- name: rep-user
  literals:
  - rep-user=rep-user
- name: rep-password
  literals:
  - rep-password=Good$^Password!
resources:
  - mariadb-slave.yaml


[vagrant@kubemaster mariadb-slave]$ vi mariadb-slave.yaml
[vagrant@kubemaster mariadb-slave]$ cat mariadb-slave.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-slave
spec:
  selector:
    matchLabels:
      app: mariadb-slave
  serviceName: mariadb-slave
  replicas: 2
  template:
    metadata:
      labels:
        app: mariadb-slave
    spec:
      initContainers:
      - name: init-mariadb-slave
        image: ohyoungjooung2/mariadb:10.1.14-slave2
        command:
        - bash
        - "-c"
        -  |
           set -ex
           [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
           ordinal=${BASH_REMATCH[1]}
           id=$(( $ordinal + 2 ))
           echo [mysqld] > /mnt/conf.d/server-id.cnf
           echo server-id=$id >> /mnt/conf.d/server-id.cnf
           echo log-bin >> /mnt/conf.d/server-id.cnf
           echo log-basename=slave$id >> /mnt/conf.d/server-id.cnf
           echo character_set_server=utf8 >> /mnt/conf.d/server-id.cnf
           echo collation_server=utf8_general_ci >> /mnt/conf.d/server-id.cnf

        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
       
 
      containers:
      - name: mariadb
        image: ohyoungjooung2/mariadb:10.1.14-slave2
        lifecycle:
        env:
        - name: MYSQL_ALLOW_EMPTY_PASSWORD
          value: "no"
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
               name: mariadb-pass
               key: password
        - name: MYSQL_MASTER_SERVICE_HOST
          value: mariadb-master
        
        - name: MYSQL_REPLICATION_USER
          valueFrom:
            secretKeyRef:
               name: rep-user
               key: rep-user

        - name: MYSQL_REPLICATION_PASSWORD
          valueFrom:
            secretKeyRef:
               name: rep-password
               key: rep-password



        ports:
        - name: mariadb-slave
          containerPort: 3306
        volumeMounts:
        - name: mariadb-slave
          mountPath: /var/lib/mysql
          subPath: mariadb-sv
        - name: conf     
          mountPath: /etc/mysql/conf.d


        resources:
          requests:
            cpu: 500m
            memory: 1Gi

        livenessProbe:
          exec:
            command: ["mysqladmin", "ping"]
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5


      volumes:
      - name: conf
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      #name: data
      name: mariadb-slave
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "mariadb-slave-sc"
      resources:
        requests:
          storage: 1Gi
[vagrant@kubemaster mariadb-slave]$ kb apply -k .
secret/mariadb-pass-hmt2hb8m6g created
secret/rep-password-8bhkm96dbt created
secret/rep-user-bttd79k4t8 created
statefulset.apps/mariadb-slave created
[vagrant@kubemaster mariadb-slave]$ kb get po
NAME               READY   STATUS     RESTARTS   AGE
mariadb-master-0   1/1     Running    11         21h
mariadb-slave-0    0/1     Init:0/1   0          3s
[vagrant@kubemaster mariadb-slave]$ kb get po
NAME               READY   STATUS    RESTARTS   AGE
mariadb-master-0   1/1     Running   11         21h
mariadb-slave-0    1/1     Running   0          6s
mariadb-slave-1    0/1     Pending   0          1s
[vagrant@kubemaster mariadb-slave]$ kb get po
NAME               READY   STATUS            RESTARTS   AGE
mariadb-master-0   1/1     Running           11         21h
mariadb-slave-0    1/1     Running           0          8s
mariadb-slave-1    0/1     PodInitializing   0          3s
[vagrant@kubemaster mariadb-slave]$ kb get po
NAME               READY   STATUS    RESTARTS   AGE
mariadb-master-0   1/1     Running   11         21h
mariadb-slave-0    1/1     Running   0          10s
mariadb-slave-1    1/1     Running   0          5s


4) mariadb replication test
[vagrant@kubemaster mariadb-slave]$ kb exec -it mariadb-slave-0 bash
root@mariadb-slave-0:/# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.1.14-MariaDB-1~jessie mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mariadb-master
                  Master_User: rep-user
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master1-bin.000018
          Read_Master_Log_Pos: 329
               Relay_Log_File: slave2-relay-bin.000003
                Relay_Log_Pos: 27075
        Relay_Master_Log_File: master1-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1146
                   Last_Error: Error 'Table 'mysql.tmp_proxies_priv' doesn't exist' on query. Default database: 'mysql'. Query: 'INSERT INTO  proxies_priv SELECT * FROM tmp_proxies_priv WHERE @had_proxies_priv_table=0;'
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 26785
              Relay_Log_Space: 6400572
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1146
               Last_SQL_Error: Error 'Table 'mysql.tmp_proxies_priv' doesn't exist' on query. Default database: 'mysql'. Query: 'INSERT INTO  proxies_priv SELECT * FROM tmp_proxies_priv WHERE @had_proxies_priv_table=0;'
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
      Replicate_Do_Domain_Ids:
  Replicate_Ignore_Domain_Ids:
                Parallel_Mode: conservative
1 row in set (0.00 sec)

ERROR: No query specified

MariaDB [(none)]> STOP SLAVE;
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> START SLAVE;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mariadb-master
                  Master_User: rep-user
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master1-bin.000018
          Read_Master_Log_Pos: 329
               Relay_Log_File: slave2-relay-bin.000006
                Relay_Log_Pos: 1959260
        Relay_Master_Log_File: master1-bin.000004
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 1958970
              Relay_Log_Space: 6372576
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 80826
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
      Replicate_Do_Domain_Ids:
  Replicate_Ignore_Domain_Ids:
                Parallel_Mode: conservative
1 row in set (0.00 sec)

ERROR: No query specified

[vagrant@kubemaster mariadb-slave]$ kb exec -it mariadb-slave-1 bash
root@mariadb-slave-1:/# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 41
Server version: 10.1.14-MariaDB-1~jessie mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> STOP SLAVE;
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> START SLAVE;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mariadb-master
                  Master_User: rep-user
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: master1-bin.000018
          Read_Master_Log_Pos: 329
               Relay_Log_File: slave3-relay-bin.000006
                Relay_Log_Pos: 4221995
        Relay_Master_Log_File: master1-bin.000004
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 4221705
              Relay_Log_Space: 6372576
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 80945
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:
      Replicate_Do_Domain_Ids:
  Replicate_Ignore_Domain_Ids:
                Parallel_Mode: conservative
1 row in set (0.00 sec)

MariaDB [(none)]> exit
Bye
root@mariadb-slave-1:/# exit
exit

[vagrant@kubemaster mariadb-slave]$ kb exec -it mariadb-master-0 -- bash

MariaDB [(none)]> CREATE DATABASE tttt;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> use tttt;
Database changed
MariaDB [tttt]> create table tt (name varchar(20));
Query OK, 0 rows affected (0.04 sec)

MariaDB [tttt]> insert into tt values('ttttt');
Query OK, 1 row affected (0.01 sec)

MariaDB [tttt]> insert into tt values('tttttdddd');
Query OK, 1 row affected (0.00 sec)

MariaDB [tttt]> insert into tt values('Tom');
Query OK, 1 row affected (0.01 sec)

MariaDB [tttt]> insert into tt values('Jerry');
Query OK, 1 row affected (0.00 sec)



MariaDB [tttt]> exit
Bye
root@mariadb-master-0:/# exit
exit

[vagrant@kubemaster mariadb-slave]$ kb exec -it mariadb-slave-0 bash
root@mariadb-slave-0:/# mysql -e 'select * from tttt.tt'
+-----------+
| name      |
+-----------+
| ttttt     |
| tttttdddd |
| Tom       |
| Jerry     |
+-----------+

[vagrant@kubemaster mariadb-slave]$ kb exec -it mariadb-slave-1 bash
root@mariadb-slave-1:/# mysql -e 'select * from tttt.tt'
+-----------+
| name      |
+-----------+
| ttttt     |
| tttttdddd |
| Tom       |
| Jerry     |
+-----------+


Conclusion)
In this  article, I introduce how to provisioing mariadb slave and and storages.
In real production, database server's installation would be better on bare metal server, but this solution could be considered if someone like
all pod and container native.



If you know more about statefulset, please visit https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/