HI.!
This short article shows how to top each node and pod to monitor cluster.
Warning)Do not do this in production! Plz!
Prereq)
Linux command basic, kubernetes kubeadm cluster...that you can find from googling or my blog!
#My k8s version is v1.15.0. Previous version is differnet ,it might be.
#In that occasion, my github article cloud help.=> https://github.com/ohyoungjooung2/kvm-vagrant-docker-k8s-config/blob/69cee8c9a585b2de7b5d0a88d50edc57a97f8b58/autoscaling-metrics-server-k8s-1.11-flannel
#Let's create self signed...
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$ openssl req -nodes -new -x509 -keyout ca.key -out aggre-ca.crt
Generating a RSA private key
............................................................+++++
.................................+++++
writing new private key to 'ca.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:KR
State or Province Name (full name) [Some-State]:SEOUL
Locality Name (eg, city) []:BIGCITY
Organization Name (eg, company) [Internet Widgits Pty Ltd]:BIGCITY.COM
Organizational Unit Name (eg, section) []:BIGCITY.COM.COR
Common Name (e.g. server FQDN or YOUR name) []:BIGCITY.COM.COR
Email Address []:
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$
openssl req -out client.csr -new -newkey rsa:4096 -nodes -keyout
client.key -subj "/CN=aggregator/O=kube-system:masters"
Can't load /home/oyj/.rnd into RNG
139859889062336:error:2406F079:random
number generator:RAND_load_file:Cannot open
file:../crypto/rand/randfile.c:88:Filename=/home/oyj/.rnd
Generating a RSA private key
..++++
...............................................................................................................................................++++
writing new private key to 'client.key'
-----
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$
openssl x509 -req -days 1000000 -in client.csr -CA aggre-ca.crt -CAkey
ca.key -set_serial 01 -out aggre-client.crt
Signature ok
subject=CN = aggregator, O = kube-system:masters
Getting CA Private Key
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$ ls -lrth | tail -5
-rw------- 1 oyj oyj 1.7K 7월 9 17:43 ca.key
-rw-rw-r-- 1 oyj oyj 1.4K 7월 9 17:44 aggre-ca.crt
-rw------- 1 oyj oyj 3.2K 7월 9 17:46 client.key
-rw-rw-r-- 1 oyj oyj 1.6K 7월 9 17:46 client.csr
-rw-rw-r-- 1 oyj oyj 1.5K 7월 9 17:46 aggre-client.crt
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$ mv client.key aggre-client.key
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$
scp -i ../id_rsa aggre-ca.crt aggre-client.crt aggre-client.key
vagrant@10.1.0.2:/home/vagrant/
aggre-ca.crt 100% 1387 461.1KB/s 00:00
aggre-client.crt 100% 1493 289.6KB/s 00:00
aggre-client.key 100% 3272 147.8KB/s 00:00
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s/metrics-server$ vagrant ssh kubemaster
Last login: Tue Jul 9 06:11:58 2019 from 192.168.121.1
[vagrant@kubemaster ~]$ ls
aggre-ca.crt aggre-client.crt aggre-client.key mariadb-master mariadb-master.tar.gz mariadb-slave wordpress
[vagrant@kubemaster ~]$ sudo cp aggre-c* /etc/kubernetes/pki/
[vagrant@kubemaster ~]$
[vagrant@kubemaster ~]$ sudo vi --cmd "set nu" /etc/kubernetes/manifests/kube-apiserver.yaml
#Comment is old key, next that line is modified to apply aggre*keys*
28 #- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
29 - --proxy-client-cert-file=/etc/kubernetes/pki/aggre-client.crt
30 #- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
31 - --proxy-client-key-file=/etc/kubernetes/pki/aggre-client.key
32 #- --requestheader-allowed-names=front-proxy-client
33 - --requestheader-allowed-names=aggregator
34 #- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
35 - --requestheader-client-ca-file=/etc/kubernetes/pki/aggre-ca.crt
#Start!..no resource metrics ..now.
oyj@oyj-X555QG:~$ kb get po
NAME READY STATUS RESTARTS AGE
mariadb-master-0 1/1 Running 26 4d14h
nfs-client-provisioner-78665db465-h98vr 1/1 Running 3 24h
[vagrant@kubemaster ~]$ kb top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
[vagrant@kubemaster ~]$ kb top pod
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ git clone https://github.com/kubernetes-incubator/metrics-server.git
Cloning into 'metrics-server'...
remote: Enumerating objects: 11185, done.
remote: Total 11185 (delta 0), reused 0 (delta 0), pack-reused 11185
Receiving objects: 100% (11185/11185), 12.08 MiB | 2.37 MiB/s, done.
Resolving deltas: 100% (5834/5834), done.
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ vi --cmd "set nu" metrics-server/deploy/1.8+/metrics-server-deployment.yaml
34 #Added from below line
35 command:
36 - /metrics-server
37 - --kubelet-insecure-tls
38 #- --kubelet-preferred-address-types=InternalIP
39 #Added part ends above line.
#First delete if it exists and then create!
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ kb delete -f metrics-server/deploy/1.8+/
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-server:system:auth-delegator" deleted
rolebinding.rbac.authorization.k8s.io "metrics-server-auth-reader" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted
serviceaccount "metrics-server" deleted
deployment.extensions "metrics-server" deleted
service "metrics-server" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-server" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-server" deleted
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ kb create -f metrics-server/deploy/1.8+/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.extensions/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ kubectl get pods -n kube-system | grep metrics
metrics-server-9fc46dbd8-p7mkm 1/1 Running 1 42s
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ kb logs -f metrics-server-6d7c9596cb-7gps7 -n kube-system
I0709 09:22:20.294897 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[vagrant@kubemaster ~]$
oyj@oyj-X555QG:~/INSTALL/u18kvk8s/k8s$ kb logs -f metrics-server-5996c7cf98-nvfbp -n kube-system
oyj@Workstation-oyj-X555QG ~/u18kvk8s/k8s/metrics-server$kb logs -f metrics-server-5996c7cf98-cxbhn -n kube-system
I0709 18:07:32.135423 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0709 18:07:32.397152 1 secure_serving.go:116] Serving securely on [::]:443
#Because like below, top node and pod is working ok now. Well, my cluster node get consumed pretty much(over 90% !)
[vagrant@kubemaster ~]$ kb top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
kubemaster 883m 44% 1107Mi 63%
kubeworker1 938m 93% 682Mi 39%
kubeworker2 940m 94% 614Mi 35%
kubeworker3 950m 95% 287Mi 16%
[vagrant@kubemaster ~]$ kb top pod
NAME CPU(cores) MEMORY(bytes)
mariadb-master-0 17m 124Mi
nfs-client-provisioner-78665db465-h98vr 2m 5Mi
No comments:
Post a Comment