DEPLOYMENT SPRINGBOOT-API-MYSQL APP ON LOCAL K8S CLUSTER USING AWS ECR.

 This article shows how to deploy java spboot-api-mysql application into local k8s(1.17) cluster. I have not enough resource on my desktop, so I use AWS ecr as a docker registory. This could be used as a hybrid cloud context. As a storage, I use nfs server that is only used in development or experimental environment.

Currently, I have this local k8s clusters.(Using vagrant and libvirt).

All command line is on Linux box(I use xubuntu).

And aws-cli is already configured on my box and cluster(k8s master)

#Cluster one master and 2 worker nodes.

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get no
NAME          STATUS   ROLES    AGE     VERSION
kubemaster    Ready    master   6d20h   v1.17.0
kubeworker1   Ready    <none>   6d20h   v1.17.0
kubeworker2   Ready    <none>   6d20h   v1.17.0

#Vagrant

oyj@controller:~/prac/u18kvk8s/k8s$ vagrant status
Current machine states:

kubemaster                running (libvirt)
kubeworker1               running (libvirt)
kubeworker2               running (libvirt)
nfserver                  running (libvirt)

1. Create spboot-api-mysl docker image.

 #git clone

oyj@controller:~/prac$ git clone https://github.com/ohyoungjooung2/spboot-api-mysl.git

 #Mysql database authorization. Currently I installed mysql on my desktop .

root@controller:~# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.30-0ubuntu0.18.04.1 (Ubuntu)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database spbootapimysl
    -> ;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON spbootapimysl.* TO 'spboot'@'%' IDENTIFIED BY '!GoodPassplz#8';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

oyj@controller:~/prac/spboot-api-mysl$ mysql -u spboot -p spbootapimysl
Enter password:


mysql> show tables;
Empty set (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| spbootapimysl      |
+--------------------+
2 rows in set (0.00 sec)

#Modify application.properties file.

oyj@controller:~/prac/spboot-api-mysl$ vi src/main/resources/application.properties

server.port=10001
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
#spring.datasource.url = jdbc:mysql://localhost:3306/spbootapimysl?characterEncoding=UTF-8&serverTimezone=UTC
spring.datasource.url = jdbc:mysql://172.17.0.1:3306/spbootapimysl?characterEncoding=UTF-8&serverTimezone=UTC
spring.datasource.username = spboot
spring.datasource.password = !GoodPassplz#8
spring.jpa.show-sql = true
spring.jpa.hibernate.ddl-auto = update
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
logging.level.web=debug

#Create jar

oyj@controller:~/prac/spboot-api-mysl$ ./mvnw install

[INFO] Installing /home/oyj/prac/spboot-api-mysl/target/spriboot-crud-api-mysql-0.0.1-SNAPSHOT.jar to /home/oyj/.m2/repository/com/myqlapi/spriboot-crud-api-mysql/0.0.1-SNAPSHOT/spriboot-crud-api-mysql-0.0.1-SNAPSHOT.jar
[INFO] Installing /home/oyj/prac/spboot-api-mysl/pom.xml to /home/oyj/.m2/repository/com/myqlapi/spriboot-crud-api-mysql/0.0.1-SNAPSHOT/spriboot-crud-api-mysql-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.499 s
[INFO] Finished at: 2020-07-13T00:17:56+09:00
[INFO] ------------------------------------------------------------------------

oyj@controller:~/prac/spboot-api-mysl$ ls target/spriboot-crud-api-mysql-0.0.1-SNAPSHOT.jar
target/spriboot-crud-api-mysql-0.0.1-SNAPSHOT.jar

#Docker build

oyj@controller:~/prac/spboot-api-mysl$ cat Dockerfile
FROM openjdk:8-jdk-alpine AS builder
WORKDIR target/dependency
ARG APPJAR=target/*.jar
COPY ${APPJAR} app.jar
RUN jar -xf ./app.jar

FROM openjdk:8-jre-alpine
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY --from=builder ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY --from=builder ${DEPENDENCY}/META-INF /app/META-INF
COPY --from=builder ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.myqlapi.crudapi.ex.SpribootCrudApiMysqlApplication"]

oyj@controller:~/prac/spboot-api-mysl$ docker build . -t spboot-api-mysl:latest

Step 8/12 : ARG DEPENDENCY=target/dependency
 ---> Using cache
 ---> f4adeb4a8754
Step 9/12 : COPY --from=builder ${DEPENDENCY}/BOOT-INF/lib /app/lib
 ---> Using cache
 ---> 5d08fda48513
Step 10/12 : COPY --from=builder ${DEPENDENCY}/META-INF /app/META-INF
 ---> Using cache
 ---> f0b5e97bbd7b
Step 11/12 : COPY --from=builder ${DEPENDENCY}/BOOT-INF/classes /app
 ---> Using cache
 ---> aa8f20b7e63e
Step 12/12 : ENTRYPOINT ["java","-cp","app:app/lib/*","com.myqlapi.crudapi.ex.SpribootCrudApiMysqlApplication"]
 ---> Using cache
 ---> 4f2a73bf2e33
Successfully built 4f2a73bf2e33
Successfully tagged spboot-api-mysl:latest

#Run docker

docker run -d -p 127.0.0.1:10000:10001 spboot-api-mysl

#Check with curl.

oyj@controller:~/prac/spboot-api-mysl$ curl http://localhost:10000/products
[]

oyj@controller:~/prac/spboot-api-mysl$
#Post and delete check.

oyj@controller:~/prac/spboot-api-mysl$ curl -H "Content-Type: applicati{ "name":"AppleCar","quantity":3,"price":20000}' http://localhost:10000/addproduct/
{"id":1,"name":"AppleCar","quantity":3,"price":20000.0}oyj@controller:~/prac/spboot-api-mysl$

oyj@controller:~/prac/spboot-api-mysl$ curl http://localhost:10000/products
[{"id":1,"name":"AppleCar","quantity":3,"price":20000.0}]oyj@controller:~/prac/spboot-api-mysl$

oyj@controller:~/prac/spboot-api-mysl$ curl -H "Content-Type:application/json" -X DELETE http://localhost:10000/delete/1
product removed!!1oyj@controller:~/prac/spboot-api-mysl$
oyj@controller:~/prac/spboot-api-mysl$ curl http://localhost:10000/products
[]oyj@controller:~/prac/spboot-api-mysl$

#It works well.^^

2. Upload image into AWS ECR.

#Create repository

oyj@controller:~/prac/spboot-api-mysl$ aws ecr create-repository --repository-name spboot-api-mysl --image-tag-mutability IMMUTABLE
{
    "repository": {
        "repositoryArn": "arn:aws:ecr:ap-northeast-2:494307375889:repository/spboot-api-mysl",
        "registryId": "494307375889",
        "repositoryName": "spboot-api-mysl",
        "repositoryUri": "494307375889.dkr.ecr.ap-northeast-2.amazonaws.com/spboot-api-mysl",
        "createdAt": "2020-07-13T01:01:31+09:00",
        "imageTagMutability": "IMMUTABLE",
        "imageScanningConfiguration": {
            "scanOnPush": false
        }
    }
}

#Tag and push

oyj@controller:~/prac/spboot-api-mysl$ docker tag spboot-api-mysl:latest 494307375889.dkr.ecr.ap-northeast-2.amazonaws.com/spboot-api-mysl:latest

oyj@controller:~/prac/spboot-api-mysl$ docker push  494307375889.dkr.ecr.ap-northeast-2.amazonaws.com/spboot-api-mysl:latest
The push refers to repository [494307375889.dkr.ecr.ap-northeast-2.amazonaws.com/spboot-api-mysl]
23e0111c8ffa: Pushed
5c43cd603e6d: Pushed
c20cb2b5afc9: Pushed
edd61588d126: Pushed
9b9b7f3d56a0: Pushed
f1b5933fe4b5: Pushed
latest: digest: sha256:29a19a56a09bae22a1b83b20d869d8b217fc1eb4f390192c7b97a6ee9e10a702 size: 1575

#Image List

oyj@controller:~/prac/spboot-api-mysl$ aws ecr list-images --repository-name spboot-api-mysl
{
    "imageIds": [
        {
            "imageDigest": "sha256:29a19a56a09bae22a1b83b20d869d8b217fc1eb4f390192c7b97a6ee9e10a702",
            "imageTag": "latest"
        }
    ]
}

3. Create mysql service and api,mysql pod deployment on local k8s cluster.

   ref) How to create nfs provisioner.

   https://wnapdlf.blogspot.com/2019/07/ubuntu18kvmvagrantkubernetes-9nfs.html

  

  1) nfs storage : create storage class.

    [vagrant@kubemaster sprngbootmysqlapi]$ vi stclassForApi.yaml
[vagrant@kubemaster sprngbootmysqlapi]$ cat stclassForApi.yaml
#For k8s 1.17
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: api-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false" # When set
        
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl create -f stclassForApi.yaml
storageclass.storage.k8s.io/api-nfs-storage created
[vagrant@kubemaster sprngbootmysqlapi]$

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get sc api-nfs-storage
NAME              PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
api-nfs-storage   fuseim.pri/ifs   Delete          Immediate           false                  52s

  2) Claim storage.

[vagrant@kubemaster sprngbootmysqlapi]$ cat stclaimForMysl.yaml
#For 1.17 for now(18?)
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: st-claim-for-api-mysql
  annotations:
    volume.beta.kubernetes.io/storage-class: "api-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

#For mysql server

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl create -f stclaimForMysl.yaml
persistentvolumeclaim/st-claim-for-api-mysql created
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get pvc | grep api
st-claim-for-api-mysql     Bound    pvc-59ccf3a8-fe96-4e99-922b-4b091e04d596   2Gi        RWX            api-nfs-storage       8s

#For api app

[vagrant@kubemaster sprngbootmysqlapi]$ vi stclaimForapp.yaml
[vagrant@kubemaster sprngbootmysqlapi]$ cat stclaimForapp.yaml
#For 1.17 for now(18?)
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: st-claim-for-api-app
  annotations:
    volume.beta.kubernetes.io/storage-class: "api-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl create -f stclaimForapp.yaml
persistentvolumeclaim/st-claim-for-api-app created
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get pvc st-claim-for-api-app
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
st-claim-for-api-app   Bound    pvc-bb6afcc0-adf5-424b-816b-a9014b67ad76   1Gi        RWX            api-nfs-storage   10s

  3) kustomization.yaml creation.

[vagrant@kubemaster sprngbootmysqlapi]$ cat kustomization.yaml

secretGenerator:
- name: mysql-root-pass
  literals:
  - password=StrongPass$^^$
- name: spriapi-database
  literals:
  - database=spbootapimysl
  - url='jdbc:mysql://spboot-api-mysl-host:3306/spbootapimysl?useSSL=false&serverTimezone=UTC&useLegacyDatetimeCode=false'
- name: mysql-user-pass
  literals:
  - username=spriboot
  - password=!passGood$8

resources:
  - spboot-api-mysl-dp.yaml
  - api-mysl-srv-dp.yaml

 4) Create api-mysl-srv-dp.yaml (deployment) and spboot-api-mysl-dp.yaml

  [vagrant@kubemaster sprngbootmysqlapi]$ cat api-mysl-srv-dp.yaml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: spboot-api-mysl-srv
  labels:
    app: spboot-api-mysl-srv
spec:
  selector:
    matchLabels:
      app: spboot-api-mysl-srv
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: spboot-api-mysl-srv
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-root-pass
              key: password
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: spriapi-database
              key: database
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: username
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: password

        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: st-claim-for-api-mysql

[vagrant@kubemaster sprngbootmysqlapi]$ cat spboot-api-mysl-dp.yaml
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: spboot-api-mysl
  labels:
    app: spboot-api-mysl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spboot-api-mysl
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: spboot-api-mysl
    spec:
      containers:
      - image: 494307375889.dkr.ecr.ap-northeast-2.amazonaws.com/spboot-api-mysl:latest
        name: spboot-api-mysl
        env:
        - name: SPRING_DATASOURCE_USERNAME
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: username
           
        - name: SPRING_DATASOURCE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: password
        - name: SPRING_DATASOURCE_URL
          valueFrom:
            secretKeyRef:
              name: spriapi-database
              key: url
        ports:
        - containerPort: 10001
          name: spboot-api-mysl

        volumeMounts:
        - name: spboot-api-mysl-persistent-storage
          mountPath: /tmp
      #this is generate with bash shell below. 5)
      imagePullSecrets:
       - name: ap-northeast-2-ecr-registry

      volumes:
      - name: spboot-api-mysl-persistent-storage
        persistentVolumeClaim:
          claimName: st-claim-for-api-app

 5)Get authrized to retrive ecr docker image.

   [vagrant@kubemaster sprngbootmysqlapi]$ vi get_aws_token_ecr.sh
[vagrant@kubemaster sprngbootmysqlapi]$ cat get_aws_token_ecr.sh
#!/usr/bin/env bash
#This script is from stackexchange.com. Many thanks.
ACCOUNT=494307375889
REGION=ap-northeast-2
SECRET_NAME=${REGION}-ecr-registry
EMAIL=email@email.com

#
#

TOKEN=`aws ecr --region=$REGION get-authorization-token --output text \
    --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`

#
#  Create or replace registry secret
#


kubectl delete secret --ignore-not-found $SECRET_NAME
kubectl create secret docker-registry $SECRET_NAME \
    --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
    --docker-username=AWS \
    --docker-password="${TOKEN}" \
    --docker-email="${EMAIL}"
[vagrant@kubemaster sprngbootmysqlapi]$ bash get_aws_token_ecr.sh
secret "ap-northeast-2-ecr-registry" deleted
secret/ap-northeast-2-ecr-registry created

6) Create mysql service(svc).

[vagrant@kubemaster sprngbootmysqlapi]$ vi api-mysl-svc.yaml
[vagrant@kubemaster sprngbootmysqlapi]$ cat api-mysl-svc.yaml
#For 1.17
apiVersion: v1
kind: Service
metadata:
  name: spboot-api-mysl-host
  labels:
    app: spboot-api-mysl-srv
spec:
  ports:
    - port: 3306
  selector:
    app: spboot-api-mysl-srv
    tier: mysql
  clusterIP: None
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl create -f api-mysl-svc.yaml
service/spboot-api-mysl-host created
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get svc
NAME                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes                     ClusterIP   10.96.0.1    <none>        443/TCP    6d23h
spboot-api-mysl-host           ClusterIP   None         <none>        3306/TCP   6s

7) Execute kustomization.yaml

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl apply -k .
secret/mysql-root-pass-82tbbm75f9 created
secret/mysql-user-pass-c2t6ghdcf5 created
secret/spriapi-database-9b2mktcc2h created
deployment.apps/spboot-api-mysl-srv created
deployment.apps/spboot-api-mysl configured
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get po
NAME                                      READY   STATUS              RESTARTS   AGE
nfs-client-provisioner-67b6b87bc9-fgmtr   1/1     Running             2          2d
nfs-client-provisioner-67b6b87bc9-tcw5r   1/1     Running             2          2d
spboot-api-mysl-865f787d56-q656t          0/1     ContainerCreating   0          2s
spboot-api-mysl-srv-858f7db96-v8qhq       1/1     Running             0          74s
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get po
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-67b6b87bc9-fgmtr   1/1     Running   2          2d
nfs-client-provisioner-67b6b87bc9-tcw5r   1/1     Running   2          2d
spboot-api-mysl-865f787d56-q656t          1/1     Running   0          5s
spboot-api-mysl-srv-858f7db96-v8qhq       1/1     Running   0          77s
[vagrant@kubemaster sprngbootmysqlapi]$

 8) #Test with curl.

   [vagrant@kubemaster sprngbootmysqlapi]$ curl http://10.244.1.35:10001/products
[]
[vagrant@kubemaster sprngbootmysqlapi]$ curl -H "Content-Type: application/json" -X POST -d '{"name":"ApplCar","quantity":3,"price":10000}' http://10.244.1.35:10001/addproduct/
{"id":1,"name":"ApplCar","quantity":3,"price":10000.0}[vagrant@kubemaster sprngbootmysqlapi]$

[vagrant@kubemaster sprngbootmysqlapi]$ curl http://10.244.1.35:10001/products
[{"id":1,"name":"ApplCar","quantity":3,"price":10000.0}][vagrant@kubemaster sprngbootmysqlapi]$
[vagrant@kubemaster sprngbootmysqlapi]$

 9) Create svc as node port.

[vagrant@kubemaster sprngbootmysqlapi]$ vi svc-spboot-api-mysql.yaml
[vagrant@kubemaster sprngbootmysqlapi]$ kubectl create -f svc-spboot-api-mysql.yaml
service/spboot-api-mysl-svc created
[vagrant@kubemaster sprngbootmysqlapi]$ cat svc-spboot-api-mysql.yaml
apiVersion: v1
kind: Service
metadata:
  name: spboot-api-mysl-svc
  labels:
    app: spboot-api-mysl
spec:
  ports:
    - port: 10001
  selector:
    app: spboot-api-mysl
  type: NodePort
  ports:
  - port: 10001
    #3000~32767
    nodePort: 32222
    targetPort: 10001
    protocol: TCP
    name: http

[vagrant@kubemaster sprngbootmysqlapi]$ kubectl get svc spboot-api-mysl-svc
NAME                  TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
spboot-api-mysl-svc   NodePort   10.107.183.192   <none>        10001:32222/TCP   34s

10) Node port test(my nodes are vagrant's each node ip: 10.1.0.3, 10.1.0.4)

   oyj@controller:~$ curl http://10.1.0.3:32222/products/
[{"id":1,"name":"ApplCar","quantity":3,"price":10000.0}]oyj@controller:~$
oyj@controller:~$ curl http://10.1.0.4:32222/products/
[{"id":1,"name":"ApplCar","quantity":3,"price":10000.0}]o

Conclusion)

Deployment api app(java) on k8s cluster using AWS ECR repo is kinda exceptional and fun. This cloud be very useful in development environment.

Besides, aws eks is somewhat expensive?, so local k8s cluster(not minikube) is good for testing I suppose.

Thanks for reading.