쿠버네티스 테스트 환경에 CI/CD 구성(VScode → jenkins → Github → ArgoCD → Kubernetes)

hippo 데브옵스 2022. 7. 30. 18:12

2022. 7. 30. 18:12

1. github에 cdn_vtx_nginx Repository 생성

github의 Repository 생성

VS Code의 코드를 통해 github에 등록하기 위해 기초 적업 진행 → 향후 add, commit, push를 통해 쉽게 github에 동기화 가능

$ echo "# cdn_vtx_nginx" >> README.md
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin https://github.com/HippoMans/cdn_vtx_nginx.git
$ git push -u origin main

Jenkins에서 자동으로 Repository를 일기 위해서 Webhook 설정 →Payload URL의 설정은 Jenkins IP:12080

2. 이미지를 push, pull할 Docker Private Repository에 cdn_vtx_nginx Repository 생성

Docker Private Repository는 Harbor Private Repository를 사용

3. Jenkins Pipeline 생성

Pipeline 이름 → cdn_vtx_nginx
GitHub Project url → https://github.com/HippoMans/cdn_vtx_nginx/
GitHub hook trigger for GITScm polling → enable
Pipeline script from SCM 설정
1. Repository URL → https://github.com/HippoMans/cdn_vtx_nginx
2. Credentials → Github과 연결할 수 있도록 설정
3. Branch Specifier → /main
4. Script Path → Jenkinsfile

4. ArgoCD에 cdn_vtx_nginx APP 생성

이름 → cdn-vtx-nginx
SYNC POLICY → Automatic
Repository URL → https://github.com/HippoMans/cdn_vtx_nginx.git
Path → .(dot)
Cluster URL → https://kubernetes.default.svc (쿠버네티스 내부 URL)
namespace → cdn-vtc-nginx

5. Jenkinsfile 생성 내용

Jenkins Pipeline 내용 → Github Repository에 Jenkisfile으로 업로드

$ vi Jenkinsfile
podTemplate(label: 'docker-build',
  containers: [
    containerTemplate(
      name: 'git',
      image: 'alpine/git',
      command: 'cat',
      ttyEnabled: true
    ),
    containerTemplate(
      name: 'docker',
      image: 'docker',
      command: 'cat',
      ttyEnabled: true
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
  ]
) {
    node('docker-build') {
        def dockerHubCred = "private-docker-registry-ID"
        def appImage

        stage('Start'){
            slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Start -> URL : ${env.BUILD_URL}")
        }

        stage('Checkout'){
            container('git'){
                try{
                    checkout scm
                    slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Checkout Success.")
                }catch(e){
                    slackSend(Channel: '#hippo_cicd', color: '#ff0000', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Checkout Failed.")
                    throw e
                }
            }
        }

        stage('Build'){
            try{
                container('docker'){
                    script {
                        appImage = docker.build("cdn_vtx_nginx/nginx")
                    }
                    slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Build Success.")
                }
            }catch(e){
                slackSend(Channel: '#hippo_cicd', color: '#ff0000', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Build Failed.")
                throw e
            }
        }

        stage('Test'){
            container('docker'){
                try {
                    script {
                        appImage.inside {
                            sh 'nginx -t'
                        }
                        slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Test Success.")
                    }
                }catch(e){
                    slackSend(Channel: '#hippo_cicd', color: '#ff0000', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Test Failed.")
                    throw e
                }
            }
        }

        stage('Push'){
            container('docker'){
                try {
                    script {
                        docker.withRegistry('https://www.hippo.com', dockerHubCred){
                        appImage.push("${env.BUILD_NUMBER}")
                        appImage.push("latest")
                        }
                    slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Push Success.")
                    }   
                }catch(e){
                    slackSend(Channel: '#hippo_cicd', color: '#ff0000', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Push Failed.")
                    throw e
                }
            }
        }
        stage('Complete') {
            sh "echo 'Checkout & Build & Test & Push is finished'"
            slackSend(Channel: '#hippo_cicd', color: '#000dff', message: "${env.JOB_NAME}[#${env.BUILD_NUMBER}] Pipeline Complete.")
        }
    }   
}

6. Dockerfile 생성 내용

생성된 Dockerfile 내용 → Github Repository에 Dockerfile으로 업로드

$ vi Dockerfile
FROM centos:7

RUN yum update -y && \
    yum install -y GeoIP \
             wget \
             nginx-module-geoip \
             perl-ExtUtils-Embed \
             geoip-devel \
             gperftools-devel \
             gcc \
             g++ \
             cpp \
             gcc-c++ \
             openssl \
             openssl-devel \
             gd \
             gd-devel \
             libxml2-devel \
             bzip2-devel \
             curl-devel \
             libicu-devel \
             libmcrypt \
             libmcrypt-devel \
             openldap \
             openldap-devel \
             git \
             libxslt-devel

WORKDIR /root

RUN wget http://nginx.org/download/nginx-1.20.1.tar.gz && \
    tar -C /root/ -xzvf nginx-1.20.1.tar.gz

RUN git clone git://github.com/vozlt/nginx-module-vts.git

WORKDIR /root/nginx-1.20.1

RUN  ./configure --prefix=/usr/share/nginx \
             --sbin-path=/usr/sbin/nginx \
             --modules-path=/usr/lib64/nginx/modules \
             --conf-path=/etc/nginx/nginx.conf \
             --error-log-path=/var/log/nginx/error.log \
             --http-log-path=/var/log/nginx/access.log \
             --http-client-body-temp-path=/var/lib/nginx/tmp/client_body \
             --http-proxy-temp-path=/var/lib/nginx/tmp/proxy \
             --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi \
             --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi \
             --http-scgi-temp-path=/var/lib/nginx/tmp/scgi \
             --pid-path=/run/nginx.pid \
             --lock-path=/run/lock/subsys/nginx \
             --user=nginx \
             --group=nginx \
             --with-compat \
             --with-debug \
             --with-file-aio \
             --with-ipv6 \
             --with-google_perftools_module \
             --with-http_addition_module \
             --with-http_auth_request_module \
             --with-http_dav_module \
             --with-http_degradation_module \
             --with-http_flv_module \
             --with-http_gunzip_module \
             --with-http_gzip_static_module \
             --with-http_image_filter_module=dynamic \
             --with-http_mp4_module \
             --with-http_perl_module=dynamic \
             --with-http_random_index_module \
             --with-http_realip_module \
             --with-http_secure_link_module \
             --with-http_image_filter_module \
             --with-http_slice_module \
             --with-http_ssl_module \
             --with-http_stub_status_module \
             --with-http_sub_module \
             --with-http_v2_module \
             --with-http_xslt_module=dynamic \
             --with-mail=dynamic \
             --with-mail_ssl_module \
             --with-pcre \
             --with-pcre-jit \
             --with-stream=dynamic \
             --with-stream_ssl_module \
             --with-stream_ssl_preread_module \
             --with-threads \
             --with-http_xslt_module=dynamic \
             --with-http_image_filter_module=dynamic \
             --with-http_geoip_module \
             --with-http_perl_module=dynamic \
             --with-mail=dynamic \
             --add-module=/root/nginx-module-vts/ \
             --with-stream=dynamic

RUN make && make install

WORKDIR /root
RUN useradd -r nginx
RUN mkdir -p /var/lib/nginx/tmp/client_body
RUN mkdir -p /etc/nginx/sites-available
RUN mkdir -p /var/cache/nginx/temp/

ADD http://20.10.195.172/nginx_config/nginx_cache_default.conf /etc/nginx/nginx.conf  # nginx 설정 파일 (제공 X)
ADD http://20.10.195.172/nginx_config/nginx_sites_cache.conf /etc/nginx/sites-available/cache.conf   # vhost 파일 (제공 X)
ADD http://20.10.195.172/nginx_config/cache_params /etc/nginx/cache_params  # cache_params 내용 (제공 X)

VOLUME ["/cache1/", "/cache1/"]
VOLUME ["/cache2/", "/cache2/"]

EXPOSE 80
EXPOSE 443

CMD ["nginx", "-g", "daemon off;"]

7. 쿠버네티스 배포 yaml 파일

쿠버네티스에 nginx 배포할 yaml 파일 내용 → Github Repository에 업로드되면 ArgoCD에서 해당 내용 배포

$ vi cdn_vtx_nginx.yml
apiVersion: v1
kind: Namespace
metadata:
  name: cdn-vtx-nginx
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cdn-vtx-nginx-deployment
  namespace: cdn-vtx-nginx
spec:
  selector:
    matchLabels:
      nginx: cdn-vtx-nginx
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        nginx: cdn-vtx-nginx
    spec:
      containers:
      - name: container
        image: www.hippo.com/cdn_vtx_nginx/nginx:latest    ## harbor private registry의 Repository 위치
        ports:
        - containerPort: 80
        volumeMounts:
         - name: cache1
           mountPath: /cache1
         - name: cache2
           mountPath: /cache2
      nodeSelector:
        kubernetes.io/hostname: [worker node의 hostname]
      volumes:
      - name: cache1
        hostPath:
          path: /cache1
          type: Directory
      - name: cache2
        hostPath:
          path: /cache2
          type: Directory
      imagePullSecrets:
      - name: harbor-secret
---

apiVersion: v1
kind: Service
metadata:
  name: cdn-vtx-nginx-loadbalancer
  namespace: cdn-vtx-nginx
spec:
  selector:
    nginx: cdn-vtx-nginx
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
---

apiVersion: v1
kind: Service
metadata:
  name: cdn-vtx-nginx-service
  namespace: cdn-vtx-nginx
spec:
  selector:
    nginx: cdn-vtx-nginx
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
---

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL3Rlc3RoaXBwby5teXNrY2RuLmNvbSI6eyJ1c2VybmFtZSI6ImFkbWluIiwicGFzc3dvcmQiOiJIYXJib3JBZG1pbjEyMzQiLCJlbWFpbCI6InRlQGdtYWlsLmNvbSIsImF1dGgiOiJZV1J0YVc0NlNHRnlZbTl5UVdSdGFXNHhNak0wIn19fQ==
kind: Secret
metadata:
  name: harbor-secret
  namespace: cdn-vtx-nginx
type: kubernetes.io/dockerconfigjson