Kubernetes에 CVAT 설치하기 - Helm으로 CVAT 설치

Helm으로 CVAT 설치

Kubernetes에 CVAT 설치하기

Yolo 모델 학습에 도전하였던 포스팅에서 Auto Labeling Tool로 CVAT을 사용하였습니다.

[Vision] 엔지니어의 Yolo 도전기(1) - CVAT을 통한 Auto Labeling

해당 포스팅에서는 CVAT을 Docker Compose로 구축하였습니다.

CVAT이 Docker Image를 제공해 주기 때문에 쉽게 Docker Compose로 설치가 가능하였습니다.

요즘 container 환경은 Kubernetes Cluster에서 구성됩니다.

이번 포스팅에서 Docker Compose가 아닌 CVAT을 Kubernetes 환경에 구축하는 것을 정리하였습니다.

설치 준비

Helm을 통해 Kubernetes 환경에 CVAT을 설치를 합니다.

CVAT 공식 Helm은 Helm Chart Repository에 없습니다.

대신 CVAT Github에 Helm Chart가 있어 해당 Github을 clone합니다.

$ git clone https://github.com/opencv/cvat.git
$ ls ./cvat/helm-chart/
analytics  Chart.lock  charts  Chart.yaml  nuclio_func_common_files  README.md  templates values.yaml

근데 여기서 바로 Helm Chart로 install 하지 않습니다.

왜냐하면 다른 dependency Helm Chart를 사용하기 때문에 dependency build를 진행합니다.

dependecny는 Chart.lock 파일을 통해서 확인 가능합니다.

# helm dependency 확인
$ cat Chart.lock 
dependencies:
- name: redis
  repository: https://charts.bitnami.com/bitnami
  version: 17.3.18
- name: postgresql
  repository: https://charts.bitnami.com/bitnami
  version: 12.1.15
- name: nuclio
  repository: https://nuclio.github.io/nuclio/charts
  version: 0.12.1
- name: vector
  repository: https://helm.vector.dev
  version: 0.19.2
- name: clickhouse
  repository: https://charts.bitnami.com/bitnami
  version: 3.0.7
- name: grafana
  repository: https://grafana.github.io/helm-charts
  version: 6.51.5
- name: traefik
  repository: https://helm.traefik.io/traefik
  version: 10.24.0
digest: sha256:d10e65dd88c6a413c4772c0ac065dd93fab0cd2d4b5e85e8e3dc4de2c0e06a44
generated: "2023-05-04T14:09:48.289956122+09:00"

$ helm dependency build

walk.go:74: found symbolic link in path: /home/ubuntu/1_k8s_deploy/cvat/helm-chart/analytics resolves to /home/ubuntu/1_k8s_deploy/cvat/components/analytics. Contents of linked file included and used
walk.go:74: found symbolic link in path: /home/ubuntu/1_k8s_deploy/cvat/helm-chart/analytics resolves to /home/ubuntu/1_k8s_deploy/cvat/components/analytics. Contents of linked file included and used
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://helm.vector.dev" chart repository
...Successfully got an update from the "https://nuclio.github.io/nuclio/charts" chart repository
...Successfully got an update from the "https://grafana.github.io/helm-charts" chart repository
...Successfully got an update from the "https://helm.traefik.io/traefik" chart repository
...Successfully got an update from the "https://charts.bitnami.com/bitnami" chart repository
...Successfully got an update from the "https://charts.bitnami.com/bitnami" chart repository
...Successfully got an update from the "https://charts.bitnami.com/bitnami" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "openebs" chart repository
...Successfully got an update from the "argo-cd" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 7 charts
Downloading redis from repo https://charts.bitnami.com/bitnami
Downloading postgresql from repo https://charts.bitnami.com/bitnami
Downloading nuclio from repo https://nuclio.github.io/nuclio/charts
Downloading vector from repo https://helm.vector.dev
Downloading clickhouse from repo https://charts.bitnami.com/bitnami
Downloading grafana from repo https://grafana.github.io/helm-charts
Downloading traefik from repo https://helm.traefik.io/traefik
Deleting outdated charts

꽤 많은 depency를 가지고 있는 것을 확인할 수 있습니다.

• Redis -> CVAT Job Queue
• PostgreSQL -> CVAT Back-end DB
• Nuclio -> Model Deploy
• 분석용 Tool
  • Vector -> Data pipeline
  • Clickhouse -> 오픈 소스 OLAP DBMS
  • Grafana -> Dashboard
• traefik -> Edge Router, Reverse Proxy 역할(Kubernetes Ingress)

CVAT 공식 Github에 2020년 10월에 올라와 있는 Architecture 이미지입니다.

참고하시면 좋을 거 같습니다.

출처 : https://github.com/opencv/cvat/issues/2372

이번 포스팅에서는 CVAT 분석용으로 쓰이는 Vector, Clickhouse, Grafana는 제외하고 install 하겠습니다.

$ cat Chart.yaml

dependencies:
  - name: redis
    version: "17.3.*"
    repository: https://charts.bitnami.com/bitnami
    condition: redis.enabled

  - name: postgresql
    version: "12.1.*"
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled

  - name: nuclio
    version: 0.12.1
    repository: https://nuclio.github.io/nuclio/charts
    condition: nuclio.enabled

  - name: vector
    version: "0.19.*"
    repository: https://helm.vector.dev
    condition: analytics.enabled

  - name: clickhouse
    version: "3.0.*"
    repository: https://charts.bitnami.com/bitnami
    condition: analytics.enabled

  - name: grafana
    version: "6.51.*"
    repository: https://grafana.github.io/helm-charts
    condition: analytics.enabled

  - name: traefik
    version: 10.24.0
    repository: https://helm.traefik.io/traefik
    condition: ingress.enabled

Chart.yaml을 확인해 보면 analytics을 여부에 따라 install 되는 것을 확인할 수 있습니다.

values-override.yaml 작성

이제 values-override.yaml 파일을 작성하겠습니다.

helm에서는 values.yaml 파일 내 정의된 값들을 기반으로 template 내 정의된 Object을 생성합니다.

원본 values.yaml을 수정 안 하고 수정할 값들만 따로 파일로 정의하면 overriding이 됩니다.

$ vi values-override.yaml 

ingress:
  hosts:
    - host: k8s.sample.com
      paths:
        - path: /api
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /admin
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /static
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /django-rq
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /git
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /opencv
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /profiler
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path : /
          pathType: "Prefix"
          service:
            name: frontend-service
            port: 80
nuclio:
  enabled: true
  dashboard:
    containerBuilderKind: "kaniko"
    nodePort: 32070
  registry:
    secretName: registry-credential
    pushPullUrl: repo.coxspace.biz
analytics:
  enabled: false
traefik:
  logs:
    general:
      level: INFO
  ports:
    web:
      nodePort: 32080
    websecure:
      nodePort: 32443
  service:
    type: NodePort
    externalIPs:
      - 15.164.69.11

 

ingress 설정

ingress 부분은 traefik과 관련되어 있습니다.

traefik이 Edge Router로 path에 따라 front, backend 서비스로 라우팅 해줍니다.

CVAT을 사용하는 유저의 최접점이 traefik입니다.

먼저 host명을 세팅합니다.(host명은 예시, host명을 안 넣으면 ip로 접근 가능)

paths은 굳이 바꿀 필요 없이 그대로 넣어줍니다.

ingress:
  hosts:
    - host: k8s.sample.com
      paths:
        - path: /api
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /admin
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /static
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /django-rq
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /git
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /opencv
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path: /profiler
          pathType: "Prefix"
          service:
            name: backend-service
            port: 8080
        - path : /
          pathType: "Prefix"
          service:
            name: frontend-service
            port: 80

Nuclio 설정

Nuclio는 Serverless function로 사용하는 오픈 소스 Tool입니다.

CVAT에서는 Nuclio를 통해 모델을 배포하여 해당 모델을 기반으로 Auto Labeling을 진행합니다.

설정하기에 앞서 Nuclio container 기반으로 동작합니다.

Nuclio function을 배포할 Docker Registry가 있어야 합니다.

Local이나 Nexus Repository가 필요합니다.

Nexus Repsository 구성은 따로 포스팅하도록 하겠습니다.

그래서 접근에 필요한 secret을 생성합니다.

$ kubectl create secret docker-registry registry-credential \
    --namespace cvat \
    --docker-username ${username} \
    --docker-password ${password} \
    --docker-server repo.example.com \
    --docker-email ignored@nuclio.io

그리고 Container Runtime을 Kubernetes container Runtime에 맞춰줍니다.

제가 테스트로 쓰고 있는 Kubernetes는 container이기 때문에 kaniko로 지정합니다.

Nuclio WEB UI를 제공하기 때문에 NodePort로 expose 합니다.

nuclio:
  enabled: true
  dashboard:
    containerBuilderKind: "kaniko"
    nodePort: 32070
  registry:
    secretName: registry-credential
    pushPullUrl: repo.example.com

analytics 설정

앞서 이야기하였지만 이번 포스팅에서는 분석용 Tool은 Install 하지 않습니다.

그래서 Vector, Clickhouse, Grafana는 구성되지 않습니다.

traefik 설정

CVAT 유저는 traefik을 통해서 접근합니다.

해당 Service 설정을 잘해줘야 합니다.

HTTP, HTTPS로 expose 할 Port, externalIP를 설정합니다.

추가적으로 더 세팅하길 원한다면 traefik Helm Chart values.yaml를 참고하여 값을 넣어줍니다.

traefik:
  logs:
    general:
      level: INFO
  ports:
    web:
      nodePort: 32080
    websecure:
      nodePort: 32443
  service:
    type: NodePort
    externalIPs:
      - ***.***.***.***

Helm install

Helm Chart value 세팅이 끝나면 이제 helm install를 합니다.

$ helm install cvat --namespace cvat --create-namespace -f values-override.yaml ./

$ k get all -n cvat
NAME                                                  READY   STATUS    RESTARTS        AGE
pod/cvat-backend-server-84d8fbcb4d-t9hvc              1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-utils-ff99d54c4-t89m9                1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-annotation-575fd9fbb8-q75tl   1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-export-59c69cfcd7-2l2mf       1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-export-59c69cfcd7-sbzwv       1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-import-6554bfdf5b-7mwlj       1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-import-6554bfdf5b-sk84f       1/1     Running   1 (7h57m ago)   24h
pod/cvat-backend-worker-webhooks-59cf9bc7b4-gflws     1/1     Running   1 (7h57m ago)   25h
pod/cvat-frontend-5944b9dd89-ggsnh                    1/1     Running   1 (7h57m ago)   25h
pod/cvat-nuclio-controller-6c4bd9fb66-g8b2j           1/1     Running   1 (7h57m ago)   24h
pod/cvat-nuclio-dashboard-5487cdd6c7-zfcmb            1/1     Running   1 (7h57m ago)   24h
pod/cvat-opa-66d9c4746d-mthcl                         1/1     Running   1 (7h57m ago)   25h
pod/cvat-postgresql-0                                 1/1     Running   1 (7h57m ago)   25h
pod/cvat-redis-master-0                               1/1     Running   1 (7h57m ago)   25h
pod/cvat-redis-replicas-0                             1/1     Running   1 (7h57m ago)   25h
pod/cvat-redis-replicas-1                             1/1     Running   1 (7h57m ago)   25h
pod/cvat-redis-replicas-2                             1/1     Running   1 (7h57m ago)   25h
pod/cvat-traefik-6d555df4bc-m9cbh                     1/1     Running   1 (7h57m ago)   25h

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
service/cvat-backend-service    ClusterIP   10.96.214.64     <none>         8080/TCP                     25h
service/cvat-frontend-service   ClusterIP   10.109.166.120   <none>         80/TCP                       25h
service/cvat-nuclio-dashboard   NodePort    10.108.61.147    <none>         8070:32070/TCP               24h
service/cvat-postgresql         ClusterIP   10.101.8.124     <none>         5432/TCP                     25h
service/cvat-postgresql-hl      ClusterIP   None             <none>         5432/TCP                     25h
service/cvat-redis-headless     ClusterIP   None             <none>         6379/TCP                     25h
service/cvat-redis-master       ClusterIP   10.99.163.211    <none>         6379/TCP                     25h
service/cvat-redis-replicas     ClusterIP   10.96.32.225     <none>         6379/TCP                     25h
service/cvat-traefik            NodePort    10.102.233.73    15.164.69.11   80:32080/TCP,443:32443/TCP   25h
service/opa                     ClusterIP   10.100.251.124   <none>         8181/TCP                     25h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cvat-backend-server              1/1     1            1           25h
deployment.apps/cvat-backend-utils               1/1     1            1           25h
deployment.apps/cvat-backend-worker-annotation   1/1     1            1           25h
deployment.apps/cvat-backend-worker-export       2/2     2            2           25h
deployment.apps/cvat-backend-worker-import       2/2     2            2           25h
deployment.apps/cvat-backend-worker-webhooks     1/1     1            1           25h
deployment.apps/cvat-frontend                    1/1     1            1           25h
deployment.apps/cvat-nuclio-controller           1/1     1            1           24h
deployment.apps/cvat-nuclio-dashboard            1/1     1            1           24h
deployment.apps/cvat-opa                         1/1     1            1           25h
deployment.apps/cvat-traefik                     1/1     1            1           25h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/cvat-backend-server-84d8fbcb4d              1         1         1       24h
replicaset.apps/cvat-backend-server-84dd7cb9d9              0         0         0       25h
replicaset.apps/cvat-backend-utils-7468fd8b99               0         0         0       25h
replicaset.apps/cvat-backend-utils-ff99d54c4                1         1         1       24h
replicaset.apps/cvat-backend-worker-annotation-575fd9fbb8   1         1         1       24h
replicaset.apps/cvat-backend-worker-annotation-7c45cfb875   0         0         0       25h
replicaset.apps/cvat-backend-worker-export-59c69cfcd7       2         2         2       24h
replicaset.apps/cvat-backend-worker-export-7bb55998bd       0         0         0       25h
replicaset.apps/cvat-backend-worker-import-6554bfdf5b       2         2         2       24h
replicaset.apps/cvat-backend-worker-import-7fb747bc7f       0         0         0       25h
replicaset.apps/cvat-backend-worker-webhooks-59cf9bc7b4     1         1         1       25h
replicaset.apps/cvat-frontend-5944b9dd89                    1         1         1       25h
replicaset.apps/cvat-nuclio-controller-6c4bd9fb66           1         1         1       24h
replicaset.apps/cvat-nuclio-dashboard-5487cdd6c7            1         1         1       24h
replicaset.apps/cvat-nuclio-dashboard-5cf4f4d454            0         0         0       24h
replicaset.apps/cvat-nuclio-dashboard-6457df4969            0         0         0       24h
replicaset.apps/cvat-nuclio-dashboard-7786dfb57b            0         0         0       24h
replicaset.apps/cvat-opa-66d9c4746d                         1         1         1       25h
replicaset.apps/cvat-traefik-6d555df4bc                     1         1         1       25h

NAME                                   READY   AGE
statefulset.apps/cvat-postgresql       1/1     25h
statefulset.apps/cvat-redis-master     1/1     25h
statefulset.apps/cvat-redis-replicas   3/3     25h

지정한 Host명과 traefik HTTP Port로 접근하면 CVAT UI로 접근 가능한 것을 알 수 있습니다.

CVAT UI

정리

CVAT을 Helm을 통해 Kubernetes 환경에 구축하였습니다.

이번 경험을 통해 CVAT에서 많은 Tool이 사용되고 있는 것을 확인할 수 있었습니다.

처음 보는 것도 많았기 때문에 CVAT Architecture을 따로 포스팅하며 각 Tool에 대해 살펴보려 합니다.