Kimuksung
Kimuksung 안녕하세요. 분산처리에 관심이 많은 생각하는 주니어 Data Enginner입니다.

kubernetes Airflow 구성하기1 with. kind,helm

kubernetes Airflow 구성하기1 with. kind,helm

Kubernetes Cluster = docker에 1master-3worker로 구성

airflow 환경 구성 및 namespace airflow에 별도 구성

Cluster를 구성하기 위해 아래와 같이 구성하여 준다.

MAC환경에서 진행하였다.


kind 설치

  • https://kind.sigs.k8s.io/docs/user/quick-start/

    1
2

      $ brew install kind
  $ kind version

helm 설치

  • https://helm.sh/ko/docs/intro/install/
  • https://github.com/helm/helm/releases
1
2
3
4

$ curl -O https://get.helm.sh/helm-v3.12.2-darwin-amd64.tar.gz
$ tar -zxvf helm-v3.12.2-darwin-amd64.tar.gz
$ mv darwin-amd64/helm /usr/local/bin/helm
$ helm version

kubectl 설치

  • 이전 작업하면 설치 완료
  • 필요하다면 검색하면 금방 나올것이다.


k8s cluster 생성

1

kind create cluster --name airflow-cluster --config kind-cluster.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# kind-cluster.yml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "node=worker_1"
    extraMounts:
      - hostPath: ./data
        containerPath: /tmp/data
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "node=worker_2"
    extraMounts:
      - hostPath: ./data
        containerPath: /tmp/data
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
          node-labels: "node=worker_3"
    extraMounts:
      - hostPath: ./data
        containerPath: /tmp/data


helm을 활용한 airflow 배포

1
2
3
4

$ helm repo add apache-airflow https://airflow.apache.org
$ helm repo update
$ helm search repo airflow
$ helm install airflow apache-airflow/airflow --namespace airflow --debug

1
2

# pod 진행 상황 확인
$ kubectl get pod -n airflow -w

1
2
3
4

# 서비스 확인 후 포팅
$ kubectl get service -n airflow
$ kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow --context kind-airflow-cluster
$ localhost:8080

configmap을 활용하여 airflow에 반영

  • Airflow 공식에서 yml을 다운받아준다.

  • revision은 yml이 변경된 횟수

    1
2
3
4
5

      $ helm show values apache-airflow/airflow > airflow_info.yaml
    
  $ helm ls -n airflow
  # 아래 값 수정 후 반영
  $ helm upgrade --install airflow apache-airflow/airflow -n airflow -f airflow_info.yaml --debug

  • executor

    1

      executor: "KubernetesExecutor"

  • extraEnvFrom

    1
2
3
4
5

      extraEnvFrom: |
    - secretRef:
        name: 'aws-info'
    - configMapRef:
        name: 'airflow-s3'

1
2
3
4
5
6

$ kubectl apply -f airflow_config.yml
$ kubectl apply -f airflow_secret.yml
$ kubectl describe cm airflow-s3 -n airflow
$ kubectl exec -it pod_name -n airflow bash
$ echo $AIRFLOW_VAR_MY_S3_BUCKET
$ echo $AIRFLOW_VAR_MY_S3_PREFIX

1
2
3
4
5
6
7
8
9

# airflow_config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: airflow
  name: airflow-s3
data:
  AIRFLOW_VAR_MY_S3_BUCKET: "test_s3_bucket_name"
  AIRFLOW_VAR_MY_S3_PREFIX: "test_s3_prefix_name"

1
2
3
4
5
6
7
8
9

# airflow_secret.yml
apiVersion: v1
kind: Secret
metadata:
  namespace: airflow
  name: aws-info
stringData:
  aws_credential_id: id_test
  aws_credential_key: key_test