Ansible를 통한 Kafka 설치

Ansible를 통한 Kafka 설치

Ansible를 통한 Kafka 설치

Kafka는 Cluster로 구성되기 때문에 Server마다 같은 작업을 반복해야 한다.

Infra 자동화 구성 Tool인 Ansible을 통해 편하게 설치하는 작업을 해보려 한다.

Ansible은 laC(Infrastructure as Code) 개념이 도입된 Tool로 Code를 통해 Infra 세팅을 도와준다.

AWS EC2 3개의 Instance를 세팅하여 Kafka Cluster를 구성해보았다.

Ansible 설치

먼저 첫번째 Server에 Ansible을 설치하였다.

Ansible은 python기반이라 pip를 통해 설치가 가능하다.

Ansible 버전별 필요한 python 버전은 아래 표에 정리하였다.

Ansible Version	Required Python Version
2.11	2.7 / 3.5 ~ 3.9(3.8 이상 추천)
2.12	3.8 ~ 3.10
2.13	3.8 ~ 3.10
2.14	3.9 ~ 3.11

$ python3 -m pip install ansible

pip로 Ansible을 설치했으면 Ansible command를 통해 config 파일을 생성한다.

이번 예제에서는 Ansible log파일 path만 설정해주었다.

$ ansible-config init --disabled -t all > ansible.cfg

$ vi ansible.cfg
[defaults]
log_path = /home/ec2-user/playbooks/ansible.log

그리고 hosts 파일을 설정하여 Ansible를 적용할 server list와 설정값을 세팅하였다.

이번 예제에서는 AWS EC2에 설치를 진행하고 있다.

그래서 Private IPv4 주소를 입력해준다.

pem파일 path를 넣어 Ansible에서 pem파일을 통해 각 server에 접속할 수 있도록 한다.

$ vi hosts

[kafkaservers]
{server1.ip}
{server2.ip}
{server3.ip}

[kafkaservers:vars]
ansible_ssh_private_key_file= ~/kafka-node.pem

playbook 세팅

Ansible은 playbook이라는 yaml에 설정한 대로 Infra를 세팅 및 provisioning을 한다.

여기에서는 2가지 파일(install.yaml, kafka-config.yaml)로 나누어서 세팅하였다.

1. 설치 전 필요한 세팅과 Kafka 설치
2. Kafka Configuration

playbook 2가지 파일의 상세 내용은 아래에서 확인 가능하다.

playbook 세팅까지 끝나면 아래 command로 Ansible은 세팅한다.

$ ansible-playbook -i hosts install.yaml kafka-config.yaml

install.yaml

- name: Kafka Zookeeper Install
  hosts: kafkaservers
  become: yes
  vars:
    - installation_dir: /opt/kafka
  tasks:
    - name: Install JDK 11
      yum:
        name: java-11-amazon-corretto.x86_64
        state: present
    - name: Create a group
      group:
        name: kafka
        state: present
    - name: Create a user
      user:
        name: kafka
        state: present
        group: kafka
    - name: Create a Directory /opt/kafka
      become: yes
      file:
        path: "{{installation_dir}}" 
        state: directory
        mode: 0755
        owner: kafka
        group: kafka
    - name: Download and Unarchive Kafka
      become: yes
      become_user: kafka
      unarchive:
        src: https://archive.apache.org/dist/kafka/3.1.1/kafka_2.13-3.1.1.tgz
        dest: "{{installation_dir}}" 
        mode: 0755
        remote_src: yes
    - name: Move all the files to parent Directory
      shell:
        mv {{installation_dir}}/kafka_*/* {{installation_dir}}/.
    - name: Update the Java Heap Size for Kafka
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/bin/kafka-server-start.sh" 
        regexp: 'export KAFKA_HEAP_OPTS=(".+")'
        replace: 'export KAFKA_HEAP_OPTS="-Xmx520M -Xms520M"'
        backup: yes
    - name: Create a Service file for ZooKeeper with Copy module
      copy:
        dest: /etc/systemd/system/zookeeper.service
        content: |
          [Unit]
          Requires=network.target remote-fs.target
          After=network.target remote-fs.target
          [Service]
          Type=simple
          User=kafka
          ExecStart={{installation_dir}}/bin/zookeeper-server-start.sh {{installation_dir}}/config/zookeeper.properties
          ExecStop={{installation_dir}}/bin/zookeeper-server-stop.sh
          Restart=on-abnormal
          [Install]
          WantedBy=multi-user.target
        mode: 0755

    - name: Create a Service file for Kafka with Copy module
      copy:
        dest: /etc/systemd/system/kafka.service
        content: |
          [Unit]
          Requires=zookeeper.service
          After=zookeeper.service
          [Service]
          Type=simple
          User=kafka
          ExecStart=/bin/sh -c '{{installation_dir}}/bin/kafka-server-start.sh {{installation_dir}}/config/server.properties'
          ExecStop={{installation_dir}}/bin/kafka-server-stop.sh
          Restart=on-abnormal
          [Install]
          WantedBy=multi-user.target
        mode: 0755
    - name: Start Services
      tags: startservices
      become: yes
      systemd:
        name: '{{item}}'
        state: started
        enabled: yes
      with_items:
        - "kafka" 
        - "zookeeper"

Kafka-config.yaml

- name: Kafka update config
  hosts: kafkaservers
  become: yes
  vars:
    seq: "{{ groups.kafkaservers.index(inventory_hostname) + 1 }}" 
    zookeepers: "{{ groups.kafkaservers | join(':2181,') }}:2181" 
    installation_dir: /opt/kafka
  tasks:
    - name: Update Broker ID 
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/config/server.properties" 
        regexp: 'broker.id=0'
        replace: 'broker.id={{ seq }}'
        backup: yes
    - name: Update listeners
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/config/server.properties" 
        regexp: '#listeners=PLAINTEXT://:9092'
        replace: 'listeners=PLAINTEXT://:9092'
        backup: yes
    - name: Update advertiesd listeners
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/config/server.properties" 
        regexp: '#advertised.listeners.+'
        replace: 'advertised.listeners=PLAINTEXT://{{ ansible_eth0.ipv4.address }}:9092'
        backup: yes
    - name: Update zookeeper server in kafka config
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/config/server.properties" 
        regexp: 'zookeeper.connect=.+'
        replace: 'zookeeper.connect={{ zookeepers }}'
        backup: yes      
    - name: Update Zookeeper config
      become: yes
      become_user: kafka
      blockinfile:
        path: "{{installation_dir}}/config/zookeeper.properties" 
        marker: "" 
        block: |
          initLimit=10
          syncLimit=5
    - name: Update Zookeeper Servers config
      become: yes
      become_user: kafka
      blockinfile:
        path: "{{installation_dir}}/config/zookeeper.properties" 
        marker: "" 
        block: |
          server.{{ item.0 + 1 }}={{ item.1 }}:2888:3888
      with_indexed_items: "{{ groups.kafkaservers }}" 
    - name: Make Zookeeper MyId file
      become: yes
      become_user: kafka
      shell:
        mkdir -p /tmp/zookeeper && echo {{ seq }} > /tmp/zookeeper/myid

[참고사이트]

Getting started with Ansible — Ansible Documentation