Background

I used a lot ELK service during my work in the telecommunication industry. It is used for both costumer service and developer to confirm complains from customers. Usually, I didn’t know how kibana shows the result from my development until I deployed and it risks alot since potentially can affect the production. So, by knowing how ELK work locally, we can see the kibana view and find the bug during the development activity.

Prerequisition

docker
docker compose

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.1
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # Adjust memory as needed
      - xpack.security.enabled=false # Disable security for simplicity in this example
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.1
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - xpack.security.enabled=false # Disable security for simplicity in this example
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.1
    container_name: filebeat
    user: root # Needed to access logs of other containers. Remove/adjust if possible
    volumes:
      - ./container/elk/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro # For docker logs
      - /var/run/docker.sock:/var/run/docker.sock:ro # Needed for docker logs input
      - filebeat-data:/usr/share/filebeat/data
      - ./container/elk/filebeat/logs:/diameter-logs:ro
    depends_on:
      - elasticsearch


configs:
  mqtt_conf:
    file: ./container/mosquitto.conf

volumes:
  elasticsearch-data:
  filebeat-data:

configuration file filebeat.yml

filebeat.inputs:
- type: tcp
  host: "localhost:5999"
  processors:
    - decode_json_fields:
        fields: ["field1", "field2"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: false


setup.kibana:
  host: "kibana:5601"

setup.dashboards.enabled: true
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  indices:
    - index: "filebeat-diameter-%{+yyyy.MM.dd}"
      when.contains:
        tags: "diameter"
    - index: "filebeat-docker-%{+yyyy.MM.dd}"
      when.not.contains:
        tags: "diameter"

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

ELK Fundamental

Background

Prerequisition