ELK Kuberntes Conf

Description

整体架构

• Elasticsearch
  • 存储数据
• Kibana
  • 数据可视化
• Fluentd
  • Client收集数据
  • Server接收Client的数据，并转发到Elasticsearch
• Metricbeat
  • 收集性能指标数据
  • 替代方案：Heapster
• Kube-state-metrics
  • listens to the Kubernetes API server and generates metrics about the state of the objects

---

Elasticsearch

组成

• Master
  • intended for clustering management only, no data, no HTTP API
• Client
  • intended for document pre-processing during ingestion
• Data
  • intended for client usage and data

Fluentd

配置文件

• Server

# Takes the messages sent over TCP
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match **>
  @id fluentd-server
  @type elasticsearch
  @log_level info
  include_tag_key true
  host elasticsearch
  port 9200
  logstash_format true
  index_name fluentd
  type_name fluentd

  <buffer>
    @type file
    path /var/log/fluentd/kubernetes.system.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_thread_count 2
    flush_interval 5s
    retry_forever
    retry_max_interval 30
    chunk_limit_size 2M
    queue_limit_length 4
    overflow_action block
  </buffer>
</match>

• Client

containers.input.conf: |-
<source>
  @id fluentd-containers.log
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag raw.kubernetes.*
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json          time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
  @id raw.kubernetes
  @type detect_exceptions
  remove_tag_prefix raw
  message log
  stream stream
  multiline_flush_interval 5
  max_bytes 500000
  max_lines 1000
</match>

output.conf: |-
# Enriches records with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match **>
  @type forward
  time_format %Y-%m-%dT%H:%M:%S.%N%:z
    <server>
      host fluentd-service
      port 24224
    </server>
  flush_interval 1s
</match>

---

Metricbeat

配置文件

注意：使用该配置文件，Kibana可以直接开启Kuberntes Dashboard（默认配置文件无法开启）

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  #  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: true

    # To enable hints based autodiscover uncomment this:
    #metricbeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      host: ${NODE_NAME}
    #      hints.enabled: true

    processors:
      - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
    setup.kibana:
      host: kibana-service
      setup.kibana.protocol: "http"
    setup.dashboards:
      enabled: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  #  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10255"]
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        # Uncomment this to get k8s events:
        - event
      period: 10s
      host: ${NODE_NAME}
      hosts: ["kube-state-metrics:8080"]

Refer to

https://github.com/pires/kubernetes-elasticsearch-cluster

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch