登录 注册

fluent-bit debug 调试，采集 kubernetes podIP

本贴最后更新于 1123 天前，其中的信息可能已经时移俗易

有时候调试 fluent-bit 的配置，达到想要的输出效果，并不是件简单的事情，以下通过 debug 镜像调试 fluent-bit 采集 kubernetes Pod 的 IP。

fluent-bit 官方文档给出了用于调试的镜像：
https://docs.fluentbit.io/manual/installation/docker

dockerhub 仓库链接为：
https://hub.docker.com/r/fluent/fluent-bit/

部署 fluent-bit-debug

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: fluent-bit-debug
  name: fluent-bit-debug
  namespace: kubesphere-logging-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit-debug
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: fluent-bit-debug
      name: fluent-bit-debug
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        command:
        - /usr/local/bin/sh
        - -c
        - sleep 9999    
        image: fluent/fluent-bit:1.6.9-debug
        imagePullPolicy: IfNotPresent
        name: fluent-bit
        ports:
        - containerPort: 2020
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/docker/containers
          name: varlibcontainers
          readOnly: true
        - mountPath: /fluent-bit/config
          name: config
          readOnly: true
        - mountPath: /var/log/
          name: varlogs
          readOnly: true
        - mountPath: /var/log/journal
          name: systemd
          readOnly: true
        - mountPath: /fluent-bit/tail
          name: positions
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: fluent-bit
      serviceAccountName: fluent-bit
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: varlibcontainers
      - name: config
        secret:
          defaultMode: 420
          secretName: fluent-bit-debug-config
      - hostPath:
          path: /var/log
          type: ""
        name: varlogs
      - hostPath:
          path: /var/log/journal
          type: ""
        name: systemd
      - emptyDir: {}
        name: positions

用到的 secret：fluent-bit-debug-config 如下，包含两个 key：
在这里插入图片描述

第一个 parsers.conf：为空即可；配置细节可以参考官方文档：
Configuration File

第二个 fluent-bit.conf 配置需根据情况配置，以下主要在不同场景，给出 fluent-bit.conf。也会涉及到 kubesphere、Filter CRD 一起使用。

fluent-bit 使用

exec 进入容器可以使用/fluent-bit/bin/fluent-bit 调试：

/fluent-bit/bin #  ./fluent-bit -h
Usage: fluent-bit [OPTION]

Available Options
  -b  --storage_path=PATH       specify a storage buffering path
  -c  --config=FILE     specify an optional configuration file
  -d, --daemon          run Fluent Bit in background mode
  -f, --flush=SECONDS   flush timeout in seconds (default: 5)
  -F  --filter=FILTER    set a filter
  -i, --input=INPUT     set an input
  -m, --match=MATCH     set plugin match, same as '-p match=abc'
  -o, --output=OUTPUT   set an output
  -p, --prop="A=B"      set plugin configuration property
  -R, --parser=FILE     specify a parser configuration file
  -e, --plugin=FILE     load an external plugin (shared lib)
  -l, --log_file=FILE   write log info to a file
  -t, --tag=TAG         set plugin tag, same as '-p tag=abc'
  -T, --sp-task=SQL     define a stream processor task
  -v, --verbose         increase logging verbosity (default: info)
  -H, --http            enable monitoring HTTP server
  -P, --port            set HTTP server TCP port (default: 2020)
  -s, --coro_stack_size Set coroutines stack size in bytes (default: 24576)
  -q, --quiet           quiet mode
  -S, --sosreport       support report for Enterprise customers
  -V, --version         show version number
  -h, --help            print this help

Inputs
  cpu                   CPU Usage
  mem                   Memory Usage
  thermal               Thermal
  kmsg                  Kernel Log Buffer
  proc                  Check Process health
  disk                  Diskstats
  systemd               Systemd (Journal) reader
  netif                 Network Interface Usage
  docker                Docker containers metrics
  docker_events         Docker events
  tail                  Tail files
  dummy                 Generate dummy data
  head                  Head Input
  health                Check TCP server health
  collectd              collectd input plugin
  statsd                StatsD input plugin
  serial                Serial input
  stdin                 Standard Input
  syslog                Syslog
  exec                  Exec Input
  tcp                   TCP
  mqtt                  MQTT, listen for Publish messages
  forward               Fluentd in-forward
  random                Random

Filters
  alter_size            Alter incoming chunk size
  aws                   Add AWS Metadata
  record_modifier       modify record
  throttle              Throttle messages using sliding window algorithm
  kubernetes            Filter to append Kubernetes metadata
  modify                modify records by applying rules
  nest                  nest events by specified field values
  parser                Parse events
  expect                Validate expected keys and values
  grep                  grep events by specified field values
  rewrite_tag           Rewrite records tags
  lua                   Lua Scripting Filter
  stdout                Filter events to STDOUT

Outputs
  azure                 Send events to Azure HTTP Event Collector
  azure_blob            Azure Blob Storage
  bigquery              Send events to BigQuery via streaming insert
  counter               Records counter
  datadog               Send events to DataDog HTTP Event Collector
  es                    Elasticsearch
  exit                  Exit after a number of flushes (test purposes)
  file                  Generate log file
  forward               Forward (Fluentd protocol)
  http                  HTTP Output
  influxdb              InfluxDB Time Series
  logdna                LogDNA
  loki                  Loki
  kafka                 Kafka
  kafka-rest            Kafka REST Proxy
  nats                  NATS Server
  nrlogs                New Relic
  null                  Throws away events
  plot                  Generate data file for GNU Plot
  pgsql                 PostgreSQL
  slack                 Send events to a Slack channel
  splunk                Send events to Splunk HTTP Event Collector
  stackdriver           Send events to Google Stackdriver Logging
  stdout                Prints events to STDOUT
  syslog                Syslog
  tcp                   TCP Output
  td                    Treasure Data
  flowcounter           FlowCounter
  gelf                  GELF Output
  cloudwatch_logs       Send logs to Amazon CloudWatch
  kinesis_firehose      Send logs to Amazon Kinesis Firehose
  s3                    Send to S3

Internal
 Event Loop  = epoll
 Build Flags =  FLB_HAVE_HTTP_CLIENT_DEBUG FLB_HAVE_PARSER FLB_HAVE_RECORD_ACCESSOR FLB_HAVE_STREAM_PROCESSOR JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_AWS FLB_HAVE_SIGNV4 FLB_HAVE_SQLDB FLB_HAVE_METRICS FLB_HAVE_HTTP_SERVER FLB_HAVE_SYSTEMD FLB_HAVE_FORK FLB_HAVE_TIMESPEC_GET FLB_HAVE_GMTOFF FLB_HAVE_UNIX_SOCKET FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_UTF8_ENCODER FLB_HAVE_LUAJIT FLB_HAVE_C_TLS FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY

简单配置文件

以下用一个简单的配置文件采集 calico-node-* pod 的日志：

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*_kube-system_calico-node-*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/bin/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    docker
    Tag    kube.*
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    true
[Output]
    Name    stdout
    Match_Regex    (?:kube|service)\.(.*)

fluent-bit-debug 容器内使用以下命令启动测试：

/fluent-bit/bin # ./fluent-bit -c /fluent-bit/config/fluent-bit.conf

可以看到 stdout 日志输出：

[0] kube.var.log.containers.calico-node-lp4lm_kube-system_calico-node-cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81.log: [1634870260.700108403, {"log"=>"{"log":"2021-10-22 02:37:40.699 [INFO][85] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface bond4: 172.24.248.50/30\n","stream":"stdout","time":"2021-10-22T02:37:40.700056471Z"}", "kubernetes"=>{"pod_name"=>"calico-node-lp4lm", "namespace_name"=>"kube-system", "pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd", "annotations"=>{"kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"}, "host"=>"node02", "container_name"=>"calico-node", "docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81", "container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55", "container_image"=>"calico/node:v3.19.1"}}]

截取重要部分，可以看到没经过处理采集到的 k8s 日志格式。

[
	{
		"kubernetes"=>{
			"pod_name"=>"calico-node-lp4lm",
			"namespace_name"=>"kube-system",
			"pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
			"annotations"=>{
				"kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"
			},
			"host"=>"node02",
			"container_name"=>"calico-node",
			"docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
			"container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
			"container_image"=>"calico/node:v3.19.1"
		}
	}
]

增加 nest Filter

将 kubernetes 块展开，并添加 kubernetes_前缀：

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_

这次测试输出，截取重要部分：

{
	"kubernetes_pod_name"=>"calico-node-lp4lm",
	"kubernetes_namespace_name"=>"kube-system",
	"kubernetes_pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
	"kubernetes_annotations"=>{
		"kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"
	},
	"kubernetes_host"=>"node02",
	"kubernetes_container_name"=>"calico-node",
	"kubernetes_docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
	"kubernetes_container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
	"kubernetes_container_image"=>"calico/node:v3.19.1"
}

移除掉 kubernetes_annotations 块

[Filter]
    Name    modify
    Match    kube.*
    Remove    kubernetes_annotations

移除掉 kubernetes_annotations 块中的某字段

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove    kubernetes_annotations_kubectl.kubernetes.io/restartedAt

或者用正则：

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove_regex    kubernetes_annotations_kubectl*

修改 kubernetes_annotations 块中 key 名称

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Rename    kubernetes_annotations_kubectl.kubernetes.io/restartedAt podIPs

修改之后：

[
	{
		"kubernetes_pod_name"=>"calico-node-lp4lm",
		"kubernetes_namespace_name"=>"kube-system",
		"kubernetes_pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
		"kubernetes_host"=>"node02",
		"kubernetes_container_name"=>"calico-node",
		"kubernetes_docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
		"kubernetes_container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
		"kubernetes_container_image"=>"calico/node:v3.19.1",
		"podIPs"=>"2021-10-20T23:00:27+08:00"
	}
]

结合 ks 配置采集 podIPs

结合 kubesphere Filter CR，配置采集 podIPs，并去掉其他不相关的 annotations。

因使用 calico 作为 CNI，所以在 pod annotations 中会被添加上 podIP 相关的注解。

需要保留注解中的某一个 key（cni.projectcalico.org/podIPs），移除掉其他 key，所以下面将要保留的 key 修改名称之后，移除掉整个 annotations。

kubernetes Filter CR 配置如下：

apiVersion: logging.kubesphere.io/v1alpha2
kind: Filter
metadata:
  labels:
    logging.kubesphere.io/component: logging
    logging.kubesphere.io/enabled: 'true'
  name: kubernetes
  namespace: kubesphere-logging-system
spec:
  filters:
    - kubernetes:
        annotations: true
        kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubeURL: 'https://kubernetes.default.svc:443'
        labels: false
    - nest:
        addPrefix: kubernetes_
        nestedUnder: kubernetes
        operation: lift
    - nest:
        addPrefix: kubernetes_annotations_
        nestedUnder: kubernetes_annotations
        operation: lift
    - modify:
        rules:
          - remove: stream
          - remove: kubernetes_pod_id
          - remove: kubernetes_host
          - remove: kubernetes_container_hash
          - rename:
              kubernetes_annotations_cni.projectcalico.org/podIPs: kubernetes_podIPs
          - removeRegex: kubernetes_annotations*
    - nest:
        nestUnder: kubernetes_annotations
        operation: nest
        removePrefix: kubernetes_annotations_
        wildcard:
          - kubernetes_annotations_*
    - nest:
        nestUnder: kubernetes
        operation: nest
        removePrefix: kubernetes_
        wildcard:
          - kubernetes_*
  match: kube.*

由 kubernetes Filter CR 生成的 fluent-bit config 配置如下（只看 Filter 部分，Input、Output CR 被省略）

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    docker
    Tag    kube.*
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    true
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
    Rename    kubernetes_annotations_cni.projectcalico.org/podIPs    kubernetes_podIPs
    Remove_regex    kubernetes_annotations*
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_annotations_*
    Nest_under    kubernetes_annotations
    Remove_prefix    kubernetes_annotations_
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    es-cdc-a-es-http.cdc.svc.xke.test.cn
    Port    9200
    HTTP_User    elastic
    HTTP_Passwd    elasticpwd
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp

可以在 kibana 看到采集到的 podIP：
在这里插入图片描述

  • Kubernetes

    Kubernetes 是 Google 开源的一个容器编排引擎，它支持自动化部署、大规模可伸缩、应用容器化管理。

    110 引用 • 54 回帖
  • 日志
    45 引用 • 105 回帖
北京 位置
127 4 16 11 10 3 83

相关帖子

回帖
fluent-bit debug 调试，采集 kubernetes podIP

欢迎来到这里！

我们正在构建一个小众社区，大家在这里相互信任，以平等 • 自由 • 奔放的价值观进行分享交流。最终，希望大家能够找到与自己志同道合的伙伴，共同成长。

注册 关于
请输入回帖内容 ...
liabio
公众号【mysmallsoup】专注于 Java、Go 语言 (Golang)、Docker、Kubernetes、软件架构 北京
回帖
1
 帖子
7
 积分
455