Log Forwarding with Fluent Bit

Fluent Bit is a lightweight log processor and forwarder used in the SambaStack monitoring reference architecture. It collects container logs from Kubernetes nodes, enriches them with metadata, and ships them to OpenSearch for indexing and search.

Reference Architecture Note: This setup uses third-party components. Versions, defaults, and command syntax may change over time. Address any issues not specific to SambaStack to the vendor or project that owns that component.

Prerequisites

Before deploying Fluent Bit, ensure you have:

Kubernetes cluster — A running cluster with SambaStack installed
OpenSearch — Deployed in the monitoring namespace. See Log Storage - OpenSearch
OpenSearch admin password — The password configured during OpenSearch deployment
Helm (latest version) — For deploying the Fluent Bit chart
kubectl — Configured with access to your target Kubernetes cluster

Fluent Bit requires a running OpenSearch instance. Complete the OpenSearch deployment before proceeding.

Resource requirements

The following are minimum resource recommendations for Fluent Bit:

Component	CPU Request	Memory Request	Storage
Fluent Bit (per node)	100m	128Mi	—

Fluent Bit runs as a DaemonSet with one pod per node. Total cluster resources scale linearly with node count.

Architecture overview

In the SambaStack monitoring reference architecture:

Fluent Bit runs as a DaemonSet, ensuring one log forwarder per Kubernetes node
Collects logs from all pods and containers via /var/log/pods/
Enriches logs with namespace, pod, and container metadata using a Lua filter
Forwards logs to OpenSearch using secure TLS connections on port 9200

This deployment uses a custom Fluent Bit configuration stored in a Kubernetes Secret and a Helm-based installation with custom values.

How Fluent Bit log collection works:

Kubernetes automatically writes container logs to /var/log/pods/ on each node
Fluent Bit reads these logs directly from the host filesystem via a DaemonSet
Multiple parsers are configured to support both CRI and Docker log formats
Fluent Bit automatically selects the correct parser at runtime
Logs are enriched with Kubernetes metadata and forwarded to OpenSearch

Deployment steps

Step 1: Add the Helm repository

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Step 2: Retrieve the OpenSearch password

Get the OpenSearch admin password you created during OpenSearch deployment:

kubectl -n monitoring get secret opensearch-initial-admin-password -o json \
  | jq -r '.data.OPENSEARCH_INITIAL_ADMIN_PASSWORD | @base64d'

Save this password for use in Step 6.

Step 3: Create the Lua script

Save the following as ~/.sambastack-observability/fluentbit/append_tags.lua:

function append_tags(tag, timestamp, record)
  local filename = record["filename"] or ""
  local namespace_tag = string.match(filename, "/pods/([^_]+)")
  local container_tag = string.match(filename, "/pods/.*/([^/]+)/")
  local pod_tag = string.match(filename, "/pods/[^_]+_([^_]+)")

  if record["log"] then
    record["log"] = '"' .. string.gsub(record["log"], '"', '\\"') .. '"'
  end

  if namespace_tag then record["namespace"] = namespace_tag end
  if pod_tag then record["pod"] = pod_tag end
  if container_tag then record["container"] = container_tag end

  return 1, timestamp, record
end

Using a separate Lua file is less error-prone than embedding the code inline in the configuration file.

Step 4: Create the configuration file

Save the following as ~/.sambastack-observability/fluentbit/fluentbit-conf.conf:

[INPUT]
  Buffer_Chunk_Size 10MB
  Buffer_Max_Size 64MB
  Exclude_Path /var/log/pods/*fluent*/*/*.log
  Log_Level info
  Mem_Buf_Limit 64MB
  Name tail
  Path /var/log/pods/*/*/*.log
  Path_Key filename
  Rotate_wait 120
  Skip_Long_Lines On
  Tag kube.*
  multiline.parser cri, docker

[FILTER]
  Match  kube.*
  Name  lua
  script /fluent-bit/sn/append_tags.lua
  call  append_tags

[FILTER]
  Key_Name  log
  Match  kube.*
  Name  parser
  Parser  json
  Reserve_Data  On

[FILTER]
  Key_Name  log
  Match  kube.*
  Name  parser
  Parser  k8s-nginx-ingress
  Reserve_Data  On

[OUTPUT]
  HTTP_Passwd  ${DEFAULT_OPENSEARCH_HTTP_PASS}
  HTTP_User  ${DEFAULT_OPENSEARCH_HTTP_USER}
  Host  ${DEFAULT_OPENSEARCH_HOST}
  Index  logs-7d
  Log_Level  info
  Match  kube.*
  Name  opensearch
  Port  ${DEFAULT_OPENSEARCH_PORT}
  Retry_Limit  False
  Suppress_Type_Name  On
  tls  On
  tls.verify  Off

[SERVICE]
  Buffer_Size 64MB
  Daemon Off
  Flush 10
  HTTP_Listen 0.0.0.0
  HTTP_Port 2020
  HTTP_Server On
  Health_Check On
  Log_Level debug
  Parsers_File /fluent-bit/etc/parsers.conf
  Trace_Error On

Configuration explanation

INPUT section: Tails log files from all pods, excluding Fluent Bit’s own logs to prevent recursion.FILTER (Lua): Runs the external append_tags.lua script to extract namespace, pod, and container names from the log file path.FILTER (parser): Attempts to parse log content as JSON or nginx ingress format.OUTPUT: Sends logs to OpenSearch with TLS enabled. The logs-7d index name is configurable.SERVICE: Enables the HTTP server on port 2020 for health checks and metrics.

Step 5: Create the configuration secret

kubectl -n monitoring create secret generic sn-fluentbit \
  --from-file=sn-fluentbit.conf=~/.sambastack-observability/fluentbit/fluentbit-conf.conf \
  --from-file=append_tags.lua=~/.sambastack-observability/fluentbit/append_tags.lua \
  --dry-run=client -o yaml | kubectl apply -f -

The secret name sn-fluentbit must match the secretName in the Helm values file. If you change this name, update the values file accordingly.

Step 6: Create the Helm values file

Save the following as ~/.sambastack-observability/fluentbit/fluentbit-values.yml:

env:
- name: DEFAULT_OPENSEARCH_HOST
  value: "opensearch-cluster-master.monitoring.svc.cluster.local"
- name: DEFAULT_OPENSEARCH_PORT
  value: "9200"
- name: DEFAULT_OPENSEARCH_HTTP_USER
  value: "admin"
- name: DEFAULT_OPENSEARCH_HTTP_PASS
  valueFrom:
    secretKeyRef:
      name: opensearch-initial-admin-password
      key: OPENSEARCH_INITIAL_ADMIN_PASSWORD

securityContext:
  privileged: true

tolerations:
- operator: Exists

daemonSetVolumes:
- name: var
  hostPath:
    path: /var
- name: etcmachineid
  hostPath:
    path: /etc/machine-id
    type: File
- name: sn-fluentbit
  secret:
    secretName: sn-fluentbit

daemonSetVolumeMounts:
- name: var
  mountPath: /var
  readOnly: true
- name: etcmachineid
  mountPath: /etc/machine-id
  readOnly: true
- name: sn-fluentbit
  mountPath: /fluent-bit/sn
  readOnly: true

args:
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/sn/sn-fluentbit.conf

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 5

config:
  inputs: ""
  filters: ""
  outputs: ""

image:
  repository: fluent/fluent-bit
  tag: "4.2.2"

Step 7: Install Fluent Bit

helm -n monitoring upgrade --install fluent-bit fluent/fluent-bit \
  -f ~/.sambastack-observability/fluentbit/fluentbit-values.yml

Verification

Check pod status

Verify Fluent Bit pods are running on each node:

kubectl -n monitoring get po -l app.kubernetes.io/name=fluent-bit

Expected output (one pod per node):

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-4zh8w   1/1     Running   0          20h
fluent-bit-sdmxg   1/1     Running   0          20h

Verify logs are reaching OpenSearch

Port-forward to the OpenSearch service:

kubectl -n monitoring port-forward svc/opensearch-cluster-master 9200:9200 &

List indices to confirm log ingestion:

curl -u "admin:<password>" -k "https://localhost:9200/_cat/indices?v"

Expected output:

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logs-7d   0ahKcye3SfajNksbfapY4w   1   1     296144            0     58.1mb         58.1mb

The presence of the logs-7d index with a non-zero docs.count confirms Fluent Bit is successfully pushing logs into OpenSearch. To stop the port-forward process:

pkill -f "port-forward.*opensearch"

Query sample logs

Verify log content with a sample query:

curl -u "admin:<password>" -k "https://localhost:9200/logs-7d/_search?size=1&pretty"

You should see log entries with namespace, pod, and container fields populated.

Success criteria

Deployment is considered successful when:

Fluent Bit DaemonSet pods are in Running state on all nodes
The logs-7d index appears in OpenSearch with documents
Sample log queries return entries with namespace/pod/container metadata

Configuration reference

Section	Parameter	Description
INPUT	`Buffer_Chunk_Size`	Initial buffer size for reading log files (10MB)
INPUT	`Buffer_Max_Size`	Maximum buffer size per monitored file (64MB)
INPUT	`Mem_Buf_Limit`	Memory buffer limit before backpressure (64MB)
INPUT	`Rotate_wait`	Seconds to wait before reading rotated files (120)
INPUT	`Exclude_Path`	Glob pattern for logs to skip (Fluent Bit’s own logs)
OUTPUT	`Index`	OpenSearch index name for logs (`logs-7d`)
OUTPUT	`tls`	Enable TLS for OpenSearch connection
OUTPUT	`tls.verify`	Verify TLS certificate (Off for self-signed)
SERVICE	`Flush`	Interval in seconds to flush records to output (10)
SERVICE	`HTTP_Port`	Port for Fluent Bit’s HTTP server metrics (2020)

Troubleshooting

Pods not starting (ImagePullBackOff)

Symptom: Fluent Bit pods show ImagePullBackOff or ErrImagePull. Solution: Verify your cluster can reach the container registry. Check for network policies or proxy requirements.

No logs appearing in OpenSearch

Symptom: The logs-7d index doesn’t exist or has zero documents. Possible causes and solutions:

OpenSearch not reachable:

# Check Fluent Bit logs for connection errors
kubectl -n monitoring logs -l app.kubernetes.io/name=fluent-bit --tail=50

Wrong password: Verify the password in fluentbit-values.yml matches the OpenSearch admin password.

Secret not mounted:

kubectl -n monitoring describe pod -l app.kubernetes.io/name=fluent-bit | grep -A5 Mounts

Logs missing metadata fields

Symptom: Logs appear in OpenSearch but namespace, pod, or container fields are missing. Cause: The Lua filter failed to parse the log file path. Solution: Check that logs are coming from the expected path pattern (/var/log/pods/*/*/*.log).

High memory usage

Symptom: Fluent Bit pods are OOMKilled or show high memory consumption. Solution: Reduce buffer sizes in the configuration:

[INPUT]
  Buffer_Chunk_Size 5MB
  Buffer_Max_Size 32MB
  Mem_Buf_Limit 32MB

Next steps

After Fluent Bit is forwarding logs:

Deploy Prometheus and Grafana — Add visualization dashboards for your logs. See Monitoring with Prometheus and Grafana.
Explore logs in Grafana — Once Grafana is deployed with the OpenSearch datasource, you can query and visualize logs using the Explore feature.

Cleanup

To remove Fluent Bit from your cluster: Uninstall the Helm release:

helm uninstall fluent-bit -n monitoring

Delete the configuration secret:

kubectl delete secret sn-fluentbit -n monitoring

Removing Fluent Bit stops log forwarding but does not delete existing logs in OpenSearch. To remove historical logs, delete the index directly in OpenSearch or wait for index lifecycle policies to expire the data.

Overview

Installation

Service Administration

Hardware Administration

Reference Architectures

Resources

Prerequisites

Resource requirements

Architecture overview

Deployment steps

Step 1: Add the Helm repository

Step 2: Retrieve the OpenSearch password

Step 3: Create the Lua script

Step 4: Create the configuration file

Step 5: Create the configuration secret

Step 6: Create the Helm values file

Step 7: Install Fluent Bit

Verification

Check pod status

Verify logs are reaching OpenSearch

Query sample logs

Success criteria

Configuration reference

Troubleshooting

Pods not starting (ImagePullBackOff)

No logs appearing in OpenSearch

Logs missing metadata fields

High memory usage

Next steps

Cleanup

Overview

Installation

Service Administration

Hardware Administration

Reference Architectures

Resources

​Prerequisites

​Resource requirements

​Architecture overview

​Deployment steps

​Step 1: Add the Helm repository

​Step 2: Retrieve the OpenSearch password

​Step 3: Create the Lua script

​Step 4: Create the configuration file

​Step 5: Create the configuration secret

​Step 6: Create the Helm values file

​Step 7: Install Fluent Bit

​Verification

​Check pod status

​Verify logs are reaching OpenSearch

​Query sample logs

​Success criteria

​Configuration reference

​Troubleshooting

​Pods not starting (ImagePullBackOff)

​No logs appearing in OpenSearch

​Logs missing metadata fields

​High memory usage

​Next steps

​Cleanup

Prerequisites

Resource requirements

Architecture overview

Deployment steps

Step 1: Add the Helm repository

Step 2: Retrieve the OpenSearch password

Step 3: Create the Lua script

Step 4: Create the configuration file

Step 5: Create the configuration secret

Step 6: Create the Helm values file

Step 7: Install Fluent Bit

Verification

Check pod status

Verify logs are reaching OpenSearch

Query sample logs

Success criteria

Configuration reference

Troubleshooting

Pods not starting (ImagePullBackOff)

No logs appearing in OpenSearch

Logs missing metadata fields

High memory usage

Next steps

Cleanup