Filebeat: Receiving Docker logs in Kafka – Source: socprime.com

To receive logs from your containers in Kafka topic, we have to do these steps:

1. Install Filebeat
2. echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
3. sudo apt-get update && sudo apt-get install filebeat

  2. Edit the configuration file /etc/filebeat/filebeat.yml with these lines

filebeat.config:   modules:     path: ${path.config}/modules.d/*.yml     reload.enabled: false  processors:   - add_cloud_metadata: ~   - add_docker_metadata: ~  filebeat.inputs: - type: container   paths:     - '/var/lib/docker/containers/*/*.log'  # ============================= Kafka Output =============================  output.kafka:   hosts: ["kafka-server:9093"]   topic: "docker-logs"   ssl.certificate_authorities: "https://socprime.com/etc/filebeat/certs/caroot.pem"   ssl.certificate: "https://socprime.com/etc/filebeat/certs/cert.pem"   ssl.key: "https://socprime.com/etc/filebeat/certs/key.pem"   max_message_bytes: 2000000

 3. Enable Filebeat service sudo systemctl enable filebeat
 4. Restart Filebeat.

Was this article helpful?

Like and share it with your peers.