Fluentd: Work With Multiple Log Sources Within a Single Instance by Using @label – Source: socprime.com

@label is a feature that defines multiple processing pipelines within a single instance. Labels allow you to route log data through different pipelines, applying different configurations or processing steps to different kinds of logs.

How @label Works

@label is used to route logs to specific pipelines. For instance, you might want to route logs from different applications or services to different destinations or process them differently depending on the type of log.

@label Usage Example

Here’s an example of how @label is used in a Fluentd configuration file:

     @type tcp     tag tcp.events # required     @label                              @test-label-001              @type none          port 20001   # optional. 5170 by default     bind 0.0.0.0 # optional. 0.0.0.0 by default     delimiter "n" # optional. "n" (newline) by default        @type                               http     @label                              @test-label-002     port 9881     bind 0.0.0.0     body_size_limit 32m     keepalive_timeout 10s                 @type                                                   record_transformer         enable_ruby                                             true                      @timestamp                                         ${Time.now.strftime('%Y-%m-%dT%H:%M:%S.%9N%z')}             message                                            test message                          @type stdout                    @type stdout      

Explanation

Inputs:

• @type tcp: Specifies that this input source listens for incoming data over TCP.
• @type http: Specifies that this input source listens for incoming HTTP requests.

Labels:

• Logs from @test-label-001 are routed to the pipeline labeled @test-label-001
• Logs from @test-label-002 are routed to the pipeline labeled @test-label-002.

Outputs:

• Logs from tcp source are printed to stdout in the @test-label-001 pipeline with a filter to add new fields.
• Logs from http source are printed to stdout in the @test-label-002 pipeline without any filter.

When to Use

• Separation of Concerns: Use labels when you want to handle different log sources separately, either for routing to different destinations or applying different processing.
• Performance Optimization: Labels can help optimize performance by allowing you to isolate pipelines, reducing the complexity of a single pipeline.

Key Points

• @label defines a separate pipeline.
• Each pipeline can have its own configuration.
• @label allows you to isolate the processing of different logs in the same Fluentd instance.

Was this article helpful?

Like and share it with your peers.