Fluentd: How to Change Tags During Log Processing. – Source: socprime.com

I have a case where I need to drop unnecessary logs. I found a plugin that helps do that.
The rewrite_tag_filter plugin is used to dynamically modify the tags of incoming log records based on their content. You can rewrite tags,  route logs more effectively, organize them based on certain conditions, and ensure logs are processed by different filters or outputs.

For example, in the code below, the rewrite_tag_filter is used to change the tags of logs based on the contents of the User-Agent header in HTTP request logs.

   @type rewrite_tag_filter        key                             $.httpRequest.headers.User-Agent     pattern                         /bUptimeRobotb/     tag                             test-drop           key                             $.httpRequest.headers.User-Agent     pattern                         /bPingdomb/     tag                             test-drop           key                             $.httpRequest.headers.User-Agent     pattern                         /bPingdomb/     invert                          true     tag                             test-all    

Description

Match Directive ():
This block applies to all logs whose tags start with test-raw. Logs that match this pattern will be passed to the rewrite_tag_filter plugin.

First Rule:
( key $.httpRequest.headers.User-Agent pattern /bUptimeRobotb/ tag test-drop ):
This rule checks if the User-Agent field in the httpRequest headers contains the string UptimeRobot.

If the pattern matches (i.e., the User-Agent contains UptimeRobot), the tag is changed to test-drop. Logs with this User-Agent are flagged for “dropping” or special handling.

Second Rule:
( key $.httpRequest.headers.User-Agent pattern /bPingdomb/ tag test-drop ):

Similarly, this rule checks if the User-Agent field contains Pingdom.

If the pattern matches (i.e., the User-Agent contains Pingdom), the tag is again changed to test-drop.
This ensures that the Pingdom monitoring service logs are also marked as “drop” logs.

Third Rule:
( key $.httpRequest.headers.User-Agent pattern /bPingdomb/ invert true tag test-all ):
This rule is similar to the second rule but includes invert: true.

The invert: true option inverts the matching condition. Logs where the User-Agent does not contain the string Pingdom.

If the User-Agent does not contain Pingdom, the tag is changed to test-all, which indicates normal processing for these logs.

Logs with the test-drop tag (from UptimeRobot and Pingdom user agents) will be routed to the /var/log/blocked_requests.log file.

   @type file   path /var/log/blocked_requests.log  # Logs with this tag will go to a drop file 

Logs with the test-all tag (from requests that do not contain Pingdom) will be routed to the /var/log/allowed_requests.log file.

   @type file   path /var/log/allowed_requests.log  # Logs with this tag will go to a different file 

This plugin allows you to categorize logs and route them to different outputs for further processing or analysis. In your example, the tag is changed for logs coming from monitoring services like UptimeRobot and Pingdom, allowing you to handle these logs separately from normal traffic.

This is a custom plugin, so you will need to install it using the command below:

fluent-gem install fluent-plugin-rewrite-tag-filter

Was this article helpful?

Like and share it with your peers.