Fluentd: How to Use a Parser With Regular Expression (regexp) – Source: socprime.com

This guide explains configuring Fluentd to extract structured data from unstructured log messages using the parser plugin with a regular expression (regexp). If you need to extract specific fields, such as log_source and index, from a log message, you can do this as follows.

Input Log:

 {   "message": "Log source 'WinCollect DSM - SRV-AD-001' has stopped emitting events" }

Configuration:

   @type parser   key_name message   reserve_data true        @type regexp     expression /'(?[^']+)s-s(?[^']+)'/    

Explanation:

• key_name message: Specifies that the message field should be parsed.
• reserve_data true: Keeps the original message field along with the extracted fields.
• regexp expression:
  • (?[^']+): Captures the text before - as log_source.
  • (?[^']+): Captures the text after - as index.

Output Log:

{   "message": "Log source 'WinCollect DSM - SRV-AD-001' has stopped emitting events",   "log_source": "WinCollect DSM",   "index": "SRV-AD-001" }

If you need to extract fields such as timestamp, level, module, and message from logs with timestamps, you can do this as follows:

Input Log:

{   "message": "2024-12-18 10:15:30 ERROR [Auth] Login failed for user 'jdoe'" }

Configuration:

   @type parser   key_name message   reserve_data true        @type regexp     expression /(?d{4}-d{2}-d{2} d{2}:d{2}:d{2})s+(?[A-Z]+)s+[(?[^]]+)]s+(?.*)/    

Explanation:

• (?d{4}-d{2}-d{2} d{2}:d{2}:d{2}): Extracts the timestamp.
• (?[A-Z]+): Captures the log level (e.g., ERROR).
• (?[^]]+): Extracts the module name (e.g., Auth).
• (?.*): Captures the remaining log message.

Output Log:

{   "message": "2024-12-18 10:15:30 ERROR [Auth] Login failed for user 'jdoe'",   "timestamp": "2024-12-18 10:15:30",   "level": "ERROR",   "module": "Auth",   "message": "Login failed for user 'jdoe'" }

If you need to extract key-value pairs from a log message, you can do this as follows:

Input Log:

{   "message": "user=jdoe status=failed ip=192.168.12.1" }

Configuration:

   @type parser   key_name message   reserve_data true        @type regexp     expression /user=(?w+)s+status=(?w+)s+ip=(?[^s]+)/    

Explanation:

• (?w+): Captures the username.
• (?w+): Extracts the status (e.g., failed).
• (?[^s]+): Captures the IP address.

Output Log:

{   "message": "user=jdoe status=failed ip=192.168.12.1",   "user": "jdoe",   "status": "failed",   "ip": "192.168.12.1" }

Was this article helpful?

Like and share it with your peers.