Standard Logstash Template for Event Processing (Gold Template) – Source: socprime.com

This standard template for configuring Logstash pipelines, commonly referred to as a “gold template,” ensures consistent metadata enrichment for events processed through Logstash, making it particularly useful in environments where data comes from diverse sources.

Configuration Template

Below is the template with an explanation of its key components:

filter {     if [type] == "TYPE_NAME" {  # Replace 'TYPE_NAME' with your specific type         ruby {             init => "require 'socket'"  # Load the 'socket' library for hostname resolution             code => "                 event.set('[receipt0][time]', LogStash::Timestamp.new(Time.now))  # Add the current timestamp                 event.set('[receipt0][hostname]', Socket.gethostname)  # Add the Logstash server's hostname             "         }         if [host] =~ /d+.d+.d+.d+/ {  # Check if 'host' matches an IP address format             mutate { rename => { "host" => "[agent][ip]" }}  # Rename 'host' to '[agent][ip]' if it is an IP address         } else {             mutate { rename => { "host" => "[agent][hostname]" }}  # Rename 'host' to '[agent][hostname]' otherwise         }         mutate {             add_field => { "[receipt0][type]" => "logstash" }  # Add a field to indicate processing by Logstash         }     } }

Key Features

Ruby Block for Metadata Enrichment

Purpose: the Ruby block adds dynamic metadata to the event, including:

• Current processing time ([receipt0][time])

• Hostname of the Logstash server processing the event ([receipt0][hostname])

Advantages: this metadata provides a traceable record of when and where the event was processed, aiding in debugging and monitoring.

Host Field Renaming

Logic: Depending on whether the host field contains an IP address or a hostname, it is renamed to [agent][ip] or [agent][hostname], respectively.

Benefit: this standardizes the structure of events for downstream systems, simplifying data querying and analysis.

Adding Logstash Type Information: the [receipt0][type] field explicitly marks the event as processed by Logstash, helping identify its origin.

How to Use

1. Replace TYPE_NAME. Replace "TYPE_NAME" with the appropriate event type you want this template to process (e.g., "auditd", "syslog", etc.).
2. Insert into Pipeline Configuration. Add the template to your Logstash pipeline configuration under the filter section.
3. Test the Configuration: validate your configuration using:
   • bin/logstash –path.config /path/to/config –config.test_and_exit
4. Deploy to Production. Once tested, deploy the updated configuration to your production environment.

Example Use Case

Suppose you are processing syslog events, and your configuration looks like this:

filter {     if [type] == "syslog" {         ruby {             init => "require 'socket'"             code => "                 event.set('[receipt0][time]', LogStash::Timestamp.new(Time.now))                 event.set('[receipt0][hostname]', Socket.gethostname)             "         }         if [host] =~ /d+.d+.d+.d+/ {             mutate { rename => { "host" => "[agent][ip]" }}         } else {             mutate { rename => { "host" => "[agent][hostname]" }}         }         mutate {             add_field => { "[receipt0][type]" => "logstash" }         }     } }

When a syslog event is ingested:

• The current timestamp and Logstash hostname are added.
• The host field is standardized.
• A marker field is added to indicate Logstash processing.

Advantages of the Gold Template

• Consistent Metadata: Enriches events with uniform metadata for easier traceability.
• Standardized Field Names: Reduces ambiguity by renaming fields based on content type.
• Adaptable to Multiple Use Cases: Easily configurable for different event types.

By using this template, Logstash administrators can ensure efficient and consistent processing of events, simplifying both troubleshooting and downstream analytics.

Was this article helpful?

Like and share it with your peers.