How to Convert Arrays of Hashes Into a Structured Key-Value Format During Log Processing – Source: socprime.com

In some log formats, fields can be arrays of hashes, requiring conversion into a structured key-value format. Fluentd supports this through inline Ruby scripts, enabling transformations during log processing.

For example, I need to convert the event_data field:

{   "event_data": [     {"name": "feature_enabled", "boolValue": true},     {"name": "user_roles", "multiValue": ["admin", "editor"]},     {"name": "session_id", "value": "abc123"}   ] }

Step 1. Write Ruby-Based Transformation Logic

The transformation requires iterating over the event_data array, extracting meaningful information, and producing a hash. The logic can be implemented as:

record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h

How it works:

• Access the event_data Field:

          record['event_data'] retrieves the array of hashes.

• Iterate Over the Array:

          .map iterates over each item in the array, returning a transformed version.

• Extract Key-Value Pairs:

          [item['name'], item['boolValue'] || item['multiValue'] || item['value']]:
         item['name']: The key in the resulting hash.
         item['boolValue'] || item[‘multiValue’] || item[‘value’]: The first non-nil value among the fields.

• Convert to a Hash:

          .to_h transforms the array of key-value pairs into a hash.If event_data is missing, the original value (nil) is retained. `So to make sure that the field exists, before applying the transformation, I use !record.dig('event_data').nil?  to skip the empty field.
You will see it at step 2.

Step 2. Setup Implementing in Fluentd

Use the record_transformer filter to apply the transformation dynamically. Enable Ruby scripting with the parameter enable_ruby true.

In Fluentd Configuration:

   @type record_transformer   enable_ruby true        event_data ${!record.dig('event_data').nil? ? record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h : record['event_data']}    

Key Components:

• Conditional Logic:

       !record.dig('event_data').nil?: Ensures the field exists before applying the transformation.

• Transformation Logic:

       Converts event_data into a normalized hash using Ruby.

• Fallback:

       If event_data is missing, the original value (nil) is retained.

For example, I received this output:

{   "event_data": {     "feature_enabled": true,     "user_roles": ["admin", "editor"],     "session_id": "abc123"   } }