Fluentd: How to Make Nested Hash from Dot-Separated Keys – Source: socprime.com

When a log record has keys like test.test, you can use two methods record_transformer and explode plugin, to process these keys and create a nested structure.

For example, Input Log Record:

{   "message": "A test message",   "test.test": "value1",   "test.another": "value2" }

Method: record_transformer

     @type record_transformer     enable_ruby true     auto_typecast true     remove_keys test.test,test.another          test                               ${{"test": record['test.test']}}     test                               ${{"another": record['test.another']}}      

Method: explode

     @type explode 

Using those methods, you will get one result.

Output Log:

{   "message": "A test message",   "test": {     "test": "value1",     "another": "value2"   } }

But when using record_transformer, you have to follow many steps to create nested fields, and the result is identical to the result where you use explode and turn it on in one step.

The Explode plugin takes the top-level keys with dots and breaks them into nested structures.
This way you can do with fields that go through from the explode plugin any action without an issue

For example, Input Log Record:

{   "user.name.first": "John",   "user.name.last": "Doe",   "user.contact.email": "john.doe@example.com" }

   @type explode 

   @type record_transformer        full_name ${record["user"]["name"]["first"]} ${record["user"]["name"]["last"]}    

Output Log Record:

{   "full_name": "John Doe"   "user": {     "name": {       "first": "John",       "last": "Doe"     },     "contact": {       "email": "john.doe@example.com"     }   } }

This example creates a new full_name field by extracting data from the nested structure created by the plugin explode.

Explode is a custom plugin so you will need to install it using the command below:

gem install fluent-plugin-explode_filter

Was this article helpful?

Like and share it with your peers.