Fluent Bit allows users to modify log data through a Modify filter with conditions. Find Part 1 of this blog for basic filters here: Fluent Bit Modify Log data with Modify Filter plugin Examples

Advanced Filter with Lua

This is not always enough for complex tasks, however, so FluentBit allows users to create custom Lua scripts for use in filters. This can manipulate or filter logs with more complexity like modifying nested JSON based on condition. Let's see some examples below.

In the following example, we have to change message which is inside another object status

{   
  "status":{   
  "code":200,   
  "message":"true"   
  }
}

We are not covering the basics of Lua here. If you want to learn the basics of Lua script, see Learn Lua Script .

Writing Lua script for the Filter Logic

Write the Lua script in a script file and add that to the fluent bit configuration

Step 1 : Create script file

Create an empty file and name test script.lua on the same directory of fluentbit config file

Step 2 : Create a function inside the script file

The code below will replace status.message to "Success" if the status.code is 200

function replace_status(tag,timestamp,record)  
 if status.code == 200 then    
   status.message = "Success"  
 end  
return 0,timestamp,record
end

Step 3 : Add the Lua script to filter condition

In the filter section, the function name and script file name are mentioned which are in the same location as the configuration file (add full path if it is on different path).

[INPUT]    
   Name tcp    
   Port 8888
[FILTER]    
   Name lua    
   Match *    
   script  test.lua    
   call    replace_status
[OUTPUT]    
   Name stdout    
   Match **

Sample Input

{ 
  "status":{   
  "code":200,   
  "message":"true"  
  }
}

Sample Output

{ 
  "status":{   
  "code":200,   
  "message":"Success"  
  }
}

Mutate data of Nested child based on condition

Let us check a bit more advanced function where we need to replace a list of child item, see the sample below

Sample Input

{ 
  "data":[{   
  "item_staus":0,   
  "message":"nil"  
  },
{   
  "item_staus":1,   
  "message":"nil"  
  },
{   
  "item_staus":0,   
  "message":"nil"  
  }]
}

Replace the message to "True" if item_status is 1 and "False" if item_status is 0

function replace_status(tag, timestamp, record)    
   data = record.data    
   for itemCount = 1, #workouts do        
     if workouts[itemCount].item_status == 1 then            
        workouts[itemCount].message = "True"        
     else            
        workouts[nameCount].message = "False"             
     end    
  end    
  return 0, timestamp, record
end

Sample Output

{ 
 "data":[{   
   "item_staus":0,   
   "message":"False"  
  },
{   
   "item_staus":1,   
   "message":"True"  
  },
{   
   "item_staus":0,   
   "message":"False"  
 }]
}

Advanced loops and Conditions in Lua script can be used to filter, mutate and enrich data while passing through fluent bit

TLDR;

  • How to use filter in Fluent Bit to modify fields with Lua script
  • Loop through child object and use conditions to replace

If you are not familiar with fluent bit, see related blogs to learn how to run these on fluent bit with and without docker.