Fluentbit

The demo application comes with a pre-configured Fluent-bit instance and agent. We chose Fluent-bit as it is an open-source, versatile and very popular solution when it comes to log management, meaning it should be relatively easy for you to connect Cloud Active Defense alerts to your preferred solution.

The instance

The instance is a simple container configured in docker-compose.yaml. We expose its default port to ensure the connectivity with the agent. The fluentbit container waits for alerts to be sent to it. Whenever an alert is received, it is displayed in the console.

The instance is configured with two files that we mount as volumes - going this way allows Fluentbit to be configured without having to create a dedicated Dockerfile.

fluent-bit.conf

Comes with the following setup:

[SERVICE]

Global properties.

log_level debug:

• Verbose output, useful for troubleshooting and development. Can be safely turned to info.

parsers_file /fluent-bit/etc/custom_parsers.conf:

• The file dealing with how the content should be parsed before being forwarded (or here: display to the local console).

[INPUT]

Where and how logs are collected.

Name forward:

• The input plugin to use, in this case, forward. This plugin allows Fluent Bit to receive logs from the agent running in the proxy.

Listen 0.0.0.0:

• Fluent Bit will listen on all its available network interfaces.

Port 24224:

• The port number on which Fluent Bit will listen for incoming log data (default).

[OUTPUT]

The output destination for the processed logs.

Name stdout:

• The output plugin to use, in this case, stdout. This plugin outputs the logs to the standard output (console).

Match **:

• Pattern to match tags from incoming logs. ** matches all tags, meaning all logs will be sent to this output.

[FILTER]

Filter(s) that process logs between input and output stages.

First filter

Name parser:

• The filter plugin to use, in this case, parser. This plugin applies a parser to transform log data.

Match *:

• Specifies which logs to apply this filter to. * means this filter will apply to all logs.

Parser decoy_custom:

• The name of the parser to apply, as defined in the parsers_file specified earlier. remove_prefix is the name of the parser defined in /fluent-bit/etc/custom_parsers.conf.

Key_Name log:

• Specifies the key within the log record to apply the parser to. log is the key where the data to be parsed is located.

Second filter

Name parser:

• The filter plugin to use, in this case, parser. This plugin applies a parser to transform log data.

Match *:

• Specifies which logs to apply this filter to. * means this filter will apply to all logs.

Parser decoy_custom:

• The name of the parser to apply, as defined in the parsers_file specified earlier. remove_prefix2 is the name of the parser defined in /fluent-bit/etc/custom_parsers.conf.

Key_Name log:

• Specifies the key within the log record to apply the parser to. log is the key where the data to be parsed is located.

Third filter

Name grep:

• The filter plugin to use, in this case, grep. This plugin filters logs based on regular expressions.

Match *:

• Specifies which logs to apply this filter to. * means this filter will apply to all logs.

Regex log \b(type"\s*:\s*"(alert|event|system|debug))\b:

• Defines the regular expression to filter logs. This expression will match logs where the type field within the log object is either set to alert, event, system or debug.

parsers.conf

Defines how alerts should be parsed, comes with the following setup:

[PARSER]

First parser

Name remove_prefix

• the name of the parser. Matches the name defined in the Parser field of fluent-bit.conf

Format regex

• says that the parser works with the regular expression plugin

Regex ^\[.*?\]\s*\[wasm\]\s*\[.*?\]\s*wasm log cookie_plugin:\s*(?<log>.*)$

• the format of the regular expression: the prefix of each logs set by envoy will be retrieved and removed from the log to keep usefull information only.

Second parser

Name remove_prefix2

• the name of the parser. Matches the name defined in the Parser field of fluent-bit.conf

Format regex

• says that the parser works with the regular expression plugin

Regex ^\[.*?\]\s*\[wasm\]\s*\[.*?\]\s*wasm log:\s*(?<log>.*)$

• the format of the regular expression: the variant prefix of each logs set by envoy will be retrieved and removed from the log to keep usefull information only.

The agent

The fluent-bit agent (named Fluentd) is attached to the Proxy, so that each output sent to the proxy logs gets forwarded to the fluent-bit instance. The instance then decides to keep the event (if it's an alert) or not (otherwise).

An agent is attached to the Proxy via the following docker-compose.yaml configuration lines:

logging:
  driver: fluentd