fluent-bit output plugin for google pubsub

This plugin is used to publish data to queue in google pubsub.

You could easily use it.

Build

A bin directory already has been made binaries for mac, linux.

If you should directly make binaries for mac, linux

# local machine binary
$ bash make.sh build

# Your machine is mac, and if you should do to retry cross compiling for linux.
# A command in below is required a docker.  
$ bash make.sh build_linux

Usage

configuration options for fluent-bit.conf

Key	Description	Default
Project	google cloud project id	NONE(required)
Topic	google pubsub topic name	NONE(required)
JwtPath	jwt file path for accessible google cloud project	NONE(required)
Debug	print debug log	false(optional)
Uuid ^1	add UUID to attributes. (keyname:uuid)	false
Timeout	the maximum time that the client will attempt to publish a bundle of messages. (millsecond)	60000 (optional)
DelayThreshold	publish a non-empty batch after this delay has passed. (millsecond)	1
ByteThreshold	publish a batch when its size in bytes reaches this value.	1000000
CountThreshold	publish a batch when it has been reached count of messages.	100
JSONEncode	publish a whole log as JSON encoded string instead of per record.	false
Attributes ^2	publish a log embedded custom attributes as metadata. (value is fixed)	NONE(optional)

Example fluent-bit.conf

[Output]
    Name       pubsub
    Match      *
    Project    your-project(custom)
    Topic      your-topic-name(custom)
    Jwtpath    your-jwtpath(custom)
    Attributes mapdata(custom)
    Uuid       True
    JSONEncode True

Example exec

$ fluent-bit -c [your config file] -e pubsub.so

Note

^1^2(TBD)
The Uuid, Attributes options are not supported for concurrent map iteration error.
Fix this issue in the future.

Todo

• Add test for JSONEncode options
• Fix Uuid/Attributes options (for concurrent map iteration error)