Fields not getting masked

[f9-abs]

Hi,

I am trying to test this plugin on Windows 10 using td-agent v4, but defined fields are not getting masked. I ran td-agent in debug mode as well, but no there is message on what is going wrong. During td-agent startup, plugin does get loaded and fields to be masked are also read properly.

As you can see from attached files, I am trying to mask the key "ProviderName" and "Computer". This is the output (with dummy value for "Computer" tag), as you can see both keys are not getting masked:

--- sample log entries ---
2022-01-06 12:37:55.877141700 +0530 winevt.raw: {"ProviderName":"Nexthink Collector","ProviderGUID":"","EventID":"3221225728","Level":"2","Task":"1","Opcode":"0","Keywords":"0x80000000000000","TimeCreated":"2022/01/06 07:07:55.85937600","EventRecordID":"198171","ActivityID":"","RelatedActivityID":"","ProcessID":"0","ThreadID":"0","Channel":"Application","Computer":"N-LAPTOPID.domain.net","UserID":"","Version":"0","Description":"Connection to Nexthink Appliance cannot be established: Websocket error: Host not found: [Host not found] [appliance host: engine9.domain.net:8443]","EventData":["Connection to Nexthink Appliance cannot be established: Websocket error: Host not found: [Host not found] [appliance host: engine9.domain.net:8443]"]}

2022-01-06 12:34:13.877078900 +0530 winevt.raw: {"ProviderName":"ESENT","ProviderGUID":"","EventID":"102","Level":"4","Task":"1","Opcode":"0","Keywords":"0x80000000000000","TimeCreated":"2022/01/06 07:04:12.183182000","EventRecordID":"198155","ActivityID":"","RelatedActivityID":"","ProcessID":"0","ThreadID":"0","Channel":"Application","Computer":"N-LAPTOPID.domain.net","UserID":"","Version":"0","Description":"Video.UI (18280,P,98) {F28496CB-2EF7-4366-B722-51C0D5BFB252}: The database engine (10.00.18363.0000) is starting a new instance (0).","EventData":["Video.UI","18280,P,98","{F28496CB-2EF7-4366-B722-51C0D5BFB252}: ","0","10","00","18363","0000"]}
--- sample log entries ---

--- startup log ---
2022-01-06 12:27:03 +0530 [info]: adding match pattern="winevt.raw" type="stdout"
2022-01-06 12:27:05 +0530 [info]: adding filter pattern="**" type="masking"
black list fields:
ProviderName
Computer
--- startup log ---

Any ideas on what could be going wrong here? Thanks in advance.

test-files.zip

[kohend]

You didn't configure it properly.
The blacklist fields are the fields which contain the list of fields to mask, the list itself is not in the configuration.

[f9-abs]

In config file, I have provided path to the file containing list of fields to mask. Is this incorrect?

--- clip ---
<filter "**">
@type masking
fieldsToMaskFilePath "C:/opt/td-agent/fields-to-mask-file.txt"

--- clip ---

[kohend]

Oh, that looks OK, is the log parsed as JSON in your config? If not, it won't identify the fields properly.

[f9-abs]

Yes. I tested some other plugins (e.g. https://github.com/y-ken/fluent-plugin-anonymizer), and that worked fine.

[f9-abs]

Hi. Can you please share any updates on this? BTW, does this plugin support masking based on regex instead of keys?

[shyimo]

Hi @f9-abs. Thanks for the issue.
we will try to reproduce it the problem and let you know about the findings