Skip to content

Custom Fields

Jump to bottom
Karsten Schnitter edited this page Mar 17, 2022 · 4 revisions

Disclaimer: This page describes a feature, that is designed for the use with SAP BTP Application Logging. If you use this library outside this environment, it will most likely not be needed.

Introduction to Custom Fields

Within the SAP Business Techonolgy Platform log messages are indexed into Elasticsearch. They can be searched by any terms (words) appearing in any field of the emitted JSON object. But only certain predefined fields are included as indexed field, which allows filtering and limiting searches on those fields. These predefined fields are known as Context Fields in the library and exposed via LogContext. getContextFieldsKeys() (c.f. LogContext). More detailed information is available as asciidoc.

Users can add so called Custom Fields. These are key-value-pairs, that can be provided in two ways:

  • Declaration of MDC entries, e.g. LogContext.add(key, value) or MDC.put(key, value)
  • Parameters of log messages via CustomField, e.g. LOG.info(message, CustomField.customField(key, value))

Any such field will be added to the JSON message as a top-level entity. Log message parameters take precedence over MDC fields. Hence, it is possible to overwrite MDC fields by this feature.

Within SAP BTP Application Logging service only the Context Fields will be parsed and indexed in Elasticsearch. In order to extend this to Custom Fields, these fields have to be sent as a nested object within the JSON message. This is achieved by declaring all possible custom fields in the logging configuration (s.b.). Then the key-value-pairs will be indexed as keyword fields in Elasticsearch. This allows users to use these fields in custom queries and visualisations to improve their Elasticsearch experience. The generated JSON structure for one custom field looks like this:

{
    "#cf": {
        "string": [
            {"k": "key", "v": "value", "i": 0}
        ]
    }
}

The whole entry will only be generated if there is at least one custom field in the log message. The index i refers to the position of the Custom Field during declaration in the logging configuration. This ensures, that the index is stable regardless of the number of actual emitted Custom Fields, as long as the configuration remains unchanged. It is possible to emit a custom field both at top level and inside this nested structure. These fields are called Retained Fields. You can declare a Context Field as a Custom Field, but it will always be a Retained Field and hence appear in both places.

Declaration of Custom Fields

To emit log message parameters or MDC fields as Custom Fields the library supports configuration for both logging backends Log4j and Logback. In the standard configuration, such fields will be moved from top-level to the #cf nested object within the JSON object. To reduce the message size, such fields will no longer be available at top-level. But there is an additional configuration, to retain the top-level copy. Note, that Context Fields will never be removed from top-level, but only a copy will be created in the #cf object.

Log4j Example

The following configuration shows two MDC fields custom-field and retained-field, that are declared as custom fields with the latter one declared to retain a top-level copy.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" strict="true"
	packages="com.sap.hcp.cf.log4j2.converter,com.sap.hcp.cf.log4j2.layout">
	<Appenders>
		<Console name="STDOUT-JSON" target="SYSTEM_OUT" follow="true">
			<JsonPatternLayout charset="utf-8">
				<customField mdcKeyName="custom-field" />
				<customField mdcKeyName="retained-field" retainOriginal="true" />
			</JsonPatternLayout>
		</Console>
	</Appenders>
	<Loggers>
		<Root level="${LOG_ROOT_LEVEL:-WARN}">
			<AppenderRef ref="STDOUT-JSON" />
		</Root>
		<Logger name="com.sap.hcp.cf" level="INFO" />
	</Loggers>
</Configuration>

Logback Example

The following configuration shows two MDC fields custom-field and retained-field, that are declared as custom fields with the latter one declared to retain a top-level copy.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false" scan="false">
    <appender name="STDOUT-JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.sap.hcp.cf.logback.encoder.JsonEncoder">
        	<customFieldMdcKeyName>custom-field</customFieldMdcKeyName>
        	<customFieldMdcKeyName>retained-field</customFieldMdcKeyName>
        	<retainFieldMdcKeyName>retained-field</retainFieldMdcKeyName>
        </encoder>
    </appender>
    <root level="${LOG_ROOT_LEVEL:-WARN}">
        <appender-ref ref="STDOUT-JSON" />
    </root>
    <logger name="com.sap.hcp.cf" level="INFO" />
</configuration>

Custom Fields API of the Library

MDC Fields

SLF4j defines the Mapped Diagnostic Context (MDC) as a map of key-value-pairs to allow tagging common sets of log messages by custom entries. All MDC entries will be attached to the generated JSON object as top-level entries by the logging library. However, they will not be indexed by the SAP Cloud Platform Logging Service, unless they are Context Fields.

Typed Parameter of Log Message

SLF4j allows arbitrary parameters to a log message, usually used for formatting the log message. The library offers a special parameter type CustomField, that will be emitted as Custom Field in the generated log message. An example of this feature looks as follows:

import static com.sap.hcp.cf.logging.common.customfields.CustomField.customField;

// omitted for brevity 

LOGGER.info("Logging with custom field {}", customField("customKey", "customValue"));
Clone this wiki locally