Configuring logging in helm chart

[dimabarbul]

Hello
I need help on configuring logging for Ditto. We install Ditto using helm chart. By default, Ditto is quite chatty and logs much information which makes troubleshooting harder. We'd like to leave only important messages being logged to stdout.
Unfortunately, I didn't find any guide on this topic. So here is what we did.

We've managed to disable Pekko cluster logs and some Ditto distributed data logs. However, there are other messages, like, health status, other distributed data logs etc.

As I see in files under /ditto/deployment/helm/ditto/logback-config folder (for example, connectivity.xml), only log level for org.eclipse.ditto can be customized using environment variable. But raising this level to WARN will hide some important messages, for example, failures in connections or when connection actors lose connection and reestablish it.

So we thought that we can take blacklist approach - set log level to WARN only for those loggers that we want to produce less logs.
But I can't figure out how we can use custom logback XML file, because at logback-config.xml it creates config maps with XML files provided with Ditto and only the file name can be changed (which does not seem to impact anything).
Alternative way we found is to call Ditto API to manage log levels at runtime after the deployment, but this approach have several drawbacks (like, when we need to change levels, we need to restart impacted services to "forget" runtime customizations and apply new settings).

Any help is highly appreciated.

[thjaeckle]

Hi @dimabarbul

The Helm chart is prepared to configure "custom config files" for the Ditto services:

ditto/deployment/helm/ditto/values.yaml

Lines 155 to 161 in 4d9372b

	# customConfigFile configures that a custom "Logback" config file should be used instead of the one bundled
	# with Ditto on the classpath
	customConfigFile:
	# enabled if enabled, a custom logback.xml file added to the Ditto containers will be used for logging configuration
	enabled: true
	# fileName passed as Java system property "-Dlogback.configurationFile"
	fileName: logback.xml

That means that you can simply provide your custom logback XML configfile e.g. via a ConfigMap and mount it into the filesystem of the container.

E.g. for things-search:

ditto/deployment/helm/ditto/templates/thingssearch-deployment.yaml

Lines 147 to 149 in 4d9372b

	{{- if .Values.global.logging.customConfigFile.enabled }}
	-Dlogback.configurationFile=/opt/ditto/{{ .Values.global.logging.customConfigFile.fileName }}
	{{- end }}

And the mount:

ditto/deployment/helm/ditto/templates/thingssearch-deployment.yaml

Lines 301 to 305 in 4d9372b

	{{- if .Values.global.logging.customConfigFile.enabled }}
	- name: ditto-custom-log-config
	mountPath: /opt/ditto/{{ .Values.global.logging.customConfigFile.fileName }}
	subPath: {{ .Values.global.logging.customConfigFile.fileName }}
	{{- end }}

And the volume:

ditto/deployment/helm/ditto/templates/thingssearch-deployment.yaml

Lines 355 to 358 in 4d9372b

	{{- if .Values.global.logging.customConfigFile.enabled }}
	- name: ditto-custom-log-config
	configMap:
	name: {{ .Release.Name }}-logback-config-thingssearch-xml

This is enabled by default and the following template deploys the in the Helm chart bundled logback configfiles as ConfigMaps:
https://github.com/eclipse-ditto/ditto/blob/master/deployment/helm/ditto/templates/logback-config.yaml

Maybe this needs additional Helm customization in order to get other files deployed ..

[dimabarbul]

Thanks for the response.

I was able to make it work locally with new custom configmap and customized values.yaml to mount the configmap and use custom XML on per-service level. I believe, this is what you meant, right ?

Values.yaml and k8s configmap resource

Values.yaml

# example is for connectivity service, others are configured alike
connectivity:
  # as .Values.global.logging.customConfigFile.enabled is true, the services will be configured with
  # -Dlogback.configurationFile=/opt/ditto/{{ .Values.global.logging.customConfigFile.fileName }} 
  # but it will be overwritten by systemProps
  systemProps:
    - '-Dlogback.configurationFile=/opt/ditto/extra-config.xml'
  extraVolumes:
    - name: connectivity-extension
      configMap:
        name: connectivity-extension
  extraVolumeMounts:
    - name: connectivity-extension
      mountPath: /opt/ditto/extra-config.xml
      subPath: extra-config.xml

k8s config map resource

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: connectivity-extension
data:
  extra-config.xml: |-
    <?xml version="1.0" encoding="UTF-8"?>
    <configuration>
        <appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
            <encoder class="net.logstash.logback.encoder.LogstashEncoder">
                <excludeMdcKeyName>sourceActorSystem</excludeMdcKeyName>
                <excludeMdcKeyName>pekkoUid</excludeMdcKeyName>
                <excludeMdcKeyName>pekkoTimestamp</excludeMdcKeyName>
                <mdcKeyFieldName>x-correlation-id=correlation-id</mdcKeyFieldName>
                <mdcKeyFieldName>connection-id=ditto-connection-id</mdcKeyFieldName>
                <mdcKeyFieldName>connection-type=ditto-connection-type</mdcKeyFieldName>
            </encoder>
        </appender>

        <!-- Do not print INFO cluster messages of mongo driver: -->
        <logger name="org.mongodb.driver" level="WARN"/>

        <logger name="org.apache.pekko.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes" level="WARN"/>
        <logger name="org.apache.pekko.management.cluster.bootstrap.internal.BootstrapCoordinator" level="WARN"/>
        <logger name="org.apache.pekko.actor.CoordinatedShutdown" level="INFO"/>

        <!-- CUSTOMIZATION -->
        <logger name="org.eclipse.ditto.internal.utils.persistentactors.PersistencePingActor" level="WARN"/>
        <logger name="org.eclipse.ditto.internal.utils.health.status.StatusSupplierActor" level="WARN"/>
        <logger name="org.eclipse.ditto.internal.utils.persistentactors.cleanup" level="WARN"/>
        <logger name="org.apache.pekko.cluster.ddata.Replicator" level="WARN"/>

        <logger name="org.apache.kafka" level="WARN"/>

        <!-- Log level for the application -->
        <logger name="org.eclipse.ditto" level="${LOG_LEVEL_APPLICATION:-INFO}"/>

        <root level="${LOG_LEVEL:-INFO}">
            <appender-ref ref="jsonConsoleAppender"/>
        </root>
    </configuration>

Another question on logging configuration: is there a way to forward failures happened in connections (during mapping or sending) to stdout?
Currently, as I see, the failures are logged separately (in DB, I guess?) and moreover - they are auto-muted by default in an hour (can be customized using CONNECTIVITY_LOGGER_LOG_DURATION environment variable). But I'd like to have all logs that need attention in stdout.

[thjaeckle]

Another question on logging configuration: is there a way to forward failures happened in connections (during mapping or sending) to stdout?
Currently, as I see, the failures are logged separately (in DB, I guess?) and moreover - they are auto-muted by default in an hour (can be customized using CONNECTIVITY_LOGGER_LOG_DURATION environment variable). But I'd like to have all logs that need attention in stdout.

No, there is no such option yet.
Currently, the logs are collected just in memory (therefore only 1 hour) - this was intended to e.g. "debug" a connection easily using the Ditto UI.

However, logs of Ditto's connectivity service should also contain failures via the normal logger ..
What you need is probably just a way to identify those (maybe an MDC - similar to correlation-id or connection-id entry added to the logs would be an option then to "tag" them).

I would avoid logging plain to stdout - as this interferes with service container logging.

[dimabarbul]

Understood, I agree that plain text logs would be bad in stdout.

As for identifying the records with some connection failure information, did I get it right, that it will require modifications in code to add the MDC (e.g., ditto-connection-failure=true) to such records, and later it would be possible to raise level to WARN but still log such records using, for example, MDCFilter (as described in Logback docs) to ACCEPT those records?

[thjaeckle]

Yes, Ditto code would need to be modified to add those MDC values👍

And yes, that would be one option with the MDCFilter ..
Another one .. "normally" you would have a logging backend (like Elasticsearch or Grafana Loki or another logging system) - and those are normally configured to "map" mdc-entries to e.g. labels or fields to add to a search index.
Doing so, you can simply e.g. select all logs with a given MDC entry, or define an alert on such logs based on the MDC values.