No way to disable durable keys with DData when using Akka.Cluster.Sharding

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.27 and later
Which Akka.NET Modules? Akka.Cluster.Hosting

Describe the bug

Here's the basic problem.

Both:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithDistributedData(options =>
            {
                options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            })
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName,
            });
        return builder;
    }
}

And

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithDistributedData(options =>
            {
                options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            })
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName,
            });
        return builder;
    }
}

Still result in DData.Durable writing out durable keys and creating a new LMDB file.

Previously, the following command would have (edit: might have?) worked:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName, 
                DistributedData = { Durable = new DurableOptions() { Keys = [] }} // now-deprecated property
            });
        return builder;
    }
}

But that only worked because the WithShardRegion<T> command worked by applying HOCON globally, which was incorrect and wrong.

The suggested workaround for setting the ShardOptions.DistributedData property in the comments is to apply HOCON globally. That's what I'm effectively doing in the above code sample. However, this doesn't work because:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding/ClusterShardingGuardian.cs

Lines 302 to 309 in 355439e

	var config = Context.System.Settings.Config.GetConfig("akka.cluster.sharding.distributed-data")
	.WithFallback(Context.System.Settings.Config.GetConfig("akka.cluster.distributed-data"));
	var configuredSettings = ReplicatorSettings.Create(config);
	var settingsWithRoles = configuredSettings.WithRole(shardingSettings.Role);
	if (shardingSettings.RememberEntities)
	return settingsWithRoles;
	else
	return settingsWithRoles.WithDurableKeys(ImmutableHashSet<string>.Empty);

The sharding system will set durable keys anyway - so this makes me wonder, perhaps, if this is really an issue with Akka.Cluster.Sharding itself. I would love not to have an LMDB database created when I'm using remember-entities=on. I guess maybe we need to check the remember entities storage mode and not enable durable keys until we find one where that's explicitly set to ddata?

Expected behavior

I should be able to use remember-entities without LMDB instances being created unless I'm using DData for storage.

Actual behavior

I get an LMDB database no matter what I do.

Environment

Windows and Linux.

[Aaronontheweb]

cc @OnurGumus

[Aaronontheweb]

One design idea I had to fix this code is to do this:

internal static ReplicatorSettings GetReplicatorSettings(ClusterShardingSettings shardingSettings)
{
    var config = Context.System.Settings.Config.GetConfig("akka.cluster.sharding.distributed-data")
        .WithFallback(Context.System.Settings.Config.GetConfig("akka.cluster.distributed-data"));
    var configuredSettings = ReplicatorSettings.Create(config);
    var settingsWithRoles = configuredSettings.WithRole(shardingSettings.Role);
    if (shardingSettings.RememberEntities && shardingSettings.RememberEntitiesStore == RememberEntitiesStore.DData)
        return settingsWithRoles; // only enable durable keys when using DData for remember-entities
    else
        return settingsWithRoles.WithDurableKeys(ImmutableHashSet<string>.Empty);
}

That would stop the DData directory from being created unless we're explicitly using RememberEntitiesStore.DData as the remember-entities provider.

Another approached I've considered, and this is not mutually exclusive, is making it possible to pass in a ReplicatorSettings value as part of the ClusterShardingSettings class. This would make it possible to disable durable keys even when you're using DData as the remember-entities mode (which might be useful during local dev or testing.)

[Aaronontheweb]

Have a beta of this going into v1.5.28-beta1

[Aaronontheweb]

Can confirm that this is still not a totally solved problem in v1.5.30 - said as much when we merged #7327

[Aaronontheweb]

I can validate that this DOES disable DData from writing to disk:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder.WithShardRegion<DrawingSessionActor>("drawing-session",
                (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
                new DrawingSessionActorMessageExtractor(), new ShardOptions()
                {
                    StateStoreMode = StateStoreMode.DData,
                    RememberEntities = true,
                    RememberEntitiesStore = RememberEntitiesStore.DData,
                    Role = clusterRoleName
                })
            // .WithDistributedData(options =>
            // {
            //     options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            // })
            .AddHocon("akka.cluster.sharding.distributed-data.durable.keys = []", HoconAddMode.Prepend);
        return builder;
    }
}

It looks like this is an Akka.Hosting issue with options.Durable = new DurableOptions() { Keys = [] }; // disable persistence not working as expected - and that's probably an easy fix.