HA: Limit max open database connections to 1

[yhabteab]

Previously, the HA feature was allowed to open max_connections database connection in parallel to other Icinga DB components. Meaning, Icinga DB wasn't limited to the configured max_connections, but effectively to 2 * max_connections. Additionally, this masked a serious bug in the HA#realize() method, where we start a new transaction after each retry without rolling back in case of an error, leading to connections not being released before exceeding the ctx deadline.

[yhabteab]

The HA transaction is rolled back correctly with:

• Abort HA Realization Logic After Timeout #800

[yhabteab]

This is the proof that we actually leak database connections but only for as long as the context is not cancelled.

package main

import (
	"context"
	"github.com/icinga/icinga-go-library/logging"
	"github.com/icinga/icingadb/internal/command"
	"go.uber.org/zap"
	"log"
	"time"
)

func main() {
	cmd := command.New()

	db, err := cmd.Database(logging.NewLogger(zap.NewNop().Sugar(), time.Second))
	if err != nil {
		log.Fatal(err)
	}
	db.SetMaxOpenConns(3)

	go func() {
		time.Sleep(1 * time.Second)

		ctx, _ := context.WithDeadline(context.Background(), time.Now().Add(10*time.Second))
		_, _ = db.BeginTxx(ctx, nil)
		return
	}()
	go func() {
		time.Sleep(1 * time.Second)

		ctx, _ := context.WithDeadline(context.Background(), time.Now().Add(10*time.Second))
		_, _ = db.BeginTxx(ctx, nil)
		return
	}()

	time.Sleep(3 * time.Second)

	ctx, cancelFunc := context.WithDeadline(context.Background(), time.Now().Add(2*time.Second))
	defer cancelFunc()
	_ = db.PingContext(ctx)
	log.Printf("OpenConnections: %v", db.Stats().OpenConnections)
	log.Printf("InUse: %v", db.Stats().InUse)
}

~/Workspace/go/icingadb (set-ha-conn-limit ✗) go run ./cmd/decoder/decoder.go --config config.example.yml
2024/10/23 16:19:59 OpenConnections: 3
2024/10/23 16:19:59 InUse: 2

[lippserd]

I would remove this information as it is an implementation detail. Sure, users might be surprised if there are 17 database connections, but I doubt anyone will ever count them.

Instead, we could reduce the size of the main pool by one. But I wouldn't attach too much importance to that.

[yhabteab]

Since the size of the main pool is initialized in the library, we don't want to adjust it just because of Icinga DB's HA feature. Thus, I removed the highlighted info box.

[lippserd]

The example configuration would also have to be adjusted. But please just remove it completely. It is irrelevant.

[yhabteab]

As we discussed offline, I have completely removed the extra info.

[lippserd]

I want to summarize our discussion: With this PR, we currently have only one connection that isn't tied to max_connections. However, there's a chance more could be added or changed, leaving it up to the user to determine and maintain the correct max_connections value if the goal is to truly limit Icinga DB database connections to that number. Therefore, if max_connections is intended to represent the actual maximum connections, it should be implemented accordingly, rather than documented that some connections are unaffected by that setting.