Flaky test TestUpstreamProcessTaintedAuthority

[rturner3]

--- FAIL: TestUpstreamProcessTaintedAuthority (60.05s)
    manager_test.go:411: 
        	Error Trace:	/home/runner/work/spire/spire/pkg/server/ca/manager/manager_test.go:411
        	Error:      	deadline reached
        	Test:       	TestUpstreamProcessTaintedAuthority
    manager_test.go:1423: 
        	Error Trace:	/home/runner/work/spire/spire/pkg/server/ca/manager/manager_test.go:1423
        	            				/home/runner/work/spire/spire/pkg/server/ca/manager/manager_test.go:1418
        	            				/home/runner/work/spire/spire/pkg/server/ca/manager/manager_test.go:414
        	Error:      	Received unexpected error:
        	            	datastore-sql: context deadline exceeded
        	Test:       	TestUpstreamProcessTaintedAuthority
FAIL
FAIL	github.com/spiffe/spire/pkg/server/ca/manager	60.670s

https://github.com/spiffe/spire/actions/runs/12816442140/job/35737575426?pr=5718

[sorindumitru]

Looked a bit into this when I saw this issue before. The error seems to be due to getting a "no such table" error from go-sqlite3 somewhere else in the stack, here it is with some extra logging:

2025/01/20 22:12:46 ERROR Could not prepare x509-ca error="failed to fetch bundle: rpc error: code = Unknown desc = datastore-sql: no such table: bundles"
--- FAIL: TestUpstreamProcessTaintedAuthority (60.05s)
    manager_test.go:411:
                Error Trace:    /home/sorin/github/spiffe/spire/pkg/server/ca/manager/manager_test.go:411
                Error:          deadline reached
                Test:           TestUpstreamProcessTaintedAuthority
    manager_test.go:1425:
                Error Trace:    /home/sorin/github/spiffe/spire/pkg/server/ca/manager/manager_test.go:1425
                                                        /home/sorin/github/spiffe/spire/pkg/server/ca/manager/manager_test.go:1420
                                                        /home/sorin/github/spiffe/spire/pkg/server/ca/manager/manager_test.go:414
                Error:          Received unexpected error:
                                datastore-sql: context deadline exceeded
                Test:           TestUpstreamProcessTaintedAuthority
FAIL
FAIL    github.com/spiffe/spire/pkg/server/ca/manager   205.087s
FAIL

https://github.com/mattn/go-sqlite3 has something to say about this error in the README, but we're already doing what it says. Maybe it's a bug in sqlite3 or go-sqlite3.

I add a FetchBundle call to the fakedatastore upon creation and that didn't check didn't fail but the test itself still failed with the same error. I know that if the last connection to the in-memory database closes the database gets removed, but that shouldn't happen in our case since we should have 2 idle connections.

If we change the fakedatastore to use a file in a temp directory, it looks better. go test -failfast -timeout 1200s -count 65536 ./pkg/server/ca/manager/ -run TestUpstreamProcessTaintedAuthority ran successfully for me twice. So that could be a workaround to the issue.