Cluster restart question

[mbj4668]

I have a question about what can be expected after a cluster restart.

I have a cluster with three nodes, and the machine is a simple k-v store implemented as a map, with perhaps ~100 elements.
Everything is stable, and I stop all three nodes. Then they are all restarted. Some short time after ra:restart_server(Name, {Module, node()}) has returned ok, I do a leader_query to read a value that I know was in the kv store before the restart, but at this time it is not there (I get a proper no_such_key return value from my callback module, not an error). Then after timer:sleep(1000) I try the same leader_query, and now the item is there.

So my question is if this is expected?

[michaelklishin]

We cannot suggest much without logs from all nodes.

When you stop all nodes and start one, that node won't have an online majority. So at best it can return stale read results, and in general should be considered unavailable.

When you bring more nodes back online, it will take a moment for them to elect a new leader.

With Raft, availability is the price you pay for its consistency guarantees.

[mbj4668]

When you stop all nodes and start one, that node won't have an online majority. So at best it can return stale read results, and in general should be considered unavailable.

Right, but I expected leader_query to return an error in this case - is this not correct?

[michaelklishin]

It does return an error when I run a leader query using the README example on the only running cluster node.

[michaelklishin]

And after a restart, it tries to rejoin its peers and fails with a cluster formation error.

We do not guess in this community, please put together a set of steps to reproduce or a Common Test test.

[mbj4668]

We do not guess in this community

Please note that I tried to be clear that I was merely asking a question about the expected behavior. I don't think I expressed any guesses, and I certainly do not want anyone to guess anything.

If you think that the there might be a bug involved, I will try to put together a reproduction example.

[michaelklishin]

Sorry, that's the only reason I asked. We cannot know without seeing logs from all nodes and be able to track the sequence of actions/events.