Handling of ra machine failures

[mbj4668]

I have noticed that the system ends up in a bad state (?) if the apply callback crashes (erlang:error or erlang:exit).

If apply crashes, the ra_server_proc proccess exits, and is restarted by its supervisor (ra_server_sup), but then it fails again, and the supervisor reaches its max restart, and then its supervisor (ra_server_sup_sup) detects this. However, the child ra_server_sup has restart strategy temporary, so it just ignores this error. Here's an attempt to illustrate the supervision tree:

The system is called store.

ra_sup [one_for_one, max 1 restarts in 5 secs]
  +-- PERM ra_systems_sup [one_for_one, max 1 restarts in 5 secs]
  |          +-- PERM <0.195.0>/ra_system_sup [one_for_all, max 1 restarts in 5 secs]
  |                     +-- PERM ra_store_server_sup_sup/ra_server_sup_sup [simple_one_for_one, max 1 restarts in 5 secs]
  |                     |          +-- TEMP <0.241.0>/ra_server_sup [one_for_one, max 2 restarts in 5 secs]
  |                     |                     +-- TRAN store_ra/ra_server_proc
  |                     +-- PERM ra_store_log_sup/ra_log_sup [one_for_all, max 5 restarts in 5 secs]
  |                     |          +-- PERM <0.205.0>/ra_log_wal_sup [one_for_one, max 1 restarts in 5 secs]
  |                     |          |          +-- PERM ra_store_log_wal/ra_log_wal
  |                     |          +-- PERM ra_store_segment_writer/ra_log_segment_writer
  |                     |          +-- PERM ra_store_log_meta/ra_log_meta
  |                     |          +-- PERM <0.200.0>/ra_log_pre_init
  |                     +-- PERM ra_store_log_ets/ra_log_ets
  +-- PERM ra_file_handle
  +-- PERM ra_metrics_ets
  +-- PERM ra_machine_ets

I expected the error to propagate and eventually terminate the application. What is the intended way to handle these kinds of errors? My current workaround is to find the temporary supervisor (<0.241.0> above) and monitor it from another process, but this requires peeking into the internal state of ra, which doesn't seem quite right.

[kjnilsson]

An error in a single ra server can't terminate the application as Ra is meant to hold many independent Raft clusters not just one.

Ra machines must be written not to crash, i.e. if you do anything that may error you need to catch inside your apply implementation. NB: the errors need to be deterministic as all code inside apply always needs to result in exactly the same state on all members.

[mbj4668]

I understand that it must be possible to have independent clusters. I also understand that machines should not crash. But the question is in general what happens if it crashes anyway, i.e., due to a bug in the code (user code or ra code). In my use case, I just have one cluster, and if there are any crashes due to bugs there, I want to handle these errors. With the current design, I am not able to even detect that there was a crash (unless I do the workaround I described above).

Perhaps you could consider adding a function (re_)start_server_link, which perhaps only would work for local servers, but would let the user be able to put the process under its own supervisor?