[server] ReadWriteLock on LeaderFollowerState

[KaiSernLim]

Background

TODO:

Changes

• Added a ReadWriteLock to the LeaderFollowerState in PartitionConsumptionState
• In produceToStoreBufferServiceOrKafka() on the consumer's code path, the read lock must be acquired for the duration of the message's processing in order for the condition of shouldProcessMessage() to hold. Also applies to produceToStoreBufferServiceOrKafkaInBatch().
  • Another thread trying to modify LeaderFollowerState as part of a state transition would need to wait for this consumer thread to finish processing the message and release the read lock.
• Added a unit test testShouldProcessRecord() which simulates the scenario where the consumer thread processes a batch of polled messages while another thread modifies the leader-follower state in the PCS. It's specifically testing a follower to leader transition and verifying that the leader-follower state in the PCS can't be modified while the consumer thread is processing messages.

Correctness

TODO:

Performance Impact

• Performance is bottlenecked by produceToStoreBufferServiceOrKafka(), because that is the only location where the lock is held for a long period of time. If a writer is waiting on the write lock, it must wait for all readers to release their locks. All future readers will also need to wait until the writer is done because the writer has priority, so they are also bottlenecked by produceToStoreBufferServiceOrKafka().
• Readers should not affect the performance of other readers. Therefore, there should be no performance impact until a state transition occurs, and the state needs to be overwritten.
• Since the writer simply needs to set an enum value, the critical section should be instantaneous once it manages to acquire the lock, and any slowdown while locking the writer lock should be minimal.

How was this PR tested?

CI

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.

[sixpluszero]

What does this true mean?

[KaiSernLim]

The parameter is for whether the lock is in fair-mode or not (docs link).

Basically, fair-mode will prioritize write locks trying to acquire the lock to ensure that the write lock doesn't get starved. More specifically, it'll respect approximate arrival-order policy whereas non-fair will have no order of entry guarantees.