perf: Optimize transaction write memtable speed

[sollhui]

(I only read some modules of tonbo, maybe I have some misunderstandings)

I noticed that tonbo can be inserted in two ways, transactional and non transactional:

1. For non transactional writes:

Memtable may be read and written simultaneously, and using skiplist can effectively balance read and write operations

2. For transaction writing:

It will first be written in the local BTreeMap to the transaction, and only after the commit is committed and there are no conflicts, will the memtable writing actually take place. In this case:

• There will be no queries for this transaction.
• Other transactions will not read the memtable of this transaction because I noticed it was a snapshot read.

So, for this situation, there is no need to use a balanced read-write data structure, just use an vector for append and sort during flush. Based on my experience, the write performance will be greatly improved.

If the suggestion is beneficial, I would be happy to complete this task because I think tonbo is an interesting project. :)

[ethe]

There will be no queries for this transaction.

Transaction still support get and scan, using a balanced tree might be helpful when reading from a transaction? BTW we got inspiration from mini-lsm, Tonbo replaces skip map with b-tree here because Tonbo's transaction does not support parallel read / write.

If I am misunderstanding, please be kind to share more about your thoughts.

[sollhui]

Transaction still support get and scan, using a balanced tree might be helpful when reading from a transaction? BTW we got inspiration from mini-lsm, Tonbo replaces skip map with b-tree here because Tonbo's transaction does not support parallel read / write.

If I am misunderstanding, please be kind to share more about your thoughts.

I am not referring to the use of a B-tree structure to store transactions data during the transaction process. Using a B-tree is reasonable, but when a transaction commits and writes the memtable, it is written in the same way as non transaction writing, using skiplist. According to my description above, it can be replaced with a vector,The code location is:

pub async fn commit(mut self) -> Result<(), CommitError<R>> {
    // ...get lock and  check conflict...

    let len = self.local.len();
    let is_excess = match len {
        0 => false,
        1 => {
            let new_ts = self.snapshot.increase_ts();
            let (key, record) = self.local.pop_first().unwrap();
            // write memtable
            // for transcational write, use vector or array is better
            // for non transcational write, use skiplist is better
            Self::append(self.snapshot.schema(), LogType::Full, key, record, new_ts).await?
        }
    };
}

[ethe]

I got it, it does make sense, let's have a try.