Overlap KV cache update for WindowedKeyValueCache in DecoderOnlyPipelineState

[edgchen1]

Add support for overlapping the KV cache update for a pipelined model part with the graph execution of other pipelined model parts. It only applies to DecoderOnlyPipelineState with WindowedKeyValueCache.

For example, consider a model with two parts (graph[1] and graph[2]) that have KV caches.

This is the approach in this PR:

iter 1 graph[1] run | -
iter 1 graph[2] run | iter 1 graph[1] KV cache update
iter 2 graph[1] run | iter 1 graph[2] KV cache update
iter 2 graph[2] run | iter 2 graph[1] KV cache update
iter 3 graph[1] run | iter 2 graph[2] KV cache update

For comparison, this is the existing approach:

iter 1 graph[1] run
iter 1 graph[2] run
iter 1 graph[1] KV cache update
iter 1 graph[2] KV cache update
iter 2 graph[1] run
iter 2 graph[2] run
iter 2 graph[1] KV cache update
iter 2 graph[2] KV cache update

TODO measurements

[edgchen1]

note: moved WindowedKeyValueCache to windowed_kv_cache.h/cpp

[edgchen1]

note: there's already a main in test/main.cpp so we don't need one from gtest