Support multi spans and on Document Fetcher.

[jsimnz]

Currently, we only support single spans on our document fetcher, moreover, it only supports prefix scans and ignores the "end" range stop point.

To make the fetcher more efficient we need to support mutlispans and proper range scans.

The blocking issue is go-datastore#116. Needs proper architecture/implementation on the interface package, and a series of datastore implementations. That issue will give context.

[jsimnz]

hey @AndrewSisley peeked at some of the commits youre working on for this issue in the sisley/perf/I60-badger-multi-scan branch, and I had some thoughts before you got too far into things.

As far as I can tell, you are re-binding the dependancy the internals of the DB has on badger itself. Much of the goal of relying on the ds.Datastore and the core/DSReaderWriter interfaces is to avoid such a tight coupling.

Specifically, updating the core/client interfaces

defradb/client/core.go

Line 40 in a463b6b

badgerds.IterableTxn

I think the multispan stuff should be applied to either

1. A more foundational level of the ds.Datastore interface workflow, and bubble up in the specific datastore implementations
   or
2. A short term approach utilizing the existing prefix query structures of the datastore query package and re-initializing those structures.

[AndrewSisley]

Ah don't worry about that - that is an interface I defined in the v3 package, that is not specific to badger db - it just lives there in the extreme-short-term because thats where I happened to write it down first :) none of the defra stuff in that branch should currently be (directly) dependent on any badger stuff.

If I'm understanding what you mean correctly at least :) Will have a re-read once I'm properly working again (just saw this in my inbox)

[AndrewSisley]

Marking as blocked until the benchmarks are ready - code is not properly cleaned up yet (re readability/location/etc), but the main happy path should be fairly static now computationally and we should find out whether it is worth it performance wise before continuing