Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add graph-based index merging interface #318

Open
inabao opened this issue Jan 13, 2025 · 0 comments
Open

add graph-based index merging interface #318

inabao opened this issue Jan 13, 2025 · 0 comments
Assignees
Labels
kind/feature New feature or request version/0.14

Comments

@inabao
Copy link
Collaborator

inabao commented Jan 13, 2025

  • algorithm
  • perf ref

Background
Currently, the cost of building graphs is very high, and to ensure that the insertion latency and fault recovery are acceptable, we typically want to keep the indexes that support real-time writes not too large. This leads to the need to frequently generate new small indexes, and we need to periodically rebuild indexes in the background, which significantly increases the indexing overhead.

Describe the solution you'd like
We hope to add an index merging feature to provide fast merging capabilities for indexes. At the same time, the cost of merging indexes (reusing the original graph structure) is much less than rebuilding the index.

Interface
Here is a simple interface that takes a new index, parameters, and a list of indexes to be merged:

    // Engine
    /**
     * @brief Merges multiple graph indexes into a single index.
     *
     * This function takes a specified `name` and `parameters`, along with a vector of
     * `sub_indexes`, and attempts to merge them into a single graph index. The result is
     * either a shared pointer to the newly merged `Index` or an `Error` object that describes
     * any failures encountered during the merging process.
     *
     * @param name The name assigned to the merged graph index, which will represent the
     *              combined structure of the provided sub-indexes.
     * @param parameters A JSON-like string containing various parameters that dictate 
     *                   the merging behavior and properties of the resulting index.
     * @param sub_indexes A vector of shared pointers to the `Index` objects that are to be merged.
     * @return tl::expected<std::shared_ptr<Index>, Error> An expected value that contains either
     * a shared pointer to the successfully merged `Index` or an `Error` detailing
     * why the merge operation failed.
     */
    tl::expected<std::shared_ptr<Index>, Error>
    MergeGraphIndex(const std::string& name,
                    const std::string& parameters,
                    const std::vector<std::shared_ptr<Index>>& sub_indexes);

Alternatively, we can implement an interface on the Index.

    // Index
    /**
     * @brief Merges multiple graph indexes into a single index.
     *
     * This function takes a vector of `sub_indexes`, and attempts to merge them into a single 
     * graph index. The result is either a shared pointer to the newly merged `Index` or an `Error` 
     * object that describes any failures encountered during the merging process.
     * 
     * @param sub_indexes A vector of shared pointers to the `Index` objects that are to be merged.
     * @return tl::expected<std::shared_ptr<Index>, Error> An expected value that contains either
     * a shared pointer to the successfully merged `Index` or an `Error` detailing
     * why the merge operation failed.
     */
    tl::expected<std::shared_ptr<Index>, Error>
    Merge(const std::vector<std::shared_ptr<Index>>& sub_indexes);

Implementation
The specific implementation will be updated later.

@inabao inabao assigned wxyucs and inabao and unassigned wxyucs Jan 13, 2025
@inabao inabao added kind/feature New feature or request version/0.14 labels Jan 13, 2025
@inabao inabao changed the title 新增基于图的索引合并功能 add graph-based index merging interface Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request version/0.14
Projects
None yet
Development

No branches or pull requests

2 participants