Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal metakg mishandling #896

Open
tokebe opened this issue Oct 29, 2024 · 0 comments
Open

Internal metakg mishandling #896

tokebe opened this issue Oct 29, 2024 · 0 comments
Assignees
Labels
bug Something isn't working On CI Related changes are deployed to CI server

Comments

@tokebe
Copy link
Member

tokebe commented Oct 29, 2024

Context: #851, #703

Presently, the globally-loaded MetaKG is supposed to be used instead of re-building it from the on-disk smartapi specs to avoid read/write collisions. There are 3 problems:

  1. Only one BTE process attempts to update the on-disk smartapi specs. This means that the metakg only gets globally loaded on one of the 4 live BTE processes, and the other 3 still attempt to rebuild the metakg from disk every time.
  2. The global metakg is incorrectly used on the /v1/meta_knowledge_graph endpoint. It's being returned directly instead of being passed through the meta_knowledge_graph controller's code to convert it into a TRAPI MetaKG.
  3. The global metakg is only used on the /v1/meta_knowledge_graph endpoint, and not the /v1/smartapi/:smartapiID/meta_knowledge_graph or /v1/team/:teamName/meta_knowledge_graph endpoints

So, we're in an interesting situation: as a result of these problems, BTE responds with an invalid MetaKG in 1/4 of queries to its main /v1/meta_knowledge_graph endpoint on live instances. Additionally, it's still theoretically running into read/write collisions with the on-disk specs because 3/4 of its live instances are still exclusively interacting with it rather than keeping a metakg in-memory.

So, we need a few fixes:

  1. Each BTE process needs to load a metaKG from specs in each process regardless of whether that process should be updating the local smartapi specs. Which means we need a way to tell that the file is not being actively written to.
  2. The /v1/meta_knowledge_graph endpoint needs to make use of the meta_knowledge_graph handler (which needs to be modified to optionally accept a pre-loaded metaKG and skip reading it from disk)
  3. The other two endpoints need the same treatment as the main endpoint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working On CI Related changes are deployed to CI server
Projects
None yet
Development

No branches or pull requests

2 participants