Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move bagger functionality to exports module #81

Open
65278 opened this issue Feb 10, 2020 · 0 comments
Open

Move bagger functionality to exports module #81

65278 opened this issue Feb 10, 2020 · 0 comments

Comments

@65278
Copy link
Contributor

65278 commented Feb 10, 2020

Schaufel is supposed to take over some of the jobs our postgres insert trigger used to manage. What follows is a list of places that need touching (of course the real scope of what we implement is up to discussion):

  • duplication of messages to postgres is done by having a second meta data structure. this ought to be replaced by a refcounter system
  • transformations done by the exports system should be transformed to filters applyable to the queue. this shall be possible post queue_add or pre queue_get. the aim is to avoid double work within modules
  • the table name to copy into needs to be determined by the fields in the json message
  • to transport this tablename we probably require metadata in the queue (this may also be required to forward kafka message headers)
  • a data structure needs to hold all buffers for copy. this data structure ought to be easy to iterate over, easy to alter, and have a good access time. the insert trigger at the moment has at best O(n) behaviour, so anything faster than O(n) will do. Cache locality would be preferred
  • these buffers need to be committed periodically and when full
  • in case of failure in libpq, data must not be lost. see Postgres modules don't check libpq errors #80
  • a jsonb/json binary insert transformation needs to be added to aforementioned filters. other datatypes are nice to have
  • values that are dereferenced from json ought to be deletable from its jsonc data structure
  • what remains of the original json shall also be inserted into postgres
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant