Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remote dataset updated no longer tracks file changes #118

Open
tgbugs opened this issue Jun 17, 2024 · 2 comments
Open

remote dataset updated no longer tracks file changes #118

tgbugs opened this issue Jun 17, 2024 · 2 comments

Comments

@tgbugs
Copy link
Collaborator

tgbugs commented Jun 17, 2024

This breaks sparcron completely so we need to search for a solution.

@tgbugs
Copy link
Collaborator Author

tgbugs commented Jun 18, 2024

One possibility is to hit https://api.pennsieve.io/datasets/{dataset-id}/changelog/events, however that is going to cause a massive increase in the number of requests.

Another possibility would be to subscribe to notifications?

@tgbugs
Copy link
Collaborator Author

tgbugs commented Jun 18, 2024

example output from logging post, need to figure out where the secret is stored, and we will have to enable the hook on all datasets and provide an endpoint on cassava that can be called, but this seems like it will be more efficient in the long run, will also need to retain the integer organization and dataset ids for the lookup

(#(struct:header Host utils.olympiangods.org) #(struct:header X-Real-IP 34.193.208.98) #(struct:header X-Forwarded-For 34.193.208.98) #(struct:header X-Forwarded-Host utils.olympiangods.org) #(struct:header Connection close) #(struct:header Content-Length 247) #(struct:header Accept-Encoding identity) #(struct:header Content-type application/json) #(struct:header User-Agent python-urllib3/1.26.18))
{"datasetId": "280", "organizationId": "367", "eventCategory": "FILES", "eventType": "DELETE_PACKAGE", "eventDetail": {"id": 3958821, "nodeId": "N:package:e38a145a-8f0b-4394-a854-7e804bd5efa4", "name": "code_description_lol.xlsx", "parent": null}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant