Reuters poller business logic #92
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bryophyta
force-pushed
the
pf/reuters-poller
branch
from
3ce7d8c to
7030e94
Compare
bryophyta
force-pushed
the
pf/reuters-poller
branch
2 times, most recently
from
1d949b5 to
4889fdc
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
from
fb0920c to
f6de1a7
Compare
bryophyta
force-pushed
the
pf/reuters-poller
branch
2 times, most recently
from
c983801 to
684492e
Compare
bryophyta
force-pushed
the
pf/reuters-poller
branch
3 times, most recently
from
d644737 to
ef43389
Compare
This was referenced Jan 13, 2025
bryophyta
force-pushed
the
pf/reuters-poller
branch
from
ef43389 to
04a5b6c
Compare
bryophyta
force-pushed
the
pf/reuters-poller
branch
from
04a5b6c to
8a7e365
Compare
bryophyta
force-pushed
the
pf/reuters-poller
branch
from
8a7e365 to
e358ff1
Compare
In conversation with Reuters they said that there's currently no way of
requesting just the stories that have changed since last request
('endCursor' shouldn't be relied upon for this).
So we should search for a time period that overlaps with our last
request, and page through the results. The ingestion lambda handles
deduplication now, so no need to deduplicate here unless we spot
performance issues, for instance.
bryophyta
commented
Jan 21, 2025
|
|
||
| let accessToken = ACCESS_TOKEN ?? (await auth(CLIENT_ID, CLIENT_SECRET)); | ||
|
|
||
| async function fetchWithReauth(query: string) { |
There was a problem hiding this comment.
this function is defined within the scope of the main poller function so that it can read and mutate accessToken.
I went back and forth on whether to handle this in a more 'pure' way, but as mentioned in the PR description this felt like a good balance in context. Open to other perspectives though!
andrew-nowak
approved these changes
Jan 22, 2025
| 'https://api.thomsonreuters.com/auth/reutersconnect.contentapi.read https://api.thomsonreuters.com/auth/reutersconnect.contentapi.write'; | ||
| const authUrl = 'https://auth.thomsonreuters.com/oauth/token'; | ||
| const grantType = 'client_credentials'; | ||
| const audience = '7a14b6a2-73b8-4ab2-a610-80fb9f40f769'; |
There was a problem hiding this comment.
just double checking that this value is fine to be made public? iirc it's a value that's shared by all customers, is that right?
There was a problem hiding this comment.
yep that's right -- it's from the Reuters API documentation 👍
Add more fields to JSON view in UI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
First pass at business logic for Reuters feed.
Steps:
1. Auth
2. Request feed via search for the last
n*1.2seconds, wherenis the frequency at which the poller runs.3. Pass the most recent auth token back
The poller lambda wrapper, as of #99, will check this value and update the Secret value if needed.
The implementation of (1) adds a little complexity and mutation that it would be nice to avoid, but the API guidelines suggest we shouldn't request new access tokens too often, so we should respect that. It could be refactored so that e.g.
fetchWithReauthandfetchAllPageswere defined outside the scope of the main function, and hadaccessTokenpassed as an argument, but on reflection it felt to me as though declaringaccessTokenvialetmakes it more explicit a) that it can mutate, and b) which functions can mutate it. I'm open to other perspectives though.Script
Also adds a script to help get raw item data from the Reuters API, to help with debugging.
How to test
How can we measure success?
Have we considered potential risks?
Images
Accessibility