Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Settings split 200+ results into separate scrapes #53

Open
funkonaut opened this issue Oct 30, 2020 · 2 comments
Open

Settings split 200+ results into separate scrapes #53

funkonaut opened this issue Oct 30, 2020 · 2 comments

Comments

@funkonaut
Copy link
Collaborator

For the eviction settings scraper the code should rescrape jobs that return an overflow of results (200+). The scrape jobs can be split up either recursively or into 3 day scrapes (assuming that no 3 days will have more than 200 cases). The first option is the preferred solution.

@alpiazza13 alpiazza13 self-assigned this Nov 2, 2020
@alpiazza13
Copy link
Collaborator

For the eviction settings scraper the code should rescrape jobs that return an overflow of results (200+). The scrape jobs can be split up either recursively or into 3 day scrapes (assuming that no 3 days will have more than 200 cases). The first option is the preferred solution.

After taking a look at the code, it seems as though the search is already split up into 1-day scrapes. I think it's very unlikely that any 1-day range will have over 200 results, so maybe this issue is unnecessary? lmk if you disagree though

@funkonaut
Copy link
Collaborator Author

That sounds good to me. We might want to add a new column to the DB and flag if it does but we can save that for future dev as I doubt that will happen.

@alpiazza13 alpiazza13 removed their assignment Nov 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants