Skip to content

v0.4.0 - Add Crawl-delay

Latest
Latest
Compare
Choose a tag to compare
@micahcochran micahcochran released this 27 Sep 16:55
v0.4.0
674ec47
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

From PR #11

  • Adds support for Crawl-delay directive in robots.txt (Note: Request-rate is not supported.).
  • Replaces reppy with the Python standard library urllib.robotparse
  • Typing syntax improvements.
  • Updates made to support recipe-scrapers library >=14.48.0
  • Adds Foodista to the default list of sites to scrape.
  • Adds requirements-dev.txt for development. This is mainly typing.
Assets 2
Loading