robotspy 0.12.0

• Fixed parser stalling due to agent duplication #212

robotspy 0.11.0

• Fixed network timeout issue #211

robotspy 0.10.0

• Fixed bugs in the URL path pattern matching ('?' is now handled correctly as the character '?' instead of matching any one character)
• Added tests 541230 and 541230 from Google project https://github.com/google/robotstxt-spec-test
• Thank you, @kox-solid, for being inquisitive and pointing to bugs in robotspy
• robotspy is available as a Python package at https://pypi.org/project/robotspy/

robotspy 0.9.0

• Updated the parser to behave like the Google robots parser. It now handles the product token in the user-agent line up to the last correct character instead of discarding it. See issue #209 for more details.

robotspy 0.8.0

• Addressed an issue raised when a robots.txt file is not UTF-8 encoded (thank you, @tumma72, for spotting the problem and providing a suggestion for a fix - #200)
• Added a user agent to fetch the robots.txt, as some websites, such as pages hosted on Cloudflare, may return a 403 error.
• Updated the documentation to link to RFC 9309, Robots Exclusion Protocol (REP).
• Added a GitHub action job to execute the tests against Python versions 3.8 to 3.12.