Replies: 1 comment
-
@basedygt I guess the above workaround is way to go, as |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
i know we can use
-ct
argument to specify maximum crawl time but this options sets the crawl time for entire list of URLs.I want to know how to set
-ct
for each URL instead of all URLs since i can't find any built in options inside the tool for it nor any workaround like below command worked:The reason why this is a handy feature as many websites root domain has endless contents and the scraper fails to scrape it's subdomain automatically due to this issue.
Beta Was this translation helpful? Give feedback.
All reactions