Skip to content

Commit

Permalink
add update description
Browse files Browse the repository at this point in the history
  • Loading branch information
CourierKyn committed Oct 24, 2019
1 parent ebf42be commit 4e9a135
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 4 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# jkcrawler

*2019/10/24 更新:加入哔哩哔哩相簿*

使用 Scrapy 写成的 JK 爬虫,图片源自哔哩哔哩、Tumblr、Instagram,以及微博、Twitter (待完成)

启动爬虫:
Expand Down
14 changes: 10 additions & 4 deletions genREADME.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,16 @@ def genREADME():
'```shell script',
]

if not os.path.exists('config/'):
os.mkdir('config')
os.system('scrapy list > config/scrapylist.txt')
with open('config/scrapylist.txt') as f:
if os.path.exists('data/update.md'):
with open('data/update.md') as f:
update = f.read().rstrip()
head.insert(1, update)
head.insert(1, '')

if not os.path.exists('data/'):
os.mkdir('data')
os.system('scrapy list > data/scrapylist.txt')
with open('data/scrapylist.txt') as f:
spiders = f.read().splitlines()
lines = ['scrapy crawl {spider} -o data/{spider}.jsonlines'.format(spider=spider) for
spider in spiders]
Expand Down

0 comments on commit 4e9a135

Please sign in to comment.