README.md

cnki-spyder-tool

一个中国知网的爬虫工具，给定作者，可以获得该作者的所有文献的题录

范例

使用时需要将文件tool.py放在正确的位置

import sys
sys.path.append(".")
import tool as cnki
from pprint import pprint as fprint


if __name__ == "__main__":
    tmp = cnki.get_doc_url_set("牛丽慧","南京大学") 
    fprint(tmp)
    res_file = open("res.csv","a",encoding="utf-8",newline="")
    for item in tmp:
        cnki.get_doc_bibilo(item,res_file)
    res_file.close()

文档

Help on module tool:

NAME
    tool

DESCRIPTION
    Eng:
    This module is written for scraping bibilograph of literature from cnki.
    It relies on python modules below:
    1. requests
    2. bs4
    you can easily get them using tool like pip.
    
    中文：
    这个模块用来爬取中国知网的文献题录。
    它依赖于以下的Python模块：
    1.requests
    2.bs4
    你可以使用像pip这样的工具容易地获取它们。

FUNCTIONS
    get_doc_bibilo(doc_url, res_file)
        this function return bibliograph of the specified doc url by writting into the specified file
        
        Args:
            doc_url: specified doc url
            res_file: target res file
    
    get_doc_url_set(author_name, unit_name)
        This function simulates the behavior that you open the website:
        http://yuanjian.cnki.com.cn/
        and enter the dictionary:
        {"作者":"foo","作者单位":"bar"}
        and click the search button
        as you get the response page, you can get url set of all the page,
        it needs support of function:
        def get_next_page_doc_url_set(author_name,page_num,unit_name):
        to deal with the demond of page turning
        
        Args:
            author_name:  author's name
            unit_name: author's unit name
           
        Returns:
            return set of doc urls
    
    get_next_page_doc_url_set(author_name, page_num, unit_name)
        this function support the function:
        def get_doc_url_set(author_name,unit_name):
        get urls of specified page.
        
        Args:
            author_name: author's name
            page_num: specified page number
            unit_name: author's unit name
            
        Return:
            return set of doc urls

FILE
    /root/workbench/cnki-spyder-tool/tool.py