Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

集群下出错 'WoaiduSpider' object has no attribute 'downloaded' #23

Open
CoolView opened this issue Aug 20, 2015 · 1 comment
Open

Comments

@CoolView
Copy link

错误一:

Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/home/zb/workpace/new/distribute_crawler-master/woaidu_crawler/woaidu_crawler/pipelines/mongodb_book_file.py", line 134, in process_item
        dfd.addCallback(self.item_completed, item, info)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 306, in addCallback
        callbackKeywords=kw)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 295, in addCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/home/zb/workpace/new/distribute_crawler-master/woaidu_crawler/woaidu_crawler/pipelines/mongodb_book_file.py", line 274, in item_completed
        if self.item_download[item['original_url']]:
    exceptions.KeyError: 'http://www.woaidu.org/book_196826.html'

错误二:

Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 839, in _cbDeferred
        self.callback(self.resultList)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
        self._startRunCallbacks(result)
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/home/zb/workpace/new/distribute_crawler-master/woaidu_crawler/woaidu_crawler/pipelines/mongodb_book_file.py", line 132, in process_item
        dlist = [self._process_request(r, info) for r in requests]
      File "/usr/lib/pymodules/python2.7/scrapy/contrib/pipeline/media.py", line 53, in _process_request
        if fp in info.downloaded:
    exceptions.AttributeError: 'WoaiduSpider' object has no attribute 'downloaded'
@CoolView
Copy link
Author

修改woaidu_crawler/pipelines/mongodb_book_file.py下130的
line 130:info = self.spiderinfo.spider
info = self.spiderinfo
(不保证正确)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant