Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to reduce RAM usage #51

Open
wants to merge 5 commits into
base: next
Choose a base branch
from

Commits on Jan 20, 2012

  1. Merge branch 'next'

    chriskite committed Jan 20, 2012
    Configuration menu
    Copy the full SHA
    4b378d5 View commit details
    Browse the repository at this point in the history
  2. Merge branch 'next'

    chriskite committed Jan 20, 2012
    Configuration menu
    Copy the full SHA
    531d771 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2012

  1. page_queue is constrained by a size supplied in options, default 100.

    This alleviates a problem we experienced with very responsive sites (wikipedia)
    and a moderate per-page processing time. The page_queue would grow much faster
    than it could be drained, using more and more RAM.
    
    This change means that the queue grows until full, at which point the
    Tentacles will block until the queue shrinks.
    
    This does have the impact of slowing the crawl in some cases.
    Jamie Cobbett committed Apr 20, 2012
    Configuration menu
    Copy the full SHA
    487601b View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2012

  1. Add option 'discard_page_data' to allow user to further decrease stor…

    …age use.
    Jamie Cobbett committed Apr 23, 2012
    Configuration menu
    Copy the full SHA
    2166b99 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2012

  1. Revert "Add option 'discard_page_data' to allow user to further decre…

    …ase storage use."
    
    This reverts commit 2166b99.
    Jamie Cobbett committed Apr 26, 2012
    Configuration menu
    Copy the full SHA
    8b51d29 View commit details
    Browse the repository at this point in the history