I have done some workaround across implementing ActiveMq to the messagebus so would like to contribute#350

Open

amitsing89 wants to merge 332 commits intoexception-dbwfrom master

+6,529-3,000

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 1, 2017

Commits on Nov 2, 2017

Commits on Dec 27, 2017

Commits on Jan 24, 2018

Merge pull request #306 from clarksun/patch-1
sibiryakov
authored
Merge pull request #307 from clarksun/patch-2
sibiryakov
authored
Merge pull request #309 from clarksun/patch-4
sibiryakov
authored
hbase states table option
sibiryakov
committed
reducing logging verbosity
sibiryakov
committed
Merge pull request #301 from scrapinghub/states-table
sibiryakov
authored
Send crawl stats to Kafka message bus

vshlapakov
authored and
sibiryakov
committed
Add Mesos task id as a tag for DB worker metrics

vshlapakov
authored and
sibiryakov
committed
Send backend stats to message bus

vshlapakov
authored and
sibiryakov
committed
No need to setup stats log partitions

vshlapakov
authored and
sibiryakov
committed
Decrease log-level for stats producer

vshlapakov
authored and
sibiryakov
committed
lru cache for HBaseState

vshlapakov
authored and
sibiryakov
committed

Commits on Jan 30, 2018

codecs tests fixed
sibiryakov
committed
other tests fix
sibiryakov
committed
moving stats to worker submodule
sibiryakov
committed
base stats log, zmq stats log
sibiryakov
committed
Merge pull request #313 from scrapinghub/lru-cache-states
sibiryakov
authored
Merge pull request #312 from scrapinghub/stats-log
sibiryakov
authored
Split DB worker to components based on threads

vshlapakov
authored and
sibiryakov
committed
Fix attribute error related with job_id

vshlapakov
authored and
sibiryakov
committed
Run only batchgen via thread

vshlapakov
authored and
sibiryakov
committed
Polish changes and fix minor issues

vshlapakov
authored and
sibiryakov
committed
Minor code simplification

vshlapakov
authored and
sibiryakov
committed
Initial job_id must be 0 for db worker

vshlapakov
authored and
sibiryakov
committed
Smarted backoff for batchgen component

vshlapakov
authored and
sibiryakov
committed
Do back-off if no requests to schedule

vshlapakov
authored and
sibiryakov
committed
max_request_size is set to 4Mb
sibiryakov
committed
Update kafkabus producers settings

vshlapakov
authored and
sibiryakov
committed
Move values to constants

vshlapakov
authored and
sibiryakov
committed
more precise scoring, encoding from scrapy, hash for request

vshlapakov
authored and
sibiryakov
committed

Commits on Apr 6, 2018

Commits on Apr 9, 2018

Commits on Apr 12, 2018

string formatting corrected in the connection information being logged
akshayphilar
committed

Commits on Apr 23, 2018

db worker test fix
sibiryakov
committed
Merge pull request #314 from scrapinghub/dbw-components-teardown
sibiryakov
authored
Setting for splitting partitions b/w batchgens

vshlapakov
authored and
sibiryakov
committed
conflicts resolved

vshlapakov
authored and
sibiryakov
committed
Config option to pass partitions

vshlapakov
authored and
sibiryakov
committed
No need new setting

vshlapakov
authored and
sibiryakov
committed
Minor renaming

vshlapakov
authored and
sibiryakov
committed
Wrap sending message logic

vshlapakov
authored and
sibiryakov
committed
Catch and print shutdown errors

vshlapakov
authored and
sibiryakov
committed
Non-exhausted generators must be closed manually

vshlapakov
authored and
sibiryakov
committed
conflicts resolved

vshlapakov
authored and
sibiryakov
committed
Provide SW states cache hits ratio

vshlapakov
authored and
sibiryakov
committed
Provide flushed batches count for SW states

vshlapakov
authored and
sibiryakov
committed
Simplify state stats using with a variable

vshlapakov
authored and
sibiryakov
committed
Provide stats for SW state cache evictions

vshlapakov
authored and
sibiryakov
committed
Skip links-extracted when reached limit for domain

vshlapakov
authored and
sibiryakov
committed
no need to flush on every batch
sibiryakov
committed
create_request method added to manager, etc
sibiryakov
committed
removed domain info logic from hbase.queue, domain mw enabled by default
sibiryakov
committed
enabling domain fingerprint mw by default
sibiryakov
committed

Commits on May 8, 2018

Commits on May 14, 2018

Commits on May 15, 2018

first draft version
sibiryakov
committed

Commits on May 23, 2018

Commits on May 25, 2018

proper init hierarchy
sibiryakov
committed

Commits on May 28, 2018

Commits on May 30, 2018

Commits on Jun 1, 2018

fixing distributed run mode
sibiryakov
committed

Commits on Jun 4, 2018

auto_start/test_mode attrs refactor
sibiryakov
committed

Commits on Jun 18, 2018

Commits on Jul 12, 2018

Commits on Jul 20, 2018

Commits on Jul 24, 2018

Commits on Jul 25, 2018

Commits on Aug 21, 2018

Remove invalid keyword argument 'sleep' from kafka pull
icapurro
committed

Commits on Aug 28, 2018

Removed offset unit test
icapurro
committed

Commits on Sep 11, 2018

Commits on Sep 12, 2018

Merge pull request #345 from icapurro/fix-kafka-pull
sibiryakov
authored

Commits on Nov 2, 2018

Commits on Nov 14, 2018

Update cluster-setup docs
guillermoap
committed

Commits on Nov 15, 2018

Merge pull request #351 from guillermoap/update_docs
sibiryakov
authored

Commits on Nov 29, 2018

Docs: BaseCrawlingStrategy._refresh_states() also takes a non iterable for the requests parameter.
IAlwaysBeCoding
authored

Commits on Nov 30, 2018

Update frontera/strategy/__init__.py

sibiryakov
and
IAlwaysBeCoding
authored

Commits on Dec 3, 2018

Merge pull request #355 from IAlwaysBeCoding/IAlwaysBeCoding-_refresh_states
sibiryakov
authored

Commits on Jan 14, 2019

removing call to absent API in happybase
sibiryakov
committed

Commits on Jan 21, 2019

Update after redis update
Knut O. Hellan
committed

Commits on Apr 3, 2019

Commits on Apr 12, 2019

Commits on Jul 3, 2019

Commits on Jul 5, 2019

Merge pull request #375 from liho00/liho00-patch-1
sibiryakov
authored

Commits on Oct 23, 2019

Bump to Python 3.6
Susannah Klaneček
committed
Use thriftpy2
Susannah Klaneček
committed
Add Python 3.7
Susannah Klaneček
committed
Make test_zmq_message_bus() flaky
Susannah Klaneček
committed
Add .env to gitignore
Susannah Klaneček
committed

Commits on Oct 24, 2019

Add badges
Susannah Klaneček
committed
Add pyversions badge
Susannah Klaneček
committed
Rename to "python versions"
Susannah Klaneček
committed

Commits on Oct 25, 2019

Merge pull request #381 from suzil/cleanup/add-badges
sibiryakov
authored
Move thriftpy2 to test_requires
Susannah Klaneček
committed
Also include thriftpy2 as an hbase requirement
Susannah Klaneček
committed
Merge pull request #379 from suzil/bump/py36
sibiryakov
authored

Commits on Nov 29, 2019