Crawlers should support custom steps

[ekryski]

A crawler might look something like this.

var Crawler = require('roach').job();

Crawler.extend({
  // Your extra custom functions for parsing, etc.
  custom: function(params, data, next){
    // do some custom stuff with the data
    next(data);
  }
});

Crawler.visit(this.url)
       .custom()
       .find('a.link')
       .each()
       .click()
       .done();

[bredele]

Hey Eric, I was thinking about something more like this:

var crawler = require('roach').crawler({
  custom: function(){}
});
crawler
  .get('http://something.io/file.zip')
  .pipe(crawler.unzip())
  .find('a.link')
  .custom()
  .done();

I think it would be better to access the crawler methods (get, zip, csv, etc) outside of the job.

[ekryski]

Ya that should work. Although I suspect it will look more like this:

var crawler = require('roach').crawler({
  custom: function(){}
});
crawler
  .get('http://something.io/file.zip')
  .pipe(crawler.unzip())
  .pipe(crawler.find('a.link'))
  .pipe(crawler.custom())
  .done();

[bredele]

Yes, having a pipe api would help us to be more consistent.

[bredele]

I'm going to work on this right now.

[ekryski]

Either use pipe or we use the jquery style .then().

[bredele]

Do you know a chainable library with pipe?

[bredele]

The then is more like promise based right?

[ekryski]

No I don't know one that has a 'pipe' method explicitly. And ya jQuery's .then() is around their deferred's/promises

[bredele]

I'll figured what is the best solution, I guess using promises would be easier.

[ekryski]

Ya I thought it would be promise based for sure. Look at how Casper does it. It's basically what we are going to emulate.

[bredele]

oki I'll.

[ekryski]

Could also look at how gulpjs is doing their pipe method.

[ekryski]

We might need our own through wrapper for custom steps and for the '.XLS' parser.

[bredele]

Eric, do you want the step handler in the crawler or the job? Personally I think it makes sense to have it inside the crawler (the job doesn't handler streams) as follow:

crawler('http://myfile.txt')
  .step(require('step1'))
  .step(require('step2'));

UPDATE: Having something as above is not possible because every handler return a stream.