Documentation on how to extend the crawler ecosystem

[lamchakchan]

Description

This agentless crawler ecosystem looks very interesting. I would like to contribute to it but I'm new to both this project and to Python. Can someone familiar with this project add some information around how the plugin architecture works such as prerequisites and class interface contracts. How this project is structure. If it is structured like a standard Python app, maybe a link to some source of truth on what this looks like. And if there are any Python specific tools/concepts/libs that this project depends or focuses around, that would be good to detail out as well.

[sahilsuneja1]

See Section 3.1.5 here: http://www.cs.toronto.edu/~sahil/suneja-mw17-2-TR.pdf
That would give you some information about the crawler architecture.
Also, some more arch info below: