nyshporka

Efficient Web Crawler in C++ that Uses Microservice Architecture

Installation

Build Prerequisites

• gcc, cmake, make
• crow
• libmongoc, libmongocxx
• tbb

Building

$ mkdir build;
$ cmake -B build
$ make <target> -jX

Usage

run task manager

$ build/task_manager data/config.cfg visited.txt

run a crawler

$ build/nysh_crawler <batch size> https://task.manager.ip:port

run the search backend (possibly in cli mode)

$ build/nysh_search (cli?)

To run the frontend go here