Efficient Web Crawler in C++ that Uses Microservice Architecture
- gcc, cmake, make
- crow
- libmongoc, libmongocxx
- tbb
$ mkdir build;
$ cmake -B build
$ make <target> -jX
run task manager
$ build/task_manager data/config.cfg visited.txt
run a crawler
$ build/nysh_crawler <batch size> https://task.manager.ip:port
run the search backend (possibly in cli mode)
$ build/nysh_search (cli?)
To run the frontend go here