Conduct stress-test

[soichih]

I need to conduct stress-test and see how much load it (mostly mcpub) can handle - and determine when it fails. I also need to test clustering capability for mcpub.

(Also make sure proper indexing of MongoDB)

[soichih]

I've installed 3 instances of mcpub on meshconfig-itb.grid.iu.ed (4 CPUs) and ran artillery

With --rate 10 -n 10, I didn't notice any issue (with max 433 concurrent users)

$ artillery quick --duration 60 --rate 10 -n 10 http://meshconfig-itb.grid.iu.edu/pub/config/test-dynamic
...
Complete report @ 2017-02-13T19:54:19.955Z
  Scenarios launched:  600
  Scenarios completed: 600
  Requests completed:  6000
  RPS sent: 55.29
  Request latency:
    min: 48.9
    max: 9944.7
    median: 4967
    p95: 7645.6
    p99: 8446.6
  Scenario duration:
    min: 3004.4
    max: 66495.1
    median: 53778.2
    p95: 60779.5
    p99: 62586
  Scenario counts:
    0: 600 (100%)
  Codes:
    200: 6000

However, with --rate 10 -n 15 I started noticing a few dropped requests, but most requests were successfull (with max 510 concurrent users)

$ artillery quick --duration 60 --rate 10 -n 15 http://meshconfig-itb.grid.iu.edu/pub/config/test-dynamic
...
Complete report @ 2017-02-13T19:59:15.022Z
  Scenarios launched:  600
  Scenarios completed: 552
  Requests completed:  8300
  RPS sent: 52.97
  Request latency:
    min: 64.1
    max: 13241.8
    median: 7219.4
    p95: 9776.9
    p99: 10819.2
  Scenario duration:
    min: 9318.9
    max: 118611
    median: 107212.8
    p95: 114960.6
    p99: 116512.3
  Scenario counts:
    0: 600 (108.696%)
  Codes:
    200: 8300
  Errors:
    ECONNRESET: 48

Load on mongodb was negligible during stress test so I believe index is fine, but I am only testing it with a single meshconfig.. 3 mcpub processes were consuming almost all CPU.

[apertome]

I don't think we need extensive performance testing, but I think we should spend some time doing sanity checks and make sure everything is adequately performant.