Spike in consumed capacity can cause rapid drop in provisioned capacity

[onebytegone]

On 2018-08-27, I had a table that was running at 60 RCU due to short spikes in read traffic (~1min, 100+ RCU). Once the MinimumMinutesBeforeDecreaseAfterIncrease had elapsed since the last increase, the RCU was dropped from 60 RCU to 2 RCU. Manually checking on the consumed capacity for the 20min evaluation window, 2 RCU is a little low, even when ignoring the spike to 117 RCU.

This test replicates the decision that occured at the time of the drop:

diff --git a/src/tests/boss/DecisionMaker.test.js b/src/tests/boss/DecisionMaker.test.js
index 1077400..d0cfe1a 100644
--- a/src/tests/boss/DecisionMaker.test.js
+++ b/src/tests/boss/DecisionMaker.test.js
@@ -5,17 +5,18 @@ var _ = require('underscore'),
     expect = require('expect.js'),
     rewire = require('rewire'),
     moment = require('moment'),
+    constants = require('../../constants'),
     DecisionMaker = rewire('../../boss/DecisionMaker');

 describe('DecisionMaker', function() {
    var decisionMaker, revert;

    beforeEach(function() {
-      revert = DecisionMaker.__set__({
-         console: { log: _.noop },
-      });
+      revert = DecisionMaker.__set__({});

-      decisionMaker = new DecisionMaker();
+      decisionMaker = new DecisionMaker(_.extend({}, constants.DEFAULT_RESOURCE_CONFIG, {
+         MinimumMinutesBeforeDecreaseAfterIncrease: 60,
+      }));
    });

    afterEach(function() {
@@ -209,6 +210,41 @@ describe('DecisionMaker', function() {
          sinon.assert.calledOnce(changeDenySpy);
       });

+      it.only('example of unexpected drop', function() {
+         var currentTime = '2018-08-27T19:52:30Z',
+             provisioned, usageData;
+
+         provisioned = {
+            NumberOfDecreasesToday: 3,
+            LastIncreaseDateTime: new Date('2018-08-27T18:50:00.00Z'),
+            LastDecreaseDateTime: new Date('2018-08-27T12:00:58.00Z'),
+            CapacityUnits: 60,
+         };
+
+         usageData = [
+            { Timestamp: '2018-08-27T19:33:00.000Z', Value: 2.45 },
+            { Timestamp: '2018-08-27T19:34:00.000Z', Value: 1.6416666666666666 },
+            { Timestamp: '2018-08-27T19:35:00.000Z', Value: 12.283333333333333 },
+            { Timestamp: '2018-08-27T19:36:00.000Z', Value: 117.63333333333334 },
+            { Timestamp: '2018-08-27T19:37:00.000Z', Value: 1.0833333333333333 },
+            { Timestamp: '2018-08-27T19:38:00.000Z', Value: 2.525 },
+            { Timestamp: '2018-08-27T19:39:00.000Z', Value: 3.6333333333333333 },
+            { Timestamp: '2018-08-27T19:40:00.000Z', Value: 1.6166666666666667 },
+            { Timestamp: '2018-08-27T19:41:00.000Z', Value: 3.5166666666666666 },
+            { Timestamp: '2018-08-27T19:42:00.000Z', Value: 1.9 },
+            { Timestamp: '2018-08-27T19:43:00.000Z', Value: 3.425 },
+            { Timestamp: '2018-08-27T19:44:00.000Z', Value: 3.05 },
+            { Timestamp: '2018-08-27T19:45:00.000Z', Value: 1.1 },
+            { Timestamp: '2018-08-27T19:46:00.000Z', Value: 2 },
+            { Timestamp: '2018-08-27T19:47:00.000Z', Value: 2.658333333333333 },
+            { Timestamp: '2018-08-27T19:48:00.000Z', Value: 4.408333333333333 },
+            { Timestamp: '2018-08-27T19:49:00.000Z', Value: 1.7166666666666666 },
+            { Timestamp: '2018-08-27T19:50:00.000Z', Value: 10.925 },
+            { Timestamp: '2018-08-27T19:51:00.000Z', Value: 13.05 },
+         ];
+
+         expect(decisionMaker.getUpdatedCapacity(provisioned, usageData, [], currentTime)).to.be(60);
+      });
    });

 });

The drop to 2 RCU was caused by the following: The polynomial regression returned a forecast of -4.233514 RCU (due to the single spike of 117 consumed RCUs in the 20min window the forecast is based on). This was then limited to 1 RCU. The forecast was then doubled to 2 RCU by the ForecastPaddingRule. Which then got applied to the table.

Since there was read traffic that needed more than 2 RCU, reads to the table started to be throttled and the DCM upped the provisioned RCU within a few minutes. In the end the DCM corrected this "mistake", however it would be good to not have drops like this in the first place.