Initial commit of metrics for scaling #119

dklyle · 2019-07-22T20:05:47Z

The beginning for #117

grahamwhaley

This lgtm
Thanks for putting this together @dklyle . I ran it and got the results (json) file I expected.
There are obviously lots of things we want to clean up and improve, but as a first step this will allow us to put together and test the rest of the infra whilst we improve the test itself.
Thus, I'm happy if we merge this and then improve it with further PRs.

grahamwhaley · 2019-07-23T08:40:43Z

metrics/lib/common.bash

@@ -0,0 +1,339 @@
+#!/bin/bash
+#
+# Copyright (c) 2017,2018 Intel Corporation


+2019 - and for other files as well.

grahamwhaley · 2019-07-23T08:45:32Z

metrics/lib/common.bash

+source ${LIB_DIR}/kata-common.bash
+source ${LIB_DIR}/json.bash
+source /etc/os-release || source /usr/lib/os-release
+KATA_KSM_THROTTLER="${KATA_KSM_THROTTLER:-no}"


I suspect we can drop any notion of the throttler - afaik it is not installed by kata-deploy when adding kata to k8s installs.

removed and the associated function

grahamwhaley · 2019-07-23T08:46:01Z

metrics/lib/common.bash

+KATA_KSM_THROTTLER="${KATA_KSM_THROTTLER:-no}"
+
+# Set variables to reasonable defaults if unset or empty
+DOCKER_EXE="${DOCKER_EXE:-docker}"


I think we are going to eradicate all docker stuff from this code, yes?

punted on this for now, will work on once this is merged

grahamwhaley · 2019-07-23T08:50:45Z

metrics/lib/json.bash

+			local rpath="$(get_docker_kata_path $RUNTIME)"
+			local json="$(cat << EOF
+	"kata-env" :
+	$($rpath kata-env --json)


We'll need to fix up (and write anew) this section, as it is currently pretty hard wired to try and get docker info. We want k8s info.
A quick fix/hack is to add the output of kubectl get nodes -o=json, which is way better than nothing - but longer term I wonder if we can find/extract more info to add here about the DUT.

punted on this for now

grahamwhaley · 2019-07-23T08:52:57Z

metrics/lib/kata-common.bash

+
+KATA_HYPERVISOR="${KATA_HYPERVISOR:-qemu}"
+
+die() {


we have die,warn,info in two of the lib files - we should be able to factor those out now.

removed from common.bash as in sources kata-common.bash

grahamwhaley · 2019-07-23T08:55:24Z

metrics/scaling/bb.yaml.in

@@ -0,0 +1,33 @@
+apiVersion: apps/v1


We could add a copyright here.
@krsna1729 @mcastelino - do you have a standard idiom you use for this repo? This code is being cloned from over in the Kata code base, so obviously it (and I ;-) ) come with some pre-conceptions, like adding a copyright header to every file that can take one - but, you have the top level LICENSE file that covers the whole repo with Apache2.0 - so, we are probably good just having that file and individual file over-rides where necessary. what say you?

grahamwhaley · 2019-07-23T09:00:01Z

metrics/scaling/k8s_scale.sh

+	info "And grab some stats"
+	# Tell mpstat to measure over a short period, not only so we get slightly de-noised data, but also
+	# if you don't tell it the period, you will get the avg since boot, which is not what we want.
+	local cpu_idle=$(kubectl exec -ti ${stats_pod} -- sh -c "mpstat -u 3 1 | tail -1 | awk '{print \$11}'" | sed 's/\r//')


heh, I'd forgotten I wrote that! ;-)

grahamwhaley · 2019-07-23T09:06:43Z

metrics/scaling/k8s_scale.sh

+		# FIXME - time taken in 'seconds' - we can do better than that, if we
+		# use the date nanosecond stamp, and steal the magic from the density test script.
+        # switch to 'date +%N'
+		local start_time=$(date +%s)


the json.sh lib file has ms and ns time functions we can now use here I reckon..

changed to milliseconds

grahamwhaley · 2019-07-23T17:24:17Z

metrics/scaling/k8s_scale.sh

+	# kubelet config - from 'kubectl describe node -o json' ?
+
+	# Launch our stats gathering pod
+	kubectl apply -f ${stats_pod}.yaml


I've just found, we might want to prepend the ${SCRIPT_PATH} to the front of the yaml files etc., as if you don't run the test from within its own directory right now it fails to find stats.yaml for instance.

that will be true for bb.yaml.in too, it's mergeable without updating, but will fix

updated for bb.yaml.in, generated.yaml, and stats.yaml

grahamwhaley · 2019-07-24T10:53:55Z

metrics/scaling/k8s_scale.sh

+		info "Testing replicas ${reqs} of ${NUM_PODS}"
+		# Generate the next yaml file
+
+	local runtime_command


just noticed, I think indentation is bust in this section - it is all in the for loop, so some of the lines need to be pushed in one tab.

we should incorporate these, maybe as git pre-commit hooks?
https://github.com/mvdan/sh
https://github.com/koalaman/shellcheck

@krsna1729 eek. maybe.... I think we opted out of shellcheck over in kata land as there were some things it moaned about that were particularly hard to get around iirc...

Let's see if anybody finds the time to try these out on the code.... and, reserve the right to not inflict them if they pain barrier will be too high ;-)

grahamwhaley · 2019-07-24T16:41:37Z

Merging - we will update and clean some other bits in other PRs as we grow and improve the code base.

grahamwhaley approved these changes Jul 23, 2019

View reviewed changes

grahamwhaley mentioned this pull request Jul 23, 2019

metrics: add a local report generator tool for quick compares #121

Closed

grahamwhaley reviewed Jul 23, 2019

View reviewed changes

grahamwhaley mentioned this pull request Jul 23, 2019

metrics: Add the start of the local report generator #122

Merged

dklyle added 2 commits July 23, 2019 14:37

Initial commit of metrics for scaling

bc80efc

Moving to milliseconds rather than seconds

fbd66ed

dklyle force-pushed the add-metrics branch from 4df0a8e to fbd66ed Compare July 23, 2019 20:52

grahamwhaley reviewed Jul 24, 2019

View reviewed changes

grahamwhaley merged commit 0f39518 into clearlinux:master Jul 24, 2019

dklyle deleted the add-metrics branch July 26, 2019 15:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial commit of metrics for scaling #119

Initial commit of metrics for scaling #119

dklyle commented Jul 22, 2019 •

edited

Loading

grahamwhaley left a comment

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 23, 2019

dklyle Jul 23, 2019

dklyle Jul 23, 2019

grahamwhaley Jul 24, 2019

krsna1729 Jul 24, 2019

grahamwhaley Jul 24, 2019

grahamwhaley commented Jul 24, 2019


		KATA_HYPERVISOR="${KATA_HYPERVISOR:-qemu}"

		die() {

Initial commit of metrics for scaling #119

Initial commit of metrics for scaling #119

Conversation

dklyle commented Jul 22, 2019 • edited Loading

grahamwhaley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamwhaley commented Jul 24, 2019

dklyle commented Jul 22, 2019 •

edited

Loading