From e2cccab1263d6da7cfeb02e4b7dd93574299b70b Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Mon, 21 Oct 2024 09:06:38 -0700 Subject: [PATCH 1/7] Add docs for Windows CPU and Memory Managers Signed-off-by: James Sturtevant --- .../windows-cpu-and-memory-affinity.md | 15 +++++++++ .../cpu-management-policies.md | 7 +++++ .../administer-cluster/memory-manager.md | 31 +++++++++++++++++-- .../administer-cluster/topology-manager.md | 6 ++++ 4 files changed, 56 insertions(+), 3 deletions(-) create mode 100644 content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md new file mode 100644 index 0000000000000..3af6bbce8e95a --- /dev/null +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md @@ -0,0 +1,15 @@ +--- +# Removed from Kubernetes +title: WindowsCPUAndMemoryAffinity +content_type: feature_gate + +_build: + list: never + render: false + +stages: + - stage: alpha + defaultValue: false + fromVersion: "1.32" +--- +Add CPU and Memory Affinity support to Windows nodes with [CPUManager](/docs/tasks/administer-cluster/cpu-management-policies#windows-support), [MemoryManager](/docs/tasks/administer-cluster/memory-manager.md#windows-support) and Topology manager. \ No newline at end of file diff --git a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md index 287c2ef8d8f45..67992ea60a5df 100644 --- a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md +++ b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md @@ -49,6 +49,13 @@ However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. +## Windows Support + +{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} + +The CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires a support in the container runtime. +Once feature is enabled follow the steps above to configure the [CPU Manager Policy](#configuration). + ### Configuration The CPU Manager policy is set with the `--cpu-manager-policy` kubelet diff --git a/content/en/docs/tasks/administer-cluster/memory-manager.md b/content/en/docs/tasks/administer-cluster/memory-manager.md index 300787164b2b5..b11b4e1f4ede8 100644 --- a/content/en/docs/tasks/administer-cluster/memory-manager.md +++ b/content/en/docs/tasks/administer-cluster/memory-manager.md @@ -46,7 +46,7 @@ Preceding v1.22, the `kubelet` must be started with the following flag: in order to enable the Memory Manager feature. -## How Memory Manager Operates? +## How does the Memory Manager Operates? The Memory Manager currently offers the guaranteed memory (and hugepages) allocation for Pods in Guaranteed QoS class. @@ -57,7 +57,7 @@ prepare and deploy a `Guaranteed` pod as illustrated in the section The Memory Manager is a Hint Provider, and it provides topology hints for the Topology Manager which then aligns the requested resources according to these topology hints. -It also enforces `cgroups` (i.e. `cpuset.mems`) for pods. +On Linux, it also enforces `cgroups` (i.e. `cpuset.mems`) for pods. The complete flow diagram concerning pod admission and deployment process is illustrated in [Memory Manager KEP: Design Overview][4] and below: @@ -91,6 +91,12 @@ The problem has been solved as elaborated in Also, reference [Memory Manager KEP: Simulation - how the Memory Manager works? (by examples)][1] illustrates how the management of groups occurs. +### Windows Support + +{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} + +Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. Only the [BestEffort Policy](#policy-best-effort) is supported on Windows. + ## Memory Manager configuration Other Managers should be first pre-configured. Next, the Memory Manager feature should be enabled @@ -103,7 +109,8 @@ node stability (section [Reserved memory flag](#reserved-memory-flag)). Memory Manager supports two policies. You can select a policy via a `kubelet` flag `--memory-manager-policy`: * `None` (default) -* `Static` +* `Static` (Linux only) +* `BestEffort` (Windows Only) #### None policy {#policy-none} @@ -123,6 +130,24 @@ In the case of the `BestEffort` or `Burstable` pod, the `Static` Memory Manager the default topology hint as there is no request for the guaranteed memory, and does not reserve the memory in the internal [NodeMap][2] object. +This policy is only Supported on Linux. + +#### BestEffort policy {#policy-best-effort} + +{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} + +This policy is only Support on Windows. + +On Windows, NUMA node assignment works differently than Linux. There is no mechanism +to ensure that Memory access only comes from a specific NUMA node, instead the Windows +internals will select the most optimal NUMA node based on the CPU assignment and it is +possible that Windows might use other nodes if required or deemed optimal. For this +reason this policy is best effort. + +The policy does track the amount of memory available and requested through the internal +[NodeMap][2] to make a best effort at ensuring that memory enough memory is available on +a NUMA node before making the hint to assign it to the NODE. This means that in most cases, this should function as expected. + ### Reserved memory flag The [Node Allocatable](/docs/tasks/administer-cluster/reserve-compute-resources/) mechanism diff --git a/content/en/docs/tasks/administer-cluster/topology-manager.md b/content/en/docs/tasks/administer-cluster/topology-manager.md index 5d97bf9767e2a..0dc2f90686eee 100644 --- a/content/en/docs/tasks/administer-cluster/topology-manager.md +++ b/content/en/docs/tasks/administer-cluster/topology-manager.md @@ -58,6 +58,12 @@ the pod can be accepted or rejected from the node based on the selected hint. The hint is then stored in the Topology Manager for use by the *Hint Providers* when making the resource allocation decisions. +## Windows Support + +{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} + +The Topology Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires a support in the container runtime. + ## Topology manager scopes and policies The Topology Manager currently: From ce1491d47ecaef414f0aca56a15c47239143efa7 Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Tue, 19 Nov 2024 14:05:38 -0800 Subject: [PATCH 2/7] Apply suggestions from code review Co-authored-by: Mark Rossetti Co-authored-by: Qiming Teng Co-authored-by: Aravindh Puthiyaparambil --- .../windows-cpu-and-memory-affinity.md | 5 ++++- .../cpu-management-policies.md | 6 +++--- .../administer-cluster/memory-manager.md | 19 ++++++++++--------- .../administer-cluster/topology-manager.md | 5 +++-- 4 files changed, 20 insertions(+), 15 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md index 3af6bbce8e95a..678a24803fd67 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md @@ -12,4 +12,7 @@ stages: defaultValue: false fromVersion: "1.32" --- -Add CPU and Memory Affinity support to Windows nodes with [CPUManager](/docs/tasks/administer-cluster/cpu-management-policies#windows-support), [MemoryManager](/docs/tasks/administer-cluster/memory-manager.md#windows-support) and Topology manager. \ No newline at end of file + +Add CPU and Memory Affinity support to Windows nodes with [CPUManager](/docs/tasks/administer-cluster/cpu-management-policies/#windows-support), +[MemoryManager](/docs/tasks/administer-cluster/memory-manager/#windows-support) +and topology manager. \ No newline at end of file diff --git a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md index 67992ea60a5df..b9333e55668eb 100644 --- a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md +++ b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md @@ -51,10 +51,10 @@ management policies to determine some placement preferences on the node. ## Windows Support -{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} +{{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -The CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires a support in the container runtime. -Once feature is enabled follow the steps above to configure the [CPU Manager Policy](#configuration). +The CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. +Once the feature gate is enabled, follow the steps below to configure the [CPU manager policy](#configuration). ### Configuration diff --git a/content/en/docs/tasks/administer-cluster/memory-manager.md b/content/en/docs/tasks/administer-cluster/memory-manager.md index b11b4e1f4ede8..e7316a36c8e5d 100644 --- a/content/en/docs/tasks/administer-cluster/memory-manager.md +++ b/content/en/docs/tasks/administer-cluster/memory-manager.md @@ -46,7 +46,7 @@ Preceding v1.22, the `kubelet` must be started with the following flag: in order to enable the Memory Manager feature. -## How does the Memory Manager Operates? +## How does the Memory Manager Operate? The Memory Manager currently offers the guaranteed memory (and hugepages) allocation for Pods in Guaranteed QoS class. @@ -93,9 +93,10 @@ illustrates how the management of groups occurs. ### Windows Support -{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} +{{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. Only the [BestEffort Policy](#policy-best-effort) is supported on Windows. +Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. +Only the [BestEffort Policy](#policy-best-effort) is supported on Windows. ## Memory Manager configuration @@ -130,17 +131,17 @@ In the case of the `BestEffort` or `Burstable` pod, the `Static` Memory Manager the default topology hint as there is no request for the guaranteed memory, and does not reserve the memory in the internal [NodeMap][2] object. -This policy is only Supported on Linux. +This policy is only supported on Linux. #### BestEffort policy {#policy-best-effort} -{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} +{{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -This policy is only Support on Windows. +This policy is only supported on Windows. -On Windows, NUMA node assignment works differently than Linux. There is no mechanism -to ensure that Memory access only comes from a specific NUMA node, instead the Windows -internals will select the most optimal NUMA node based on the CPU assignment and it is +On Windows, NUMA node assignment works differently from Linux. There is no mechanism +to ensure that memory access only comes from a specific NUMA node, instead the Windows +scheduler will select the most optimal NUMA node based on the CPU assignments and it is possible that Windows might use other nodes if required or deemed optimal. For this reason this policy is best effort. diff --git a/content/en/docs/tasks/administer-cluster/topology-manager.md b/content/en/docs/tasks/administer-cluster/topology-manager.md index 0dc2f90686eee..a16a8d595d606 100644 --- a/content/en/docs/tasks/administer-cluster/topology-manager.md +++ b/content/en/docs/tasks/administer-cluster/topology-manager.md @@ -60,9 +60,10 @@ resource allocation decisions. ## Windows Support -{{< feature-state for_k8s_version="v1.32" state="alpha" feature_gate_name="WindowsCPUAndMemoryAffinity" >}} +{{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -The Topology Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires a support in the container runtime. +The Topology Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature gate and +it requires a support in the container runtime. ## Topology manager scopes and policies From bdc9bef55dbd3943c1c0aeace66ed6a7dcab96d2 Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Tue, 19 Nov 2024 14:17:14 -0800 Subject: [PATCH 3/7] Updates based on feedback Signed-off-by: James Sturtevant --- .../windows-cpu-and-memory-affinity.md | 1 - .../cpu-management-policies.md | 4 ++-- .../tasks/administer-cluster/memory-manager.md | 18 +++++++++--------- 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md index 678a24803fd67..00dfa1b4c4a3a 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates/windows-cpu-and-memory-affinity.md @@ -1,5 +1,4 @@ --- -# Removed from Kubernetes title: WindowsCPUAndMemoryAffinity content_type: feature_gate diff --git a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md index b9333e55668eb..295901cbbf5a1 100644 --- a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md +++ b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md @@ -53,8 +53,8 @@ management policies to determine some placement preferences on the node. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -The CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. -Once the feature gate is enabled, follow the steps below to configure the [CPU manager policy](#configuration). +CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires support in the container runtime. +Once feature is enabled follow the steps above to configure the [CPU Manager Policy](#configuration). ### Configuration diff --git a/content/en/docs/tasks/administer-cluster/memory-manager.md b/content/en/docs/tasks/administer-cluster/memory-manager.md index e7316a36c8e5d..4d6ad746dc0fd 100644 --- a/content/en/docs/tasks/administer-cluster/memory-manager.md +++ b/content/en/docs/tasks/administer-cluster/memory-manager.md @@ -95,7 +95,7 @@ illustrates how the management of groups occurs. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires a support in the container runtime. +Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires support in the container runtime. Only the [BestEffort Policy](#policy-best-effort) is supported on Windows. ## Memory Manager configuration @@ -139,15 +139,15 @@ This policy is only supported on Linux. This policy is only supported on Windows. -On Windows, NUMA node assignment works differently from Linux. There is no mechanism -to ensure that memory access only comes from a specific NUMA node, instead the Windows -scheduler will select the most optimal NUMA node based on the CPU assignments and it is -possible that Windows might use other nodes if required or deemed optimal. For this -reason this policy is best effort. +On Windows, NUMA node assignment works differently than Linux. +There is no mechanism to ensure that Memory access only comes from a specific NUMA node. +Instead the Windows scheduler will select the most optimal NUMA node based on the CPU(s) assignments. +It is possible that Windows might use other NUMA nodes if deemed optimal by the Windows scheduler. -The policy does track the amount of memory available and requested through the internal -[NodeMap][2] to make a best effort at ensuring that memory enough memory is available on -a NUMA node before making the hint to assign it to the NODE. This means that in most cases, this should function as expected. +The policy does track the amount of memory available and requested through the internal [NodeMap][2]. +The memory manager will make a best effort at ensuring that enough memory is available on +a NUMA node before making the assignment. +This means that in most cases memory assignment should function as expected. ### Reserved memory flag From bd84bba8ad8302327286bd3b1e93a82f6429eb1a Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Thu, 21 Nov 2024 15:03:28 -0800 Subject: [PATCH 4/7] Update content/en/docs/tasks/administer-cluster/cpu-management-policies.md Co-authored-by: Qiming Teng --- .../en/docs/tasks/administer-cluster/cpu-management-policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md index 295901cbbf5a1..20496ff536e7c 100644 --- a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md +++ b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md @@ -54,7 +54,7 @@ management policies to determine some placement preferences on the node. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires support in the container runtime. -Once feature is enabled follow the steps above to configure the [CPU Manager Policy](#configuration). +Once the feature gate is enabled, follow the steps below to configure the [CPU manager policy](#configuration). ### Configuration From 6ca0d52b5caae68127c1392934377395327f202a Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Thu, 21 Nov 2024 19:37:56 -0800 Subject: [PATCH 5/7] Update content/en/docs/tasks/administer-cluster/cpu-management-policies.md Co-authored-by: Qiming Teng --- .../docs/tasks/administer-cluster/cpu-management-policies.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md index 20496ff536e7c..5a8373a94d362 100644 --- a/content/en/docs/tasks/administer-cluster/cpu-management-policies.md +++ b/content/en/docs/tasks/administer-cluster/cpu-management-policies.md @@ -53,7 +53,8 @@ management policies to determine some placement preferences on the node. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature flag and requires support in the container runtime. +CPU Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature gate +and it requires support in the container runtime. Once the feature gate is enabled, follow the steps below to configure the [CPU manager policy](#configuration). ### Configuration From 0867522df3db41f46be13d9b7b6ee73ab26be6d6 Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Thu, 21 Nov 2024 19:38:17 -0800 Subject: [PATCH 6/7] Update content/en/docs/tasks/administer-cluster/memory-manager.md Co-authored-by: Qiming Teng --- content/en/docs/tasks/administer-cluster/memory-manager.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/memory-manager.md b/content/en/docs/tasks/administer-cluster/memory-manager.md index 4d6ad746dc0fd..0f133d7db9fc5 100644 --- a/content/en/docs/tasks/administer-cluster/memory-manager.md +++ b/content/en/docs/tasks/administer-cluster/memory-manager.md @@ -95,7 +95,8 @@ illustrates how the management of groups occurs. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} -Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate and requires support in the container runtime. +Windows support can be enabled via the `WindowsCPUAndMemoryAffinity` feature gate +and it requires support in the container runtime. Only the [BestEffort Policy](#policy-best-effort) is supported on Windows. ## Memory Manager configuration From 7f411ed89175f0af96d13155232d6e506e65b99c Mon Sep 17 00:00:00 2001 From: James Sturtevant Date: Thu, 21 Nov 2024 19:38:28 -0800 Subject: [PATCH 7/7] Update content/en/docs/tasks/administer-cluster/topology-manager.md Co-authored-by: Qiming Teng --- content/en/docs/tasks/administer-cluster/topology-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/tasks/administer-cluster/topology-manager.md b/content/en/docs/tasks/administer-cluster/topology-manager.md index a16a8d595d606..69972a0b7e9b6 100644 --- a/content/en/docs/tasks/administer-cluster/topology-manager.md +++ b/content/en/docs/tasks/administer-cluster/topology-manager.md @@ -63,7 +63,7 @@ resource allocation decisions. {{< feature-state feature_gate_name="WindowsCPUAndMemoryAffinity" >}} The Topology Manager support can be enabled on Windows by using the `WindowsCPUAndMemoryAffinity` feature gate and -it requires a support in the container runtime. +it requires support in the container runtime. ## Topology manager scopes and policies