Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work on ECS (nested cgroups) #66

Open
blampe opened this issue Apr 29, 2023 · 2 comments
Open

Doesn't work on ECS (nested cgroups) #66

blampe opened this issue Apr 29, 2023 · 2 comments

Comments

@blampe
Copy link

blampe commented Apr 29, 2023

Unlike Kubernetes, ECS only allows you to apply a CPU quota at the task (pod) level. Containers in the task are always unbounded.

For example, when cpu: 1024 (1 vCPU) is provided in the task definition it gets the expected quota:

cat /sys/fs/cgroup/cpu,cpuacct/ecs/1576650513ed4c5d9328a6d67a8a741b/cpu.cfs_quota_us
100000

But providing cpu: 1024 to a container inside the same task doesn't have the same effect:

cat /sys/fs/cgroup/cpu,cpuacct/ecs/1576650513ed4c5d9328a6d67a8a741b/2030454c61d157d4c38f0606fe99667bca8961ce4e0019d3667f67e625f40c12/cpu.cfs_quota_us
-1

(The container's cpu value is only used for placement and CPU shares, but doesn't actually affect CPU scheduling aws/containers-roadmap#1862.)

If the container is using automaxprocs it only sees a quota of -1 and defaults to using all of runtime.NumCPU, even though the task's cgroup clamps it to 1 vCPU.

(I'm using cgroups v1 as an example here but the same is true with v2 as well, if you happen to be using an AL2023 AMI.)

It seems like the library could climb up the mount point to find quotas belonging to parents, but this is suboptimal if the task has more than one container.

I'm mostly writing this down to help anyone else avoid this rabbit hole.

@richardartoul
Copy link

Thank you for saving me the time lol

@rdforte
Copy link

rdforte commented Sep 20, 2024

For anyone else who stumbles across this like I did. I also experienced the issue of trying to auto set GOMAXPROCS using automaxprocs in ECS. Unfortunately that won't work because just like @blampe mentioned the containers cpu.cfs_quota_us is set to -1.

I did however manage to find a workaround. It aint pretty but what you can do as part of your app startup is leverage the ECS Metadata endpoint which you can reference as part of an env variable to pull the container and task cpu limit. You can then use the cpu limit to set GOMAXPROCS.

I've put together a repo of an example for anyone who is interested: rdforte/gomaxecs/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants