Puppet Agent 7.33.0 problem with Ruby memory bloat

[hpoznanski]

Describe the Bug

After upgrading puppet agent to version 7.33.0, we noticed a problem with ruby memory release. It appears to be a memory bloat. It does not occur on all nodes in our stack or at least it is hardly noticeable.

Expected Behavior

Releasing memory at some time interval.

Steps to Reproduce

Steps to reproduce the behavior:
Upgrade puppet version to 7.33.0 (with ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux])

Environment

• Version: Puppet Agent: 7.33.0, ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux]
• Platform: Rocky Linux 9.4 (5.14.0-362.24.1.el9_3.cloud.0.6.x86_64)

[joshcooper]

@hpoznanski do you see this behavior with other puppet-agent versions or just 7.33.0? I'd suggest running puppet in a memory profiler to capture details. This can be done by modifying the puppet bin wrapper like

--- a/bin/puppet
+++ b/bin/puppet
@@ -1,10 +1,19 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
+require 'memory_profiler'
+
 begin
   require 'puppet/util/command_line'
+  MemoryProfiler.start
   Puppet::Util::CommandLine.new.execute
 rescue LoadError => e
   $stderr.puts e.message
   exit(1)
+ensure
+  report = MemoryProfiler.stop
+  puts "writing /tmp/puppet.prof"
+  File.open('/tmp/puppet.prof', 'w') do |f|
+    report.pretty_print(f)
+  end
 end

Then run /opt/puppetlabs/puppet/bin/gem install memory_profiler and restart the agent. Let it run for a while and stop the service. It will generate a memory profile in the tmp directory identifying which objects are still reachable.