kindling probe init err while installing

[coolman-007]

A permission-denied issue occurred while executing install scripts, the following is detailed information.

kindling-falcolib-probe/4.19.91-18.al7.x86_64.o
kindling-falcolib-probe/3.10.0-1127.el7.x86_64.ko
kindling-falcolib-probe/4.19.1-1.el7.elrepo.x86_64.ko

• Mounting debugfs
• BPF probe located, it's now possible to start kindling
• Load probe succeeded, and will create /opt/kernel-support for kubernetes
  2023/04/10 10:48:27 GitCommitInfo:; go1.20.1 linux/amd64
  2023-04-10T10:48:27.957Z INFO component/telemetry.go:48 Log Initialize Success! ConsoleLevel: info,FileRotationLevel: info
  FileRotationConfig: LogFile: agent.log MaxSize: 512m backup: 5 MaxAge: 30day
  2023-04-10T10:48:27.957Z INFO component/telemetry.go:58 [CLUSTER_ID] is not found in env variable which will be set [noclusteridset]
  2023-04-10T10:48:27.957Z INFO observability/telemetry.go:157 Initializing self-observability exporter whose type is stdout
  2023-04-10T10:48:27.958Z WARN otelexporter/otelexporter.go:94 [CLUSTER_ID] is not found in env variable which will be set [noclusteridset]
  2023-04-10T10:48:27.960Z INFO otelexporter/prometheus.go:18 Prometheus Server listening at port: [:9500]
  2023/04/10 10:48:31 rcv buffer size for netlink socket is 2097152 bytes
  2023/04/10 10:48:31 initialized conntrack with target_rate_limit=500 messages/sec
  2023-04-10T10:48:31.969Z INFO analyzer/manager.go:59 Starting analyzer [networkanalyzer]
  2023-04-10T10:48:31.969Z INFO analyzer/manager.go:59 Starting analyzer [tcpmetricanalyzer]
  2023-04-10T10:48:31.969Z INFO analyzer/manager.go:59 Starting analyzer [tcpconnectanalyzer]
  2023-04-10T10:48:31.969Z INFO analyzer/manager.go:59 Starting analyzer [cpuanalyzer]
  2023-04-10T10:48:31.969Z INFO cgoreceiver/cgoreceiver.go:63 Start CgoReceiver
  Set snaplen to value: 1000
  kindling probe init err: can't create map: Permission denied2023/04/10 10:48:31 Failed to run application: failed to start application: fail to init probe

I checked that the SELinux has been disabled. But the problem is still.
What could I do next? someone could help me. Thanks.

[jundizhou]

Please check if privileged mode is enabled.
(Note: Minikube is currently not supported)

[wenhuwang]

I also install kinding agent failed. error is Error: Precompiled module at /opt/.kindling/ is not found.
I have installed the kernel-header package and rebuilt the kindling-agent image according to the documentation, but the error log has not changed。

k8s cluster env

$ kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
***** Ready master 506d v1.22.2 ***** Ubuntu 18.04.5 LTS 5.4.169-0504169-generic containerd://1.4.4
***** Ready master 506d v1.22.2 ***** Ubuntu 18.04.5 LTS 5.10.87-051087-generic containerd://1.4.4
***** Ready node 506d v1.22.2 ***** Ubuntu 18.04.6 LTS 5.10.87-051087-generic containerd://1.4.4
***** Ready node. 275d v1.22.2 ***** CentOS Linux 7 (Core) 5.4.207-1.el7.elrepo.x86_64 containerd://1.4.4

kindling-agent logs is as follows:

kubectl -n kindling logs -f kindling-agent-d5css kindling-agent
kindling-falcolib-probe/
kindling-falcolib-probe/4.19.91-26.al7.x86_64.o
...

• Mounting debugfs
• Failure to find a BPF probe, try to load kernel probe
• Unloading kindling-falcolib-probe, if present
• Trying to find precompiled kindling-falcolib-probe for 5.10.87-051087-generic
• Error: Precompiled module at /opt/.kindling/ is not found, and the agent will not work as expected
  2023/04/27 01:53:37 GitCommitInfo:; go1.20.1 linux/amd64
  2023-04-27T01:53:37.177Z INFO component/telemetry.go:52 Log Initialize Success! ConsoleLevel: info,FileRotationLevel: info
  FileRotationConfig: LogFile: agent.log MaxSize: 512m backup: 5 MaxAge: 30day
  2023-04-27T01:53:37.178Z INFO component/telemetry.go:64 [CLUSTER_ID] is not found in env variable which will be set [noclusteridset]
  2023-04-27T01:53:37.178Z INFO observability/telemetry.go:157 Initializing self-observability exporter whose type is stdout
  2023-04-27T01:53:37.179Z WARN otelexporter/otelexporter.go:94 [CLUSTER_ID] is not found in env variable which will be set [noclusteridset]
  2023-04-27T01:53:37.183Z INFO otelexporter/prometheus.go:18 Prometheus Server listening at port: [:9500]
  2023/04/27 01:53:40 rcv buffer size for netlink socket is 2097152 bytes
  2023/04/27 01:53:40 initialized conntrack with target_rate_limit=500 messages/sec
  2023-04-27T01:53:40.237Z INFO analyzer/manager.go:59 Starting analyzer [networkanalyzer]
  2023-04-27T01:53:40.237Z INFO analyzer/manager.go:59 Starting analyzer [tcpmetricanalyzer]
  2023-04-27T01:53:40.237Z INFO analyzer/manager.go:59 Starting analyzer [tcpconnectanalyzer]
  2023-04-27T01:53:40.237Z INFO analyzer/manager.go:59 Starting analyzer [cpuanalyzer]
  2023-04-27T01:53:40.237Z INFO cgoreceiver/cgoreceiver.go:63 Start CgoReceiver
  Set snaplen to value: 1000
  kindling probe init err: error opening device /host/dev/kindling-falcolib0. Make sure you have root credentials and that the kindling-falcolib-probe module is loaded.Please read faq http://kindling.harmonycloud.cn/docs/installation/faq/#error-precompiled-module-at-optkindling-is-2023/04/27 01:53:40 Failed to run application: failed to start application: fail to init probe

Needs helps, Thanks.

[wenhuwang]

Because my cluster has both ubuntu system and centos distribution, I don't know how to build a kindling-agent image that supports both ubuntu and centos distributions on one build node。

[dxsup]

From the above compiling logs, we can see:

• There is more than one kernel installed ever in this host. The script tried to compile the probe based on all these versions.
• Compile the probe for 4.15.0-162-generic: Success.
• Compile the probe for 4.15.0-163-generic: Success.
• Compile the probe for 4.15.0-45-generic: Failure. Because it didn't find the kernel-devel/kernel-header - the build directory is empty.
• Compile the probe for 5.10.87-051087-generic-lib: Failure. Because it is a tarball, not a directory. (We get the kernel versions from the file names under /lib/modules/)
• Compile the probe for 5.4.187-0504187-generic: Not know.

So, please check the probes are compiled successfully based on the kernel versions you are running.

For the second question, it is possible to support both Ubuntu and CentOS distributions on one build node. First install different versions of kernel-header/kernel-devel (based on which distribution you use, kernel-header for Ubuntu and kernel-devel for CentOS), and then follow the building doc to compile all the probes. After you get them, there are two ways to build an image supporting both Ubuntu and CentOS.

1. Build the image base on the existing kindling-agent image.

What this method does is the same as the recompile-module.sh does in the FAQ doc.
First, prepare a Dockerfile to add all the probes you have got before.

FROM kindlingproject/kindling-agent:v0.7.2
COPY ./kindling-falcolib-probe/* /opt/kindling-extra-probe/

Then, build the new image using it, you will get the image supporting both Ubuntu and CentOS distributions.

2. Build the image from the source.

First, clone the source code and compile the agent following the compiling doc.

Then, copy the tarball containing all probes you need to kindling/collector/docker/ and build the image using the DockerfileLocalProbe, you will get the image supporting both Ubuntu and CentOS distributions.

[wenhuwang]

thanks very much.
I will try to solve the first problem first, and then try to build a mirror that supports ubuntu and centos according to your plan

[wenhuwang]

hello， I scanned the QR code of the WeChat group, please pass it。

[dxsup]

Sorry for the late reply. Done.