Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new(driver/modern_bpf,userspace/libpman): support multiple programs for each event #2255

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

FedeDP
Copy link
Contributor

@FedeDP FedeDP commented Jan 23, 2025

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area driver-modern-bpf
/area libpman

Does this PR require a change in the driver versions?

What this PR does / why we need it:

Allow to specify multiple program names for each event type and try to inject each of them until success.
This allows us to inject bpf_loop sendmmsg and recvmmsg programs where supported, and fallback at a program just sending first message where it isn't.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This superseedes #2233

Does this PR introduce a user-facing change?:

new(driver/modern_bpf,userspace/libpman): support multiple programs for each event

@poiana
Copy link
Contributor

poiana commented Jan 23, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

github-actions bot commented Jan 23, 2025

Perf diff from master - unit tests

    11.30%     -0.53%  [.] sinsp::next
     3.04%     -0.31%  [.] sinsp_thread_manager::find_thread
     2.77%     +0.29%  [.] sinsp_thread_manager::get_thread_ref
     1.55%     +0.28%  [.] next
     1.13%     +0.26%  [.] sinsp::fetch_next_event
     0.34%     +0.26%  [.] sinsp_parser::parse_open_openat_creat_exit
     0.75%     +0.26%  [.] sinsp_evt::get_direction
     2.29%     +0.22%  [.] is_conversion_needed
     8.39%     -0.21%  [.] sinsp_evt::get_type
     5.44%     +0.21%  [.] next_event_from_file

Heap diff from master - unit tests

peak heap memory consumption: -5.18K
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Heap diff from master - scap file

peak heap memory consumption: 0B
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Benchmarks diff from master

Comparing gbench_data.json to /root/actions-runner/_work/libs/libs/build/gbench_data.json
Benchmark                                                         Time             CPU      Time Old      Time New       CPU Old       CPU New
----------------------------------------------------------------------------------------------------------------------------------------------
BM_sinsp_split_mean                                            +0.0138         +0.0137           148           150           148           150
BM_sinsp_split_median                                          +0.0142         +0.0141           148           151           148           151
BM_sinsp_split_stddev                                          +0.0617         +0.0614             1             2             1             2
BM_sinsp_split_cv                                              +0.0473         +0.0471             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_mean                  +0.0835         +0.0834            57            62            57            62
BM_sinsp_concatenate_paths_relative_path_median                +0.0909         +0.0907            57            62            57            62
BM_sinsp_concatenate_paths_relative_path_stddev                +0.1936         +0.1935             1             1             1             1
BM_sinsp_concatenate_paths_relative_path_cv                    +0.1016         +0.1016             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_mean                     -0.0508         -0.0509            25            24            25            24
BM_sinsp_concatenate_paths_empty_path_median                   -0.0499         -0.0499            25            24            25            24
BM_sinsp_concatenate_paths_empty_path_stddev                   -0.6380         -0.6337             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_cv                       -0.6186         -0.6141             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_mean                  +0.0318         +0.0317            58            60            58            60
BM_sinsp_concatenate_paths_absolute_path_median                -0.0090         -0.0091            58            58            58            58
BM_sinsp_concatenate_paths_absolute_path_stddev                +1.0422         +1.0427             1             3             1             3
BM_sinsp_concatenate_paths_absolute_path_cv                    +0.9793         +0.9799             0             0             0             0
BM_sinsp_split_container_image_mean                            +0.0046         +0.0046           390           392           390           392
BM_sinsp_split_container_image_median                          +0.0058         +0.0057           390           392           390           392
BM_sinsp_split_container_image_stddev                          -0.1503         -0.1514             2             2             2             2
BM_sinsp_split_container_image_cv                              -0.1543         -0.1553             0             0             0             0

@FedeDP FedeDP changed the title new(driver/modern_bpf,userspace/libpman): support multiple programs for each event wip: new(driver/modern_bpf,userspace/libpman): support multiple programs for each event Jan 23, 2025
Copy link

codecov bot commented Jan 23, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.34%. Comparing base (6c46ed3) to head (ffb1207).
Report is 54 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2255      +/-   ##
==========================================
+ Coverage   75.16%   75.34%   +0.17%     
==========================================
  Files         278      279       +1     
  Lines       34478    34368     -110     
  Branches     5922     5873      -49     
==========================================
- Hits        25916    25893      -23     
+ Misses       8562     8475      -87     
Flag Coverage Δ
libsinsp 75.34% <ø> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Jan 23, 2025

X64 kernel testing matrix

KERNEL CMAKE-CONFIGURE KMOD BUILD KMOD SCAP-OPEN BPF-PROBE BUILD BPF-PROBE SCAP-OPEN MODERN-BPF SCAP-OPEN
amazonlinux2-4.19 🟢 🟢 🟢 🟢 🟢 🟡
amazonlinux2-5.10 🟢 🟢 🟢 🟢 🟢 🟢
amazonlinux2-5.15 🟢 🟢 🟢 🟢 🟢 🟢
amazonlinux2-5.4 🟢 🟢 🟢 🟢 🟢 🟡
amazonlinux2022-5.15 🟢 🟢 🟢 🟢 🟢 🟢
amazonlinux2023-6.1 🟢 🟢 🟢 🟢 🟢 🟢
archlinux-6.0 🟢 🟢 🟢 🟢 🟢 🟢
archlinux-6.7 🟢 🟢 🟢 🟢 🟢 🟢
centos-3.10 🟢 🟢 🟢 🟡 🟡 🟡
centos-4.18 🟢 🟢 🟢 🟢 🟢 🟢
centos-5.14 🟢 🟢 🟢 🟢 🟢 🟢
fedora-5.17 🟢 🟢 🟢 🟢 🟢 🟢
fedora-5.8 🟢 🟢 🟢 🟢 🟢 🟢
fedora-6.2 🟢 🟢 🟢 🟢 🟢 🟢
oraclelinux-3.10 🟢 🟢 🟢 🟡 🟡 🟡
oraclelinux-4.14 🟢 🟢 🟢 🟢 🟢 🟡
oraclelinux-5.15 🟢 🟢 🟢 🟢 🟢 🟢
oraclelinux-5.4 🟢 🟢 🟢 🟢 🟢 🟡
ubuntu-4.15 🟢 🟢 🟢 🟢 🟢 🟡
ubuntu-5.8 🟢 🟢 🟢 🟢 🟢 🟡
ubuntu-6.5 🟢 🟢 🟢 🟢 🟢 🟢

ARM64 kernel testing matrix

KERNEL CMAKE-CONFIGURE KMOD BUILD KMOD SCAP-OPEN BPF-PROBE BUILD BPF-PROBE SCAP-OPEN MODERN-BPF SCAP-OPEN
amazonlinux2-5.4 🟢 🟢 🟢 🟢 🟢 🟡
amazonlinux2022-5.15 🟢 🟢 🟢 🟢 🟢 🟢
fedora-6.2 🟢 🟢 🟢 🟢 🟢 🟢
oraclelinux-4.14 🟢 🟢 🟢 🟡 🟡 🟡
oraclelinux-5.15 🟢 🟢 🟢 🟢 🟢 🟢
ubuntu-6.5 🟢 🟢 🟢 🟢 🟢 🟢

@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from 6096d33 to 92df24d Compare January 23, 2025 14:10
Copy link

Please double check driver/API_VERSION file. See versioning.

/hold

@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from 3bbfcfd to aa354b6 Compare January 24, 2025 10:46
…or each event.

Try to inject each of them until success.
This allows us to inject `bpf_loop` sendmmsg and recvmmsg programs where supported,
and fallback at just sending first message where it isn't.

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from aa354b6 to 24581f6 Compare January 24, 2025 10:54
@FedeDP FedeDP changed the title wip: new(driver/modern_bpf,userspace/libpman): support multiple programs for each event new(driver/modern_bpf,userspace/libpman): support multiple programs for each event Jan 24, 2025
@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 24, 2025

Removed wip since the impl is complete.
Kernel-testing matrix is now fully green. Only issue remaining are:

  • s390x is not seeing BPF_FUNC_loop symbol in events_prog_names.c
  • amd64 runner is correctly picking up sendmmsg_x and recvmmsg_x (with bpf_loop) but then it fails with:
libbpf: prog 'sendmmsg_x': BPF program load failed: Permission denied
libbpf: prog 'sendmmsg_x': -- BEGIN PROG LOAD LOG --
combined stack size of 2 calls is 576. Too large

This seems like a bug in the verifier since i cannot repro it locally, neither in arm64 CI, neither in kernel testing matrix.
Arm64 runner uses kernel 6.8.0-1020-azure while amd64 uses 6.5.0-1025-azure. Will try to investigate further.

@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 24, 2025

In the last commit i tried to split sendmmsg and recvmmsg programs in case of ret < 0 by chaining a tail call in that case.
Still no luck. I also tried to drop the __always_inline from the handle_exit function and it failed with the same error: https://github.com/falcosecurity/libs/actions/runs/12950456593/job/36123287152

Note also that the exact same code for the 2 bpf programs wasn't failing some weeks ago (ie: at this commit: 67975da).

@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from 0323c41 to 24581f6 Compare January 24, 2025 15:47
@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 24, 2025

/milestone next-driver

@poiana poiana added this to the next-driver milestone Jan 24, 2025
@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from b000c6f to d930932 Compare January 30, 2025 13:48
Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from d930932 to d4e2e2a Compare January 30, 2025 14:32
@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 31, 2025

I tried to reproduce the amd64 failures using virtme-ng:

[virtme-ng][new/support_recvmmsg_sendmmsg_bpf_loop][~/Work/libs/build] $ uname -r
6.5.0-1025-azure

sudo ./libscap/examples/01-open/scap-open --modern_bpf

[SCAP-OPEN]: Hello!

--------------------------- SCAP SOURCE --------------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
------------------------------------------------------------------


------------------------- CONFIGURATIONS -------------------------
* Print single event type: -1 (`-1` means no event to print).
* Run until '18446744073709551615' events are catched.
------------------------------------------------------------------


---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture
^C
----------------------------- STATS ------------------------------

------------> Kernel stats
Seen by driver (kernel side events): 11173
Rate of kernel side events (events/second): 11173
...

No luck :/ this should be the exact same version that is running on the amd64 runner; perhaps it is not?

All i did was:

apt download linux-image-6.5.0-1025-azure linux-modules-6.5.0-1025-azure
for d in *.deb; do dpkg -x $d .; done
vng -r ./boot/vmlinuz-6.5.0-1025-azure

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
@FedeDP FedeDP force-pushed the new/support_recvmmsg_sendmmsg_bpf_loop branch from bb4321f to ffb1207 Compare January 31, 2025 09:03
@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 31, 2025

Oh i should test with same clang version we use in the CI: clang amd64 1:14.0-55~exp2

@Apteryks
Copy link

Hi,

I've rebased my #1842 PR on this branch, and tried to build it with Clang 13 in the environment. I'm getting an undeclared error:

cd /tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/build/libpman && /gnu/store/86fc8bi3mciljxz7c79jx8zr4wsx7xw8-gcc-11.4.0/bin/gcc -DHAVE_SYS_SYSMACROS_H -DPLATFORM_NAME=\"Linux\" -DSCAP_HOSTNAME_ENV_VAR=\"SCAP_HOSTNAME\" -DSCAP_HOST_ROOT_ENV_VAR_NAME=\"HOST_ROOT\" -DSCAP_KERNEL_MODULE_NAME=\"scap\" -D__STDC_FORMAT_MACROS -Dpman_EXPORTS -I/gnu/store/li8wwfm5izk2qwmgm7yvb7bxrvc28wp6-googletest-1.12.1/include/gtest -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace/libpman/include -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace/libpman/src -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/build/skel_dir -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/build -I/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace/libscap/linux -Wall -ggdb -O2 -g -DNDEBUG -fPIC -MD -MT libpman/CMakeFiles/pman.dir/src/events_prog_table.c.o -MF CMakeFiles/pman.dir/src/events_prog_table.c.o.d -o CMakeFiles/pman.dir/src/events_prog_table.c.o -c /tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace/libpman/src/events_prog_table.c
/tmp/guix-build-falcosecurity-libs-0.20.0.drv-0/source/userspace/libpman/src/events_prog_table.c:253:52: error: ‘BPF_FUNC_loop’ undeclared here (not in a function); did you mean ‘BPF_FUNC_bind’?
  253 |         [PPME_SOCKET_RECVMMSG_X] = {{"recvmmsg_x", BPF_FUNC_loop}, {"recvmmsg_old_x", 0}},
      |                                                    ^~~~~~~~~~~~~
      |                                                    BPF_FUNC_bind
make[2]: *** [libpman/CMakeFiles/pman.dir/build.make:191: libpman/CMakeFiles/pman.dir/src/events_prog_table.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....

@FedeDP
Copy link
Contributor Author

FedeDP commented Jan 31, 2025

Which libbpf version are you using?

@Apteryks
Copy link

Apteryks commented Feb 1, 2025

These are the dependencies I tested with:

bpftool@6.12.11 c-ares@1.18.1 clang@13.0.1 curl@8.6.0 elfutils@0.187 googletest@1.12.1 grpc@1.34.0
+ jsoncpp@1.9.5 libbpf@1.4.1 libelf@0.8.13 openssl@3.0.8 pkg-config@0.29.2 protobuf@3.21.9 re2@2022-12-01 tbb@2021.6.0
+ uthash@2.1.0 valijson@1.0.2 zlib@1.3

@FedeDP
Copy link
Contributor Author

FedeDP commented Feb 3, 2025

That's weird, you are using correct versions; it seems like the build is using an older libbpf version though since its headers do not contain BPF_FUNC_loop symbol.
The same happened in the s390x CI: d4e2e2a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

None yet

3 participants