crash on __change_page_attr_set_clr

[alexeadem]

Hi, I found this really cool project to dump the list of watched inodes by process. I was looking specifically to get the the following counters.

inotify Watches 
iinotify Instances

I know I can get them by /proc/390608/fdinfo/7 and looking for

inotify wd:1 ino:1e00028 sdev:fd00002 mask:fce ignored_mask:0 fhandle-bytes:8 fhandle-type:1 f_handle:2800e00143e90e11

But this is a really expensive operation and it is polling one.

I saw this project and I thought it may help to get them directly from the kernel.

Unfortunately the module crashed:

[  +0.000017] CPU: 4 PID: 393896 Comm: insmod Tainted: G           OE      6.3.8-200.fc38.x86_64 #1
[  +0.000006] Hardware name: LENOVO 21CB000FUS/21CB000FUS, BIOS N3AET72W (1.37 ) 03/02/2023
[  +0.000002] RIP: 0010:__change_page_attr_set_clr+0xfd6/0x1140
[  +0.000008] Code: 24 18 49 89 e9 48 89 c2 4c 89 fe 48 c7 c7 d8 13 8d b3 48 89 44 24 08 4c 8d 81 ff 0f 00 00 c6 05 46 dc 22 02 01 e8 ea db 06 00 <0f> 0b 48 8b 44 24 08 e9 3c fc ff ff 48 09 d1 80 3d 2c dc 22 02 00
[  +0.000004] RSP: 0018:ffffb88d8a363b48 EFLAGS: 00010282
[  +0.000003] RAX: 0000000000000000 RBX: 0000000101a4a163 RCX: 0000000000000027
[  +0.000004] RDX: ffff93603f521548 RSI: 0000000000000001 RDI: ffff93603f521540
[  +0.000002] RBP: 0000000000101a4a R08: 0000000000000000 R09: ffffb88d8a3639d8
[  +0.000002] R10: 0000000000000003 R11: ffffffffb4146108 R12: 0000000000000001
[  +0.000002] R13: 8000000101a4a163 R14: 0000000000000001 R15: 8000000000000163
[  +0.000002] FS:  00007efc6c028740(0000) GS:ffff93603f500000(0000) knlGS:0000000000000000
[  +0.000003] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000003] CR2: 000055c23c4d1af0 CR3: 000000017e7fe000 CR4: 0000000000750ee0
[  +0.000002] PKRU: 55555554
[  +0.000002] Call Trace:
[  +0.000004]  <TASK>
[  +0.000003]  ? __change_page_attr_set_clr+0xfd6/0x1140
[  +0.000006]  ? __warn+0x81/0x130
[  +0.000013]  ? __change_page_attr_set_clr+0xfd6/0x1140
[  +0.000006]  ? report_bug+0x171/0x1a0
[  +0.000010]  ? prb_read_valid+0x1b/0x30
[  +0.000007]  ? handle_bug+0x3c/0x80
[  +0.000006]  ? exc_invalid_op+0x17/0x70
[  +0.000005]  ? asm_exc_invalid_op+0x1a/0x20
[  +0.000012]  ? __change_page_attr_set_clr+0xfd6/0x1140
[  +0.000006]  ? __change_page_attr_set_clr+0xfd6/0x1140
[  +0.000007]  ? purge_fragmented_blocks_allcpus+0x5a/0x230
[  +0.000009]  ? _vm_unmap_aliases.part.0+0xf4/0x130
[  +0.000005]  change_page_attr_set_clr+0x104/0x1a0
[  +0.000008]  ? __pfx_module_alloc+0x10/0x10
[  +0.000006]  set_memory_x+0x3b/0x50
[  +0.000009]  khook_init+0xd0/0x170 [inotify_hook]
[  +0.000016]  ? __pfx_init_module+0x10/0x10 [inotify_hook]
[  +0.000013]  inotify_hook_init+0x2a/0xff0 [inotify_hook]
[  +0.000011]  ? __pfx_init_module+0x10/0x10 [inotify_hook]
[  +0.000011]  do_one_initcall+0x5a/0x240
[  +0.000009]  do_init_module+0x4a/0x210
[  +0.000007]  __do_sys_finit_module+0xac/0x120
[  +0.000006]  do_syscall_64+0x5d/0x90
[  +0.000006]  ? ksys_mmap_pgoff+0xec/0x1f0
[  +0.000008]  ? syscall_exit_to_user_mode+0x1b/0x40
[  +0.000006]  ? do_syscall_64+0x6c/0x90
[  +0.000003]  ? exc_page_fault+0x7c/0x180
[  +0.000005]  entry_SYSCALL_64_after_hwframe+0x72/0xdc

Kernel version 6.3.8-200.fc38.x86_64

I'm still doing investigations but I just wanted to share this.

[alexeadem]

I was not using your khook code

Got passed that but cargo test fails on comm_list_add_by_name.

[  +0.000011] ------------[ cut here ]------------
[  +0.000001] kernel BUG at lib/string_helpers.c:1027!
[  +0.000004] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[  +0.000002] CPU: 2 PID: 8839 Comm: run_test Tainted: G        W  OE      6.3.8-200.fc38.x86_64 #1
[  +0.000001] Hardware name: LENOVO 21CB000FUS/21CB000FUS, BIOS N3AET72W (1.37 ) 03/02/2023
[  +0.000001] RIP: 0010:fortify_panic+0x13/0x20
[  +0.000004] Code: 41 5d c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 89 fe 48 c7 c7 10 c9 92 82 e8 cd 79 9a ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90
[  +0.000001] RSP: 0018:ffffb2864161fcc8 EFLAGS: 00010246
[  +0.000002] RAX: 0000000000000022 RBX: ffff90d82e97fac0 RCX: 0000000000000000
[  +0.000001] RDX: 0000000000000000 RSI: ffff90deff4a1540 RDI: ffff90deff4a1540
[  +0.000001] RBP: ffff90d7ca335014 R08: 0000000000000000 R09: ffffb2864161fb70
[  +0.000000] R10: 0000000000000003 R11: ffffffff83146108 R12: ffff90d809f35138
[  +0.000001] R13: 0000000000000004 R14: ffffffff84b66e80 R15: ffff90d7c7e6c700
[  +0.000001] FS:  00007fe2d66066c0(0000) GS:ffff90deff480000(0000) knlGS:0000000000000000
[  +0.000001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000000] CR2: 00007fe2d0008000 CR3: 0000000146c7a000 CR4: 0000000000750ee0
[  +0.000001] PKRU: 55555554
[  +0.000001] Call Trace:
[  +0.000001]  <TASK>
[  +0.000002]  ? die+0x36/0x90
[  +0.000002]  ? do_trap+0xda/0x100
[  +0.000002]  ? fortify_panic+0x13/0x20
[  +0.000001]  ? do_error_trap+0x6a/0x90
[  +0.000001]  ? fortify_panic+0x13/0x20
[  +0.000001]  ? exc_invalid_op+0x50/0x70
[  +0.000003]  ? fortify_panic+0x13/0x20
[  +0.000002]  ? asm_exc_invalid_op+0x1a/0x20
[  +0.000003]  ? fortify_panic+0x13/0x20
[  +0.000001]  ? fortify_panic+0x13/0x20
[  +0.000003]  comm_list_add_by_name+0x17f/0x180 [inotify_hook]
[  +0.000004]  nl_recv_msg+0x4a/0x160 [inotify_hook]
[  +0.000003]  netlink_unicast+0x19e/0x290
[  +0.000003]  netlink_sendmsg+0x254/0x4d0
[  +0.000001]  sock_sendmsg+0x93/0xa0
[  +0.000003]  ? sockfd_lookup_light+0x12/0x70
[  +0.000001]  __sys_sendto+0x120/0x170
[  +0.000002]  __x64_sys_sendto+0x24/0x30
[  +0.000001]  do_syscall_64+0x5d/0x90
[  +0.000002]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  +0.000002] RIP: 0033:0x7fe2d671d84a
[  +0.000024] Code: e0 89 7d e8 89 4d d4 e8 54 42 f7 ff 44 8b 55 d4 48 8b 55 d8 45 31 c9 89 c3 48 8b 75 e0 8b 7d e8 45 31 c0 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 36 89 df 48 89 45 e8 e8 a3 42 f7 ff 48 8b 45
[  +0.000001] RSP: 002b:00007fe2d6604140 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[  +0.000001] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe2d671d84a
[  +0.000001] RDX: 0000000000000414 RSI: 00007fe2d0009080 RDI: 0000000000000003
[  +0.000001] RBP: 00007fe2d6604170 R08: 0000000000000000 R09: 0000000000000000
[  +0.000000] R10: 0000000000000000 R11: 0000000000000246 R12: 00005644f38fc2a0
[  +0.000001] R13: ffffffffffffff00 R14: 00007fe2d0000ca0 R15: 0000000000000000
[  +0.000001]  </TASK>
[  +0.000001] Modules linked in: inotify_hook(OE) uinput xt_MASQUERADE nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc overlay rfcomm snd_seq_dummy snd_hrtimer qrtr nft_chain_nat nf_nat nft_limit xt_LOG nf_log_syslog xt_limit xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 nft_compat bnep nf_tables nfnetlink sunrpc binfmt_misc snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda iwlmvm snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda mac80211 snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus intel_tcc_cooling snd_soc_core x86_pkg_temp_thermal intel_powerclamp coretemp snd_compress kvm_intel ac97_bus snd_pcm_dmaengine snd_hda_intel libarc4 snd_intel_dspcfg snd_intel_sdw_acpi mei_hdcp mei_wdt
[  +0.000028]  mei_pxp snd_hda_codec kvm snd_hda_core uvcvideo iTCO_wdt btusb intel_pmc_bxt irqbypass snd_hwdep iTCO_vendor_support btrtl uvc rapl btbcm snd_seq videobuf2_vmalloc btintel iwlwifi videobuf2_memops pmt_telemetry processor_thermal_device_pci intel_cstate intel_rapl_msr pmt_class snd_seq_device videobuf2_v4l2 btmtk processor_thermal_device thinkpad_acpi snd_pcm videobuf2_common vfat think_lmi processor_thermal_rfim ledtrig_audio intel_uncore pcspkr fat firmware_attributes_class wmi_bmof snd_timer cfg80211 platform_profile bluetooth processor_thermal_mbox videodev processor_thermal_rapl mei_me snd i2c_i801 intel_rapl_common int3403_thermal thunderbolt mc mei i2c_smbus intel_hid rfkill idma64 intel_vsec igen6_edac soundcore soc_button_array int340x_thermal_zone sparse_keymap acpi_pad acpi_tad int3400_thermal acpi_thermal_rel joydev loop zram dm_crypt i915 i2c_algo_bit nvme drm_buddy drm_display_helper nvme_core crct10dif_pclmul crc32_pclmul crc32c_intel cec video polyval_clmulni ucsi_acpi polyval_generic
[  +0.000029]  hid_multitouch ghash_clmulni_intel typec_ucsi sha512_ssse3 ttm nvme_common typec i2c_hid_acpi i2c_hid wmi pinctrl_tigerlake serio_raw ip6_tables ip_tables fuse
[  +0.000006] Unloaded tainted modules: inotify_hook(OE):3 [last unloaded: inotify_hook(OE)]
[  +0.000003] ---[ end trace 0000000000000000 ]---

[iamhyc]

Hi, glad to know that someone like this project 😄

Firstly, I have ran cargo test on Ubuntu 22.04 with Linux kernel 5.19, everything works fine:

$ cargo test
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests src/lib.rs (target/debug/deps/inotify_lookup-da38c28cdf8ba2c9)

running 1 test
test run_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s

and the dmesg shows:

[281981.775540] inotify_hook: Inotify hook module init.
[281986.539820] inotify_hook: comm_list add "code"
[282007.983493] inotify_hook: Inotify hook module exit.

I noticed you may have removed khook due to init error on 6.3 kernel. Maybe the higher kernel version is to blame.

[alexeadem]

This project rocks.

Actually I found the problem

https://github.com/VDM-Maintainer-Group/vdm-capability-library/blob/d27aa2eeeb6dc063752f422e3940d8e3e63e342b/inotify-lookup/inotify-hook/main.c#L212C12-L212C12

There is a string overflow with strcpy. strcpy will add the null character and there was not space for it as strlen only gets the length of the string without the null. Maybe newet gcc/kernel are more strict with this.

The fix is:

TRY_BUF( item->comm_name, strlen(name) + 1) {

Isn't the cargo test cargo test supposed to return all the max_user_watches paths being used by code.

As per:
https://github.com/VDM-Maintainer-Group/vdm-capability-library/blob/d27aa2eeeb6dc063752f422e3940d8e3e63e342b/inotify-lookup/inotify-hook/main.c#L330C18-L330C19

What I was looking for is the total counter of

queued_events counters
user_instances counters
user_watches counters

Via the module.

The problem I'm having is that this doesn't trigger all the time

https://github.com/VDM-Maintainer-Group/vdm-capability-library/blob/d27aa2eeeb6dc063752f422e3940d8e3e63e342b/inotify-lookup/inotify-hook/main.c#L330C18-L330C19

Trying to figure out why...

Thanks for the help

Alex

[iamhyc]

There is a string overflow with strcpy. strcpy will add the null character and there was not space for it as strlen only gets the length of the string without the null. Maybe newet gcc/kernel are more strict with this.

The fix is:

TRY_BUF( item->comm_name, strlen(name) + 1) {

Thank you! I think you are right. I need to check the usage of all strlen in the code.

Isn't the cargo test cargo test supposed to return all the max_user_watches paths being used by code.

As per: https://github.com/VDM-Maintainer-Group/vdm-capability-library/blob/d27aa2eeeb6dc063752f422e3940d8e3e63e342b/inotify-lookup/inotify-hook/main.c#L330C18-L330C19

What I was looking for is the total counter of

queued_events counters user_instances counters user_watches counters

Via the module.

If I get you correctly, you are trying to hook inotify_add_watch and inotify_rm_watch, and then manually count the global number of queued_events, user_instances, user_watches.

Then you don't have to use this inotify-lookup, just copy the hook part and do your own magic!

But remind that the counters will miss all the things happening before the hook.