SIGSEGV when using asan-cores in qemu_launcher fuzzer

[jthorpe6]

IMPORTANT

1. You have verified that the issue to be present in the current main branch
   yes

% git log | head -n 1
commit a212d66afef398cefb5ab7d1844e935efb702d23

Describe the bug
I get a SIGSEGV when trying to run the qemu_launcher example with asan

To Reproduce
Steps to reproduce the behavior:

add cargo-make to the docker file

% echo -e "RUN cargo install cargo-make" >> Dockerfile

build the docker container image

% docker build --platform linux/amd64 -t libafl .

run the docker container, clone LibAFL and run qemu_launcher with asan

% docker run --rm --platform linux/amd64 -it libafl
root@f072d778bf3f:/libafl# cd /tmp
root@f072d778bf3f:/tmp# git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan

Cloning into 'LibAFL'...
remote: Enumerating objects: 49623, done.
remote: Counting objects: 100% (10040/10040), done.
remote: Compressing objects: 100% (2608/2608), done.
remote: Total 49623 (delta 8480), reused 8273 (delta 7371), pack-reused 39583 (from 1)
Receiving objects: 100% (49623/49623), 22.69 MiB | 5.70 MiB/s, done.
Resolving deltas: 100% (34798/34798), done.
[cargo-make] INFO - cargo make 0.37.18
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Project: qemu_launcher
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: asan
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: target_dir
[cargo-make] INFO - Running Task: arch_target_dir
[cargo-make] INFO - Running Task: deps_dir
[cargo-make] INFO - Running Task: zlib_unix_wget

SNIP

[cargo-make] INFO - Running Task: harness
[cargo-make] INFO - Execute Command: "cargo" "build" "--profile" "release" "--features" "x86_64" "--target-dir" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64"
   Compiling proc-macro2 v1.0.86
   Compiling unicode-ident v1.0.13
   Compiling rustversion v1.0.17
   
SNIP

Compiling qemu_launcher v0.13.2 (/tmp/LibAFL/fuzzers/binary_only/qemu_launcher)
   Compiling getset v0.1.3
   Compiling clap_derive v4.5.18
   Compiling thread_local v1.1.8
   Compiling num-derive v0.4.2
   Compiling syscall-numbers v4.0.0
   Compiling clap v4.5.19
   Compiling readonly v0.2.12
   Compiling capstone v0.12.0
warning: [email protected]: Qemu not found, cloning with git (d6637939526f453c69f4c6bfe4635feb5dc5c0be)...
warning: [email protected]: Version of llvm-config is 18 but needs to be at least rustc's version (19)! We will (try to) continue to build. Continue at your own risk, or rebuild with a set LLVM_CONFIG_PATH env variable, pointing to a newer version.
warning: [email protected]: wrapper.h has been regenerated.
warning: field `page_filter` is never read
   --> /tmp/LibAFL/libafl_qemu/src/modules/edges.rs:360:5
    |
357 | pub struct EdgeCoverageModule<AF, PF, V> {
    |            ------------------ field in this struct
...
360 |     page_filter: PF,
    |     ^^^^^^^^^^^
    |
    = note: `EdgeCoverageModule` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
    = note: `#[warn(dead_code)]` on by default

warning: `libafl_qemu` (lib) generated 1 warning
    Finished `release` profile [optimized + debuginfo] target(s) in 6m 18s
    
[cargo-make] INFO - Running Task: fuzzer
[cargo-make] INFO - Execute Command: "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development" "--input" "./corpus" "--output" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/" "--log" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt" "--cores" "0" "--asan-cores" "0" "--" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development"
Fuzzing stopped by user. Good bye.
[cargo-make] INFO - Build Done in 456.63 seconds.
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher#       

Try run the fuzzer with the --verbose flag

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071506a0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

Add the debug / log environment variables

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_LOG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_DEBUG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
==13629== QEMU-AddressSanitizer (v0.3)
==13629== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13629== 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071c12c0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

Environment info

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher#  lsb_release -a && \
    arch && \
    llvm-config --version && \
    rustup toolchain list && \
    rustc -V

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm
x86_64
18.1.8
nightly-x86_64-unknown-linux-gnu (default)
1.76.0-x86_64-unknown-linux-gnu
rustc 1.83.0-nightly (06bb8364a 2024-10-01)

Expected behavior
when using asan cores not to cause a SIGSEGV

Additional context
I've also tried llvm-19 because of that warning during the compilation of libafl_qemu_sys

first remove all docker images/container

% docker rmi -f $(docker images -aq) ; docker rm -f $(docker ps -aq); docker system prune -f ; docker rmi $(docker images -aq); docker rm $(docker images -aq)

Then change LLVM version from 18 to 19

% sed -i s/18/19/g ./Dockerfile
% cat Dockerfile | grep 19   
# Install clang 19, common build tools
ENV LLVM_VERSION=19

build the container image again

docker build --platform linux/amd64 -t libafl .

clone LibAFL and attempt the qemu_launcher example with asan again

docker run --rm --platform linux/amd64 -it libafl
cd /tmp
git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan

No llvm version warning, however asan cores still crashes.

export QASAN_LOG=1
export QASAN_DEBUG=1
/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development

==13784== QEMU-AddressSanitizer (v0.3)
==13784== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13784== 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x20007532860b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

[jthorpe6]

I'd like to add I did try adding the --privileged flag when running the docker container, which did not appear to change any of the results.

[rmalmain]

thank you for the report, i'll take a look at this during the week

[tokatoka]

is this fixed?

[jthorpe6]

Hello, I don't think so.

docker build --platform linux/amd64 -t libafl .

In the container

cargo install cargo-make
cd /tmp && git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan

I did get an error about an unresolved import which I fixed by adding the prelude feature to libafl_bolts in Cargo.toml. I then ran cargo make asan again

[cargo-make] INFO - Execute Command: "cargo" "build" "--profile" "release" "--features" "x86_64" "--target-dir" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64"
warning: unused manifest key: profile.release-abort.abort
   Compiling libafl_bolts v0.13.2 (/tmp/LibAFL/libafl_bolts)
   Compiling libafl_qemu_sys v0.13.2 (/tmp/LibAFL/libafl_qemu/libafl_qemu_sys)
warning: [email protected]: Version of llvm-config is 18 but needs to be at least rustc's version (19)! We will (try to) continue to build. Continue at your own risk, or rebuild with a set LLVM_CONFIG_PATH env variable, pointing to a newer version.
   Compiling libafl_qemu v0.13.2 (/tmp/LibAFL/libafl_qemu)
   Compiling libafl v0.13.2 (/tmp/LibAFL/libafl)
   Compiling libafl_targets v0.13.2 (/tmp/LibAFL/libafl_targets)
   Compiling qemu_launcher v0.13.2 (/tmp/LibAFL/fuzzers/binary_only/qemu_launcher)
    Finished `release` profile [optimized + debuginfo] target(s) in 2m 49s
[cargo-make] INFO - Running Task: fuzzer
[cargo-make] INFO - Execute Command: "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development" "--input" "./corpus" "--output" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/" "--log" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt" "--cores" "0" "--asan-cores" "0" "--" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development"
[Objective   #1]  (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
[Objective   #1]  (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
[Objective   #1]  (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
[Objective   #1]  (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
Fuzzing stopped by user. Good bye.
[cargo-make] INFO - Build Done in 174.00 seconds.

If I add the --verbose flag, we see the SIGSEGV as before.

/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective   #1]  (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
Context:
   Rax: 0x00000000000178    Rbx: 0x007ffffe07f760    Rcx: 0x00000000000178    Rdx: 0x00000000000000 
   Rsi: 0x00000000000178    Rdi: 0x00000000000178    Rbp: 0x007ffffe07f610    Rsp: 0x007ffffe07f5e8 
    R8: 0x00000000000000     R9: 0x007ffffd84f680    R10: 0x007ffffd84b878    R11: 0x007ffffd861940 
   R12: 0x00000000000000    R13: 0x007ffffe07f948    R14: 0x00ffff84d7dc38    R15: 0x007ffffd87d020 
   Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective   #1]  (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
Context:
   Rax: 0x000000000000da    Rbx: 0x007ffffe07f760    Rcx: 0x000000000000da    Rdx: 0x00000000000000 
   Rsi: 0x000000000000da    Rdi: 0x000000000000da    Rbp: 0x007ffffe07f610    Rsp: 0x007ffffe07f5e8 
    R8: 0x00000000000000     R9: 0x007ffffd84f680    R10: 0x007ffffd84b878    R11: 0x007ffffd861940 
   R12: 0x00000000000000    R13: 0x007ffffe07f948    R14: 0x00ffff84d7dc38    R15: 0x007ffffd87d020 
   Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective   #1]  (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
Context:
   Rax: 0x000000000001ab    Rbx: 0x007ffffe07f760    Rcx: 0x000000000001ab    Rdx: 0x00000000000000 
   Rsi: 0x000000000001ab    Rdi: 0x000000000001ab    Rbp: 0x007ffffe07f610    Rsp: 0x007ffffe07f5e8 
    R8: 0x00000000000000     R9: 0x007ffffd84f680    R10: 0x007ffffd84b878    R11: 0x007ffffd861940 
   R12: 0x00000000000000    R13: 0x007ffffe07f948    R14: 0x00ffff84d7dc38    R15: 0x007ffffd87d020 
   Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective   #1]  (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
                  (CLIENT) corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
Context:
   Rax: 0x000000000000e4    Rbx: 0x007ffffe07f760    Rcx: 0x000000000000e4    Rdx: 0x00000000000000 
   Rsi: 0x000000000000e4    Rdi: 0x000000000000e4    Rbp: 0x007ffffe07f610    Rsp: 0x007ffffe07f5e8 
    R8: 0x00000000000000     R9: 0x007ffffd84f680    R10: 0x007ffffd84b878    R11: 0x007ffffd861940 
   R12: 0x00000000000000    R13: 0x007ffffe07f948    R14: 0x00ffff84d7dc38    R15: 0x007ffffd87d020 
   Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202 
We imported 0 inputs from disk.
thread 'main' panicked at src/main.rs:20:26:
called `Result::unwrap()` on an `Err` value: Empty("No entries in corpus. This often implies the target is not properly instrumented.",    0: libafl_bolts::Error::empty
             at /tmp/LibAFL/libafl_bolts/src/lib.rs:371:34
      <libafl::schedulers::powersched::PowerQueueScheduler<C,O> as libafl::schedulers::Scheduler<I,S>>::next
             at /tmp/LibAFL/libafl/src/schedulers/powersched.rs:348:17
   1: <libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M,O> as libafl::schedulers::Scheduler<<<S as libafl::state::HasCorpus>::Corpus as libafl::corpus::Corpus>::Input,S>>::next
             at /tmp/LibAFL/libafl/src/schedulers/minimizer.rs:220:22
   2: <libafl::fuzzer::StdFuzzer<CS,F,OF,S> as libafl::fuzzer::Fuzzer<E,EM,ST>>::fuzz_one
             at /tmp/LibAFL/libafl/src/fuzzer/mod.rs:777:22
   3: libafl::fuzzer::Fuzzer::fuzz_loop
             at /tmp/LibAFL/libafl/src/fuzzer/mod.rs:247:13
      qemu_launcher::instance::Instance<M>::fuzz
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/instance.rs:327:13
   4: qemu_launcher::instance::Instance<M>::run
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/instance.rs:290:13
   5: qemu_launcher::client::Client::run
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/client.rs:183:17
   6: qemu_launcher::fuzzer::Fuzzer::launch::{{closure}}
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:145:35
      core::ops::function::FnOnce::call_once
             at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/core/src/ops/function.rs:250:5
      libafl::events::launcher::Launcher<CF,MT,SP>::launch_with_hooks
             at /tmp/LibAFL/libafl/src/events/launcher.rs:289:32
      libafl::events::launcher::Launcher<CF,MT,SP>::launch
             at /tmp/LibAFL/libafl/src/events/launcher.rs:181:9
      qemu_launcher::fuzzer::Fuzzer::launch
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:150:14
      qemu_launcher::fuzzer::Fuzzer::fuzz
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:81:13
   7: qemu_launcher::main
             at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/main.rs:20:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/core/src/ops/function.rs:250:5
      std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/std/src/sys/backtrace.rs:154:18
   9: main
  10: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  11: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  12: _start
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 101)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[rmalmain]

@jthorpe6 do you mind trying to run the fuzzer with gdb and checking where the segfault happens on your side?
just to check if this the same problem as in #2617

[jthorpe6]

Hey @rmalmain

I have just noticed that I do not see a SIGSEGV if the host OS that I'm running the docker container on is Linux. My main host OS is macOS, so I suspect that Rosetta is doing something when mapping memory.

Using the container on a macOS host, I am unable to run the fuzzer with gdb. This seems to be the problem I'm seeing when trying to debug.

[domenukk]

Are you running an arm docker container or a x86 docker container?

[jthorpe6]

x86 due to the --platform linux/amd64 flag

[domenukk]

So anyway running an emulator in an emulator sounds really bad for performance, why not just run it on aarch?

[jthorpe6]

oh I don't disagree with the performance hit. The fuzzer I'm working on is x86, so I'm only using an emulator inside of an emulator for development.

[domenukk]

Ideally the emulator should cross compile to aarch64 just fine - so you don't necessarily have to develop on the same architecture.

That being said, of course it shouldn't really segfault on emulated x86, either.