support linux musl

[bruno-garcia]

Relates to:

• Errors with sentry-native launching with Sentry.AspNetCore on Dockerized Native AOT builds on Linux sentry-dotnet#4127

[jpnurmi]

Preliminary analysis:

Unwinder: missing stacktrace

There's no execinfo.h in musl:

sentry-native/src/unwinder/sentry_unwinder_libbacktrace.c

Lines 3 to 11 in 8755115

	// XXX: Make into a CMake check
	// XXX: IBM i PASE offers libbacktrace in libutil, but not available in AIX
	#if defined(SENTRY_PLATFORM_DARWIN) || defined(__GLIBC__) || defined(__PASE__)
	# define HAS_EXECINFO_H
	#endif
	#ifdef HAS_EXECINFO_H
	# include <execinfo.h>
	#endif

$ make test-unit
[...]
Test unwinder...                                [ FAILED ]
  test_unwinder.c:15: Check frame_count > 0... failed
  test_unwinder.c:58: Check 0 == 1... failed
[...]
Test value_set_stacktrace...                    [ FAILED ]
  test_value.c:700: Check 0 < sentry_value_get_length(frames)... failed
FAILED: 2 of 157 unit tests have failed.

Crashpad: compilation errors

1. crashpad includes glibc-specific <sys/cdefs.h> without appropriate __GLIBC__ guard

   In file included from sentry-native/external/crashpad/util/linux/scoped_ptrace_attach.cc:17:
   sentry-native/external/crashpad/compat/linux/sys/ptrace.h:20:10: fatal error: 
   sys/cdefs.h: No such file or directory
      20 | #include <sys/cdefs.h>
         |          ^~~~~~~~~~~~~

   Potential fix:

   diff --git a/compat/linux/sys/ptrace.h b/compat/linux/sys/ptrace.h
   index 604a46ab..bc1d383d 100644
   --- a/compat/linux/sys/ptrace.h
   +++ b/compat/linux/sys/ptrace.h
   @@ -17,7 +17,9 @@
   
    #include_next <sys/ptrace.h>
   
   +#ifdef __GLIBC__
    #include <sys/cdefs.h>
   +#endif
   
    // https://sourceware.org/bugzilla/show_bug.cgi?id=22433
    #if !defined(PTRACE_GET_THREAD_AREA) && !defined(PT_GET_THREAD_AREA) && \

2. crashpad uses pread64, off64_t etc.

   In file included from sentry-native/external/crashpad/util/process/process_memory_linux.cc:27:
   sentry-native/external/crashpad/util/process/process_memory_linux.cc: In lambda function:
   sentry-native/external/crashpad/util/process/process_memory_linux.cc:48:24: error: 'pread64' was not declared in this scope; did you mean 'pread'?
      48 |           HANDLE_EINTR(pread64(mem_fd_.get(), buffer, size, address));
         |                        ^~~~~~~
   
   In file included from sentry-native/external/crashpad/util/linux/memory_map.cc:15:
   sentry-native/external/crashpad/util/linux/memory_map.h:46:5: error: 'off64_t' does not name a type; did you mean 'off_t'?
      46 |     off64_t offset;
         |     ^~~~~~~
         |     off_t

   Potential fix:

   diff --git a/CMakeLists.txt b/CMakeLists.txt
   index 7b25d488..7fa393a3 100644
   --- a/CMakeLists.txt
   +++ b/CMakeLists.txt
   @@ -143,6 +143,14 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
        )
    endif()
   
   +if(NOT MSVC)
   +    target_compile_definitions(crashpad_interface INTERFACE
   +        _FILE_OFFSET_BITS=64
   +        _LARGEFILE64_SOURCE
   +        _LARGEFILE_SOURCE
   +    )
   +endif()
   +
    add_library(crashpad::interface ALIAS crashpad_interface)
   
    add_subdirectory(compact)

   Q: Do I need to define _LARGEFILE64_SOURCE to get 64bit off_t?

[jpnurmi]

Both breakpad and crashpad are able to provide a stacktrace:

[supervacuus]

Thanks, @jpnurmi. Yeah, I remember execinfo being a problem on musl (through previous exposure). Since the downstream SDK use case requires the inproc backend (which uses that unwinder in this configuration) and having stack traces outside the crash handler is also welcome, we need to find a replacement for this. Since this is the first scenario that requests this and is bound to a specific image/Linux distro, can you check what is available in that image (rather than vendoring another unwinder)?

There is already an attempt for libunwind in the pipeline for the Native SDK, and I could also imagine adding an implementation for the actual libbacktrace. However, any attempt to add an additional unwinder should first check if the target supports both of our use cases (i.e., recovering from the signal handler + during regular operation).

[jpnurmi]

The image has libunwind v1.8.1 available but not installed by default:

$ docker run --rm -ti mcr.microsoft.com/dotnet/nightly/sdk:9.0.201-alpine3.21-aot
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.21/community/x86_64/APKINDEX.tar.gz
v3.21.3-452-gee5ce66b724 [https://dl-cdn.alpinelinux.org/alpine/v3.21/main]
v3.21.3-457-g7e6cabc3b2d [https://dl-cdn.alpinelinux.org/alpine/v3.21/community]
OK: 25405 distinct packages available
/ # apk search backtrace
openvswitch-2.17.11-r0
/ # apk search unwind
libunwind-1.8.1-r0
libunwind-dbg-1.8.1-r0
libunwind-dev-1.8.1-r0
libunwind-doc-1.8.1-r0
libunwind-static-1.8.1-r0
llvm-libunwind-19.1.4-r1
llvm-libunwind-dev-19.1.4-r1
llvm-libunwind-static-19.1.4-r1
/ # apk info libunwind
libunwind-1.8.1-r0 description:
Portable and efficient C programming interface (API) to determine the call-chain of a program

libunwind-1.8.1-r0 webpage:
https://www.nongnu.org/libunwind/

libunwind-1.8.1-r0 installed size:
175 KiB
/ # apk info --installed libunwind
/ # apk add libunwind
(1/1) Installing libunwind (1.8.1-r0)
OK: 550 MiB in 63 packages
/ # apk info --installed libunwind
libunwind

Should I give it a try with the fix/use_libunwind_as_macos_unwinder branch?

[jpnurmi]

Crash after rebasing fix/use_libunwind_as_macos_unwinder and linking to libunwind:

Test unwinder...
Program received signal SIGSEGV, Segmentation fault.
0x000073b2c0f9b53b in _ULx86_64_get_reg () from /usr/lib/libunwind.so.8
(gdb) bt
#0  0x000073b2c0f9b53b in _ULx86_64_get_reg () from /usr/lib/libunwind.so.8
#1  0x00005f238aef69a7 in sentry__unwind_stack_libunwind (addr=addr@entry=0x0, uctx=uctx@entry=0x0, ptrs=ptrs@entry=0x7ffd165ed160, max_frames=max_frames@entry=128) at /root/sentry-native/src/unwinder/sentry_unwinder_libunwind.c:43
#2  0x00005f238aef22dd in unwind_stack (addr=addr@entry=0x0, uctx=0x0, ptrs=ptrs@entry=0x7ffd165ed160, max_frames=128) at /root/sentry-native/src/unwinder/sentry_unwinder.c:34
#3  0x00005f238af09210 in invoke_unwinder (backtrace=backtrace@entry=0x7ffd165ed160) at /root/sentry-native/tests/unit/test_unwinder.c:13
#4  0x00005f238af092ae in test_sentry_unwinder () at /root/sentry-native/tests/unit/test_unwinder.c:45
#5  0x00005f238aefc4fa in test_do_run__ (test=test@entry=0x5f238af2f7c0 <test_list__+1888>, index=index@entry=0) at /root/sentry-native/include/../vendor/acutest.h:933
#6  0x00005f238aeccd53 in test_run__ (test=0x5f238af2f7c0 <test_list__+1888>, index=0, master_index=118) at /root/sentry-native/include/../vendor/acutest.h:1104
#7  main (argc=<optimized out>, argv=<optimized out>) at /root/sentry-native/include/../vendor/acutest.h:1608

A small tweak helps:

diff --git a/src/unwinder/sentry_unwinder_libunwind.c b/src/unwinder/sentry_unwinder_libunwind.c
index b13c50b..f92b8cb 100644
--- a/src/unwinder/sentry_unwinder_libunwind.c
+++ b/src/unwinder/sentry_unwinder_libunwind.c
@@ -40,7 +40,9 @@ sentry__unwind_stack_libunwind(

     size_t frame_idx = 0;
     while (unw_step(&cursor) > 0 && frame_idx < max_frames - 1) {
-        unw_get_reg(&cursor, UNW_REG_IP, ptrs[frame_idx]);
+        unw_word_t ip = 0;
+        unw_get_reg(&cursor, UNW_REG_IP, &ip);
+        ptrs[frame_idx] = (void *)ip;
         unw_word_t sp = 0;
         unw_get_reg(&cursor, UNW_REG_SP, &sp);
         printf("ip = %lx, sp = %lx\n", (long)ptrs[frame_idx], (long)sp);

Unit tests pass:

Test unwinder...                                ip = 56db69220250, sp = 7ffdbc045e30
ip = 56db692202ee, sp = 7ffdbc045e40
ip = 56db6921353a, sp = 7ffdbc0466b0
ip = 56db691e43f2, sp = 7ffdbc046710
ip = 7913e86c2496, sp = 7ffdbc046800
ip = 0, sp = 7ffdbc046848
ip = 7ffdbc046d23, sp = 7ffdbc046850
ip = 6e65732f646c6975, sp = 7ffdbc046d15
[ OK ]
Test value_set_stacktrace...                    ip = 56db69205d24, sp = 7ffdbc045dc0
ip = 56db692065d9, sp = 7ffdbc046650
ip = 56db6922611a, sp = 7ffdbc046680
ip = 56db6921353a, sp = 7ffdbc0466b0
ip = 56db691e43f2, sp = 7ffdbc046710
ip = 7913e86c2496, sp = 7ffdbc046800
ip = 0, sp = 7ffdbc046848
ip = 7ffdbc046d23, sp = 7ffdbc046850
ip = 6e65732f646c6975, sp = 7ffdbc046d15
[ OK ]

Capture crash:

# SENTRY_DEBUG=1 ./build/inproc/sentry_example crash
[sentry] INFO using database path "/root/sentry-native/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG installing signal stack (size: 65536)
[sentry] DEBUG background worker thread started
[sentry] DEBUG processing and pruning old runs
[sentry] INFO entering signal handler
ip = 7032a55624bd, sp = 7032a4bad140
ip = 7032a5562a71, sp = 7032a4bad5a0
ip = 7032a55630d9, sp = 7032a4bad5d0
ip = 7032a55dd5a4, sp = 7032a4bad600
[sentry] DEBUG captured backtrace from ucontext with 5 frames
[sentry] DEBUG captured backtrace with 5 frames
[sentry] DEBUG merging scope into event
[sentry] DEBUG trying to read modules from /proc/self/maps
[sentry] DEBUG read 18 modules from /proc/self/maps
[sentry] DEBUG adding attachments to envelope
[sentry] DEBUG sending envelope
[sentry] INFO crash has been captured
Segmentation fault (core dumped)

Send envelope:

# SENTRY_DEBUG=1 ./build/inproc/sentry_example
[sentry] INFO using database path "/root/sentry-native/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG installing signal stack (size: 65536)
[sentry] DEBUG background worker thread started
[sentry] DEBUG processing and pruning old runs
[sentry] DEBUG sending envelope
[sentry] DEBUG submitting task to background worker thread
[sentry] DEBUG executing task on worker thread
[sentry] DEBUG shutting down backend
*   Trying 192.168.86.46:9000...
[sentry] DEBUG shutting down transport
[sentry] DEBUG shutting down background worker thread
[sentry] DEBUG submitting task to background worker thread
* Connected to 192.168.86.46 (192.168.86.46) port 9000
* using HTTP/1.x
> POST /api/3/envelope/ HTTP/1.1
Host: 192.168.86.46:9000
User-Agent: sentry.native/0.8.4
Accept: */*
x-sentry-auth:Sentry sentry_key=72469cd3127ecb1bd25b1809a49c588f, sentry_version=7, sentry_client=sentry.native/0.8.4
content-type:application/x-sentry-envelope
content-length:6230

* upload completely sent off: 6230 bytes
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 08 May 2025 17:08:36 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< vary: origin, access-control-request-method, access-control-request-headers
< access-control-allow-origin: *
< access-control-expose-headers: x-sentry-error,x-sentry-rate-limits,retry-after
< cross-origin-resource-policy: cross-origin
<
{"id":"8404cd1c792b46559040b75e6b37c4c8"}* Connection #0 to host 192.168.86.46 left intact
[sentry] DEBUG executing task on worker thread
[sentry] DEBUG background worker thread shut down

Handler's stacktrace shows up:

And finally, after enabling the user context init in sentry__unwind_stack_libunwind 🎉

[jpnurmi]

Looks like there's already an old vendored copy of LLVM libunwind in external/crashpad/libunwind even though it's only used on macOS. On Linux, nongnu.org/libunwind is used from the system via pkg-config:

if(APPLE AND NOT IOS AND CRASHPAD_ENABLE_STACKTRACE)
    target_include_directories(crashpad_snapshot PRIVATE ../libunwind/include)
    target_link_libraries(crashpad_snapshot PRIVATE unwind_static)
endif()

if(LINUX AND CRASHPAD_ENABLE_STACKTRACE)
    find_package(PkgConfig REQUIRED)
    pkg_check_modules(UNWIND REQUIRED IMPORTED_TARGET libunwind-ptrace)
    target_link_libraries(crashpad_snapshot PRIVATE PkgConfig::UNWIND)
endif()

https://github.com/getsentry/crashpad/blob/2d97a484bb628b05a46203e4a05c6c827796e5cd/snapshot/CMakeLists.txt#L239-L248

@supervacuus Do you happen to know why this approach was chosen? Would libunwind-dev be a reasonable dependency on musl-based Linux systems, or should we try reusing the existing vendored copy in external/crashpad/libunwind?

[supervacuus]

@supervacuus Do you happen to know why this approach was chosen? Would libunwind-dev be a reasonable dependency on musl-based Linux systems, or should we try reusing the existing vendored copy in external/crashpad/libunwind?

Yes, this was chosen to allow for client-side stack traces gathered from the crashpad_handler (rather than from within the application), which wasn't supported by the macOS default libunwind then.

Please let's not use it:

• This is a very old commit of that particular repo/branch and no longer maintained upstream (they removed the features in the next commit :-))
• The feature for which we use it is marked as experimental in our crashpad fork, whereas basic stack-tracing support for musl (or any target, really) should "just work".
• We already know that the client-side stack traces on macOS fail in some cases, so it isn't working correctly even for that particular use case. I would not rely on it working correctly in the remaining code base either.

If we support Musl, I would prefer to introduce libunwind as a system dependency, if possible, as part of the deployment workflow. If that isn't possible, we'd have to introduce it as a vendored dependency (although I really would like to prevent that).

[jpnurmi]

If we support Musl, I would prefer to introduce libunwind as a system dependency, if possible, as part of the deployment workflow.

Initial proposal with libunwind as a system dependency when musl is detected:

• fix: support musl on Linux #1233
• fix: compilation errors with musl crashpad#125

If that isn't possible, we'd have to introduce it as a vendored dependency (although I really would like to prevent that).

👍 If we have to, vendoring the LLVM version of libunwind seems easy because it's CMake-based. Porting the GNU version's autotools build system is probably quite a bit more work.

[supervacuus]

Initial proposal with libunwind as a system dependency when musl is detected:

Thanks @jpnurmi! I commented there.

👍 If we have to, vendoring the LLVM version of libunwind seems easy because it's CMake-based. Porting the GNU version's autotools build system is probably quite a bit more work.

My preferred approach for .NET would be to use the static libunwind from Alpine and distribute the Native SDK binary for this libc target with libunwind included. For other users, it is fine to make it a system dependency (i.e., that case is covered with your PR).