Skip to content

Fix segfault importing onnxruntime from static init order fiasco - #29880

Merged
tianleiwu merged 1 commit into
mainfrom
tlwu/20260725/fix_env_crash
Jul 27, 2026
Merged

Fix segfault importing onnxruntime from static init order fiasco#29880
tianleiwu merged 1 commit into
mainfrom
tlwu/20260725/fix_env_crash

Conversation

@tianleiwu

Copy link
Copy Markdown
Contributor

Description

import onnxruntime segfaults during dlopen of onnxruntime_pybind11_state.so on Linux. This removes the global initializer in onnxruntime_pybind_state.cc that eagerly calls Env::Default(), and resolves the platform Env on first use at its two call sites instead.

Motivation and Context

The module has a namespace-scope dynamic initializer:

static Env& platform_env = Env::Default();

Since POSIX telemetry landed (#27379), Env::Default() constructs PosixEnv, whose PosixTelemetry member initializes the 1DS SDK in its constructor. That path reads defaultRuntimeConfig, a namespace-scope static ILogConfiguration defined in the 1DS SDK's RuntimeConfig_Default.hpp, which lives in a different translation unit of the same shared library.

Dynamic initialization order across translation units is unspecified, and the pybind TU's initializer runs first. Variant::merge_map therefore iterates a still zero-initialized std::map: _M_node_count == 0, but _M_header._M_left is nullptr rather than self-pointing, so begin() != end() and the loop dereferences null.

Textbook static initialization order fiasco. Backtrace (Release build relinked without --strip-all to recover symbols):

#0  std::map<..., Variant>::lower_bound        stl_map.h:1307
#1  std::map<..., Variant>::operator[]         stl_map.h:509
#2  Variant::merge_map                         VariantType.hpp:508
#3  RuntimeConfig_Default::RuntimeConfig_Default  RuntimeConfig_Default.hpp:97
#4  LogManagerImpl::LogManagerImpl             LogManagerImpl.cpp:183
#5  LogManagerFactory::Create                  LogManagerFactory.cpp:36
#6  LogManagerFactory::lease
#7  LogManagerFactory::Get                     LogManagerFactory.hpp:71
#8  LogManagerProvider::Get                    LogManagerProvider.cpp:16
#9  onnxruntime::PosixTelemetry::Initialize()
#10 onnxruntime::PosixTelemetry::PosixTelemetry()
#11 onnxruntime::(anonymous namespace)::PosixEnv::PosixEnv()
#12 onnxruntime::Env::Default()
#13 _GLOBAL__sub_I_onnxruntime_pybind_state.cc
#14 call_init                                  elf/dl-init.c:74
...
#21 _dl_open                                   elf/dl-open.c:905

The crash is independent of LD_LIBRARY_PATH and CUDA_VISIBLE_DEVICES — it happens before any ORT runtime code runs. libonnxruntime.so is unaffected because it contains no global initializer that reaches Env::Default(), which is why C/C++ and onnxruntime-genai consumers do not see it.

Both remaining uses of platform_env are inside pybind lambdas that run long after load, so calling Env::Default() there is safe. This also removes the now-stale TODO: we may delay-init this variable and a pre-existing #pragma warning(push) that should have been pop.

Note for follow-up: Env::Default() is now unsafe to call from any dynamic initializer. This was the only such call site in the tree, but hardening PosixTelemetry to defer SDK initialization out of its constructor would remove the hazard entirely.

Tests

Verified on a Linux CUDA 13 Release build (onnxruntime_USE_TELEMETRY enabled):

  • dlopen of onnxruntime_pybind11_state.so succeeds (previously SIGSEGV).
  • import onnxruntime reports the version and ['CUDAExecutionProvider', 'CPUExecutionProvider'].
  • onnxruntime.enable_telemetry_events() / disable_telemetry_events() — the two call sites changed here — work.
  • CPU and CUDA inference sessions produce correct results.
  • Reproduced and verified with LD_LIBRARY_PATH unset and CUDA_VISIBLE_DEVICES empty.

@tianleiwu
tianleiwu enabled auto-merge (squash) July 26, 2026 02:29
@tianleiwu
tianleiwu merged commit 925ada8 into main Jul 27, 2026
88 checks passed
@tianleiwu
tianleiwu deleted the tlwu/20260725/fix_env_crash branch July 27, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants