shared:ERROR: unexpected error while trying to eval ctors Fatal: ...stopping since could not flatten memory

[woodser]

Hi. I am receiving this error when building a WASM project with the upstream emscripten backend and with pthreads enabled:

shared:ERROR: unexpected error while trying to eval ctors:
Fatal:   ...stopping since could not flatten memory

I first build boost libs system, thread, chrono, serialization, filesystem, and regex with -s USE_PTHREADS=1 -s ALLOW_MEMORY_GROWTH=0 passed to emcc and -pthread passed to cxxflags.

Then I build my source project which links to the boost libs as static libraries with -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=10 -s PROXY_TO_PTHREAD=1 -s ALLOW_MEMORY_GROWTH=0 passed to emcc and -pthread passed to cxxflags.

In what appears to be one of the final steps of linking, I get this error if and only if pthreads are enabled in boost and my project source. If I don't use pthreads, both projects build and link successfully, but then I cannot use emscripten's Fetch API synchronously, which is my end goal.

I cannot find any search results on this error except the original source code in wasm-ctor-eval.cpp so it not a commonly encountered problem.

I am not using the ported boost in emscripten-ports. Perhaps I need to?

The project will ultimately be run in node. Perhaps PR #9745 is relevant?

Any advice? Thanks!

[tlively]

I haven’t seen a case where the memory couldn’t be flattened before! Do you have a reproduced you can share? I’d be happy to take a look at this.

[woodser]

Please bear with me because this project has nested git submodules and the issue is recreated on non-master branches.

The issue should be reproducible by:

1. git clone --recurse-submodules https://github.com/monero-ecosystem/monero-javascript.git
2. cd monero-javascript
3. git checkout wasm41_fetch
4. ./bin/update_submodules
5. cd ./external/monero-cpp-library/ && git checkout wasm39_monero_wallet_base
6. cd ./external/monero-core/ && git checkout wasm36_generate_genesis
7. Copy boost assets to monero-javascript/external/monero-cpp-library/external/boost-sdk
8. export EMSCRIPTEN=<emscripten_root>/emsdk/upstream/emscripten
9. cd ../../../../ <- the monero-javascript project root
10. ./bin/build-boost-emscripten.sh <- builds static boost libs to ./build/boost/libs
11. ./bin/build-emcpp.sh <- creates the error

Parameters are passed to emcc in step (10) within monero-javascript/configs/emscripten.jam.
Parameters are passed to emcc in step (11) within monero-javascript/CMakeLists.txt.

I am trying to run Monero's core wallet in a browser so it is fully client-side and trustless. I have been successful by using EM_JS and the JavaScript Request library to make XHR requests. However, 1) large requests cause a "RangeError: source is too big" on NodeJS, and 2) I expect emscripten's fetch API to be more performant than copying response buffers from JavaScript to the heap, which is significant because wallets fetch GBs of data.

Thank you for your help. Please let me know if I can assist.

[kripken]

As a workaround I think you can disable eval-ctors (-s EVAL_CTORS=0). It should be already disabled by default with the upstream backend.

Maybe the memory can't be flattened because it's relocatable (shared library, side module, etc.)?

[VirtualTim]

I'm also seeing the same issue. My build is fairly complex, so I haven't tried to produce a minimal example.
Here's my linker flags:

-O3 --llvm-opts 3 --llvm-lto 3 -s WASM=1 -s WASM_OBJECT_FILES=1
-s STRICT=1 -s ALIASING_FUNCTION_POINTERS=1 -s ASSERTIONS=0 -s DISABLE_EXCEPTION_CATCHING=0 -s DEMANGLE_SUPPORT=0 -s EVAL_CTORS=1
-s USE_ZLIB=1 -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=12 -s NO_EXIT_RUNTIME=1 -s NO_DYNAMIC_EXECUTION=1
-s TOTAL_MEMORY=640MB --memory-init-file 1 -s DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR=1 -s ENVIRONMENT='web','worker'
-s MODULARIZE=1 -s EXPORT_ES6=0 -s EXPORT_NAME='module_name' -s TEXTDECODER=2 -s ERROR_ON_UNDEFINED_SYMBOLS=0  
-s FETCH=1 -s FETCH_SUPPORT_INDEXEDDB=0 -s USE_FETCH_WORKER=0 
-s "EXPORTED_FUNCTIONS=['_calloc', '_pthread_mutexattr_init', '_pthread_mutex_init', '_emscripten_futex_wake', '_sbrk']"
-o module_name.js --post-js ../../some_files.js

and the error I get:

shared:ERROR: unexpected error while trying to eval ctors:

Fatal:   ...stopping since could not flatten memory

shared:ERROR: '/usr/bin/python /emsdk_portable/upstream/emscripten/tools/ctor_evaller.py /tmp/emscripten_temp_9X58qS/module_name.wasm.o.js.pp.js module_name.wasm 671088640 5242880 1024 /emsdk_portable/upstream/bin 0 --mvp-features --enable-threads --enable-threads --enable-bulk-memory --enable-mutable-globals --enable-sign-ext' failed (1)

I'm only seeing this after switching to the new llvm backend.

[kripken]

If you can provide a non-minimal example of the wasm file + finalize command, that would be good enough (the binaryen reducer can reduce it for us).

[tlively]

@woodser can you elaborate on step 7? What boost assets am I supposed to be copying?

Edit: Actually, just uploading the wasm file as @kripken suggested would be much better.

[kripken]

I double-checked and we do disable EVAL_CTORS in the wasm backend. But if you manually pass the flag, it will run, which may be bad as we see here. I'll open a PR to disable it entirely.

[woodser]

@tlively Step 7 refers to the boost library downloadable here: https://www.boost.org/users/download/

Here is a zip with the built wasm files: wasms.zip

"with_ctor_eval.wasm" was built without passing the EVAL_CTORS parameter and produces the flatten memory error described in this issue.

"without_ctor_eval.wasm" was built passing -s EVAL_CTORS=0. It compiles and links but I get these errors in browsers:

• Firefox: "wasm validation error: at offset 1300999: memory.init requires a DataCount section"
• Chrome: "WebAssembly.instantiate(): Compiling function [C-API] Add type system C-API #4790 failed: invalid data segment index: 3 @+1301001)"

"non_optimized.wasm" was built by removing -Oz as a parameter. It runs in the browser until an error, "pthread sent an error! undefined:undefined: undefined", which feels like progress. That's with -s PROXY_TO_PTHREAD=1.

The full list of arguments building boost is here, and the full list of arguments building the wasm file starts here.

[woodser]

@kripken Is this what is meant by finalize command? This is output when building the wasm.

-- EMCC_LINKER_FLAGS__WASM -Wall -std=c++11 --bind -s MODULARIZE=1 -s 'EXPORT_NAME="monero_cpp_library"' --llvm-lto 1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s ASSERTIONS=2 -s EXIT_RUNTIME=0 -s ASYNCIFY=1 -s 'ASYNCIFY_IMPORTS=["js_send_json_request","js_send_binary_request","emscripten_sleep"]' -s USE_PTHREADS=1 -s PTHREAD_POOL_SIZE=10 -s PROXY_TO_PTHREAD=1 -s PRECISE_F32=1 -s USE_BOOST_HEADERS=1 -s DISABLE_EXCEPTION_CATCHING=0 -s EXCEPTION_DEBUG=1 -s DEMANGLE_SUPPORT=1 -s NO_DYNAMIC_EXECUTION=1 -s NODEJS_CATCH_EXIT=0 -s EXTRA_EXPORTED_RUNTIME_METHODS='["UTF8ToString","stringToUTF8","lengthBytesUTF8","intArrayToString"]' -s WASM=1 -s ALLOW_MEMORY_GROWTH=0 --post-js /.../monero-javascript/src/module-post.js

[kripken]

@woodser I think we don't need a testcase given what I said earlier - I think the problem here is caused by explicitly passing in EVAL_CTORS with the wasm backend, where it is currently broken. #9790 will disable it even if the flag is passed in.

If you see that error despite not passing in the flag, that's very odd. EMCC_DEBUG=1 output can verify what commandline args are being passed in.