nginx:1.25.5 fails to build on arm32v5

[yosifkit]

Describe the bug

Building the mainline/debian/Dockerfile (at nginx 1.25.5) causes an error in linking with undefined references. It seems like a missing dependency.

Log:

Also available on the DOI jenkins instance (at least for a little while) https://doi-janky.infosiftr.net/job/wip/job/new/job/build-arm32v5/1693/.

make[1]: *** [Makefile:272: run-test262] Error 1
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: /tmp/cc3YvmNt.ltrans12.ltrans.o: in function `js_atomics_op.lto_priv.0':
/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55162: undefined reference to `__atomic_load_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55148: undefined reference to `__atomic_fetch_xor_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55149: undefined reference to `__atomic_exchange_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55189: undefined reference to `__atomic_compare_exchange_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55144: undefined reference to `__atomic_fetch_add_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55145: undefined reference to `__atomic_fetch_and_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55146: undefined reference to `__atomic_fetch_or_8'
/usr/bin/ld: /tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55147: undefined reference to `__atomic_fetch_sub_8'
/usr/bin/ld: /tmp/cc3YvmNt.ltrans12.ltrans.o: in function `js_atomics_store.lto_priv.0':
/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13/quickjs.c:55253: undefined reference to `__atomic_store_8'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:214: qjsc] Error 1
make[1]: Leaving directory '/tmp/tmp.FYcdktwJfh/nginx-module-njs-1.25.5+0.8.4/debian/extra/quickjs-2024.01.13'
make: *** [debian/rules:48: config.pre.nginx] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Build command 'cd nginx-module-njs-1.25.5+0.8.4 && dpkg-buildpackage -b -uc' failed.

To reproduce

Steps to reproduce the behavior:

(either on a native host or one with qemu user mode emulation)

1. docker build --platform linux/arm/v5 --progress=plain mainline/debian
2. wait ... wait some more ...
3. See error

Expected behavior

A built arm32v5 image at the end.

Your environment

• docker 26.0.0 on Docker Desktop (Windows) and on the DOI builder running Docker 20.10
• nginx:1.25.5
• platform: arm32v5

[oxpa]

Thank you for the report.

I saw the build problem (it failed a build of rootless nginx container) but didn't have a chance to look into this.
Will do my best to fix it next week.

[thresheek]

Thanks for the report @yosifkit - much appreciated. We do indeed not test our builds on armv5...

[thresheek]

Hopefully fixed by 3fb7e2e and https://hg.nginx.org/pkg-oss/rev/93ac6e194ad0, let's wait for actual builds to commence though

[thresheek]

hmm, it seems the fix wasnt enough, at least it's still failing (differently though):

ar -r -c build/libnjs.a \
	build/src/njs_diyfp.o build/src/njs_dtoa.o build/src/njs_dtoa_fixed.o build/src/njs_str.o build/src/njs_strtod.o build/src/njs_murmur_hash.o build/src/njs_djb_hash.o build/src/njs_utf8.o build/src/njs_utf16.o build/src/njs_arr.o build/src/njs_rbtree.o build/src/njs_flathsh.o build/src/njs_trace.o build/src/njs_random.o build/src/njs_malloc.o build/src/njs_mp.o build/src/njs_sprintf.o build/src/njs_utils.o build/src/njs_chb.o build/src/njs_value.o build/src/njs_vm.o build/src/njs_vmcode.o build/src/njs_lexer.o build/src/njs_lexer_keyword.o build/src/njs_parser.o build/src/njs_variable.o build/src/njs_scope.o build/src/njs_generator.o build/src/njs_disassembler.o build/src/njs_module.o build/src/njs_extern.o build/src/njs_boolean.o build/src/njs_number.o build/src/njs_symbol.o build/src/njs_string.o build/src/njs_object.o build/src/njs_object_prop.o build/src/njs_array.o build/src/njs_json.o build/src/njs_function.o build/src/njs_regexp.o build/src/njs_date.o build/src/njs_error.o build/src/njs_math.o build/src/njs_array_buffer.o build/src/njs_typed_array.o build/src/njs_promise.o build/src/njs_encoding.o build/src/njs_iterator.o build/src/njs_async.o build/src/njs_builtin.o build/external/njs_regex.o build/src/njs_buffer.o build/external/njs_crypto_module.o build/external/njs_md5.o build/external/njs_sha1.o build/external/njs_sha2.o build/external/njs_webcrypto_module.o build/external/njs_xml_module.o build/external/njs_zlib_module.o build/external/njs_fs_module.o build/external/njs_query_string_module.o build/build/njs_modules.o 
cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/lib/quickjs/ -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 \
	-pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC \
	external/njs_shell.c \
	build/libnjs.a \
	-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/build-nginx/extra/prebuilt/lib/quickjs/ -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2  -lz  -ledit
lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
/usr/bin/ld: /tmp/ccw77qKL.ltrans11.ltrans.o: undefined reference to symbol '__atomic_fetch_sub_8@@LIBATOMIC_1.0'
/usr/bin/ld: /lib/arm-linux-gnueabi/libatomic.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[1]: *** [build/Makefile:792: build/njs] Error 1
make[1]: Leaving directory '/tmp/tmp.vlBFqkNRsi/nginx-module-njs-1.26.0+0.8.4/debian/extra/njs-0.8.4'
make: *** [debian/rules:48: config.pre.nginx] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
E: Build command 'cd nginx-module-njs-1.26.0+0.8.4 && dpkg-buildpackage -b -uc' failed.

Seems like debian's patch to quickjs also adds -latomic to HOST_LIBS: https://salsa.debian.org/a11y-team/quickjs/-/commit/7b8e2c8aa73f4c3196cd7f340e1779837bf9e32c

[thresheek]

Reproduced locally, it seems like we'd need to provide -latomic for njs linking step as well..

[thresheek]

And of course simply adding -latomic to njs' --ld-opt is not enough, since those options will be added before extra-linked libraries, so it will not be picked up by ld:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# hg diff
diff -r 93ac6e194ad0 debian/Makefile.module-njs
--- a/debian/Makefile.module-njs        Tue Apr 23 18:58:26 2024 +0100
+++ b/debian/Makefile.module-njs        Wed Apr 24 19:04:17 2024 +0000
@@ -32,7 +32,7 @@
        cd $$(CURDIR)/debian/extra/njs-$(NJS_VERSION) \&\& \
        ./configure \
                --cc-opt="$$(CFLAGS) -I$$(BUILDDIR_nginx)/extra/prebuilt/include/quickjs/" \
-               --ld-opt="$$(LDFLAGS) -L$$(BUILDDIR_nginx)/extra/prebuilt/lib/quickjs/" \&\& \
+               --ld-opt="$$(LDFLAGS) -L$$(BUILDDIR_nginx)/extra/prebuilt/lib/quickjs/ -latomic" \&\& \
        make -j$$(NUMJOBS) njs \&\& \
        mv build build-cli
 endef

Resulting in:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# ar -r -c build/libnjs.a \
        build/src/njs_diyfp.o build/src/njs_dtoa.o build/src/njs_dtoa_fixed.o build/src/njs_str.o build/src/njs_strtod.o build/src/njs_murmur_hash.o build/src/njs_djb_hash.o build/src/njs_utf8.o build/src/njs_utf16.o build/src/njs_arr.o build/src/njs_rbtree.o build/src/njs_flathsh.o build/src/njs_trace.o build/src/njs_random.o build/src/njs_malloc.o build/src/njs_mp.o build/src/njs_sprintf.o build/src/njs_utils.o build/src/njs_chb.o build/src/njs_value.o build/src/njs_vm.o build/src/njs_vmcode.o build/src/njs_lexer.o build/src/njs_lexer_keyword.o build/src/njs_parser.o build/src/njs_variable.o build/src/njs_scope.o build/src/njs_generator.o build/src/njs_disassembler.o build/src/njs_module.o build/src/njs_extern.o build/src/njs_boolean.o build/src/njs_number.o build/src/njs_symbol.o build/src/njs_string.o build/src/njs_object.o build/src/njs_object_prop.o build/src/njs_array.o build/src/njs_json.o build/src/njs_function.o build/src/njs_regexp.o build/src/njs_date.o build/src/njs_error.o build/src/njs_math.o build/src/njs_array_buffer.o build/src/njs_typed_array.o build/src/njs_promise.o build/src/njs_encoding.o build/src/njs_iterator.o build/src/njs_async.o build/src/njs_builtin.o build/external/njs_regex.o build/src/njs_buffer.o build/external/njs_crypto_module.o build/external/njs_md5.o build/external/njs_sha1.o build/external/njs_sha2.o build/external/njs_webcrypto_module.o build/external/njs_xml_module.o build/external/njs_zlib_module.o build/external/njs_fs_module.o build/external/njs_query_string_module.o build/build/njs_modules.o
cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 -I/usr/include/libxml2 \
        -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC \
        external/njs_shell.c \
        build/libnjs.a \
        -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2 -lz -ledit


lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
/usr/bin/ld: /tmp/ccoES5lX.ltrans11.ltrans.o: undefined reference to symbol '__atomic_fetch_sub_8@@LIBATOMIC_1.0'
/usr/bin/ld: /usr/lib/gcc/arm-linux-gnueabi/12/libatomic.so: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

And if we move -latomic to the end of libs linked, it's all good:

root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# cc -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -o build/njs -Isrc -Ibuild -I/usr/include/libxml2 -I/usr/include/libxml2      -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -fexcess-precision=standard -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC -I/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/include/quickjs/ -g -O2 -ffile-prefix-map=/pkg-oss/debian/debuild-module-njs/nginx-1.25.5=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC  external/njs_shell.c    build/libnjs.a  -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -L/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/build-nginx/extra/prebuilt/lib/quickjs/ -latomic -lm   -lpcre2-8 -lquickjs.lto -lm -ldl -lpthread -lcrypto -lxml2 -lz -ledit -latomic
lto-wrapper: warning: using serial compilation of 14 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
root@3afdea09625d:/pkg-oss/debian/debuild-module-njs/nginx-1.25.5/debian/extra/njs-0.8.4# 

So that means we'd need a way to add it to EXTRA_LIBS in njs, which probably needs some extra patching.

[thresheek]

I just spoke to njs maintainer and I think we're going to add a configure check in njs to enable -latomic when linking on armv7l...

[thresheek]

So we discussed the approach on fixing it in upstream njs through configure checks and it felt a bit hacky to do so - especially keeping in mind that armel can be "multiarch" compiled on e.g. armv8 so multiple checks will be needed to fully cover the problem.

Long story short I've pushed a hack to our packaging which is a somewhat appropriate place to put hacks to :) https://hg.nginx.org/pkg-oss/rev/7bc1410f615c

I'll follow up with updating this repo and doing docker-library PR in a next few days.

[yosifkit]

Success! https://hub.docker.com/_/nginx/tags?page=&page_size=&ordering=&name=1.25.5