Increase severity of "request failed: error reading the headers"#25
Closed
benrubson wants to merge 0 commit intoapache:trunkfrom
benrubson:bug60681
Closed
Increase severity of "request failed: error reading the headers"#25benrubson wants to merge 0 commit intoapache:trunkfrom benrubson:bug60681
benrubson wants to merge 0 commit intoapache:trunkfrom
benrubson:bug60681
Conversation
asfgit
pushed a commit
that referenced
this pull request
Jun 25, 2020
When enabling client authentication for proxy (SSLProxyMachineCertificateFile), the client certificate callback function ssl_callback_proxy_cert uses another reference count locking type then one that is used by the caller function when trying to free the private key afterwards by using EVP_PKEY_free. This can lead to a race-condition on pkey->references resulting in a double free error. On my system, the error occurs sporadically when threaded health checking (mod_watchdog) forces two threads competing for the client's private key. For example, see following two backtraces of a coredump where thread 1 and thread 15 both run into CRYPTO_free(). Actually, the private key should never be freed during run-time nor should two threads ever enter CRYPTO_free() concurrently. (gdb) t 1 [Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))] #0 0xf7f3f329 in __kernel_vsyscall () (gdb) bt #0 0xf7f3f329 in __kernel_vsyscall () #1 0xf7cec9e7 in raise () from /lib32/libc.so.6 #2 0xf7cedfb9 in abort () from /lib32/libc.so.6 #3 0xf7d2a14d in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 (gdb) t 15 [Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))] #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 (gdb) bt #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 #1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6 #2 0xf7cd89ab in ?? () from /lib32/libc.so.6 #3 0xf7d2a148 in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 Many thanks to Armin for finding this. Github: closes #129 Submitted by: Armin Abfalterer (arminabf) Reviewed by: ylavic git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1879179 13f79535-47bb-0310-9956-ffa450edef68
asfgit
pushed a commit
that referenced
this pull request
Jun 26, 2020
EVP_PKEY_up_ref(): fix ref count locking type for proxy EVP pkey When enabling client authentication for proxy (SSLProxyMachineCertificateFile), the client certificate callback function ssl_callback_proxy_cert uses another reference count locking type then one that is used by the caller function when trying to free the private key afterwards by using EVP_PKEY_free. This can lead to a race-condition on pkey->references resulting in a double free error. On my system, the error occurs sporadically when threaded health checking (mod_watchdog) forces two threads competing for the client's private key. For example, see following two backtraces of a coredump where thread 1 and thread 15 both run into CRYPTO_free(). Actually, the private key should never be freed during run-time nor should two threads ever enter CRYPTO_free() concurrently. (gdb) t 1 [Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))] #0 0xf7f3f329 in __kernel_vsyscall () (gdb) bt #0 0xf7f3f329 in __kernel_vsyscall () #1 0xf7cec9e7 in raise () from /lib32/libc.so.6 #2 0xf7cedfb9 in abort () from /lib32/libc.so.6 #3 0xf7d2a14d in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 (gdb) t 15 [Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))] #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 (gdb) bt #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 #1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6 #2 0xf7cd89ab in ?? () from /lib32/libc.so.6 #3 0xf7d2a148 in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 Many thanks to Armin for finding this. Github: closes #129 Submitted by: Armin Abfalterer (arminabf) Reviewed by: ylavic Follow up to r1879179: CHANGES entry. Reviewed by: ylavic, jorton, rpluem git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1879224 13f79535-47bb-0310-9956-ffa450edef68
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello,
This PR solves bug 60681.
It's regarding error message
request failed: error reading the headers.Would be nice if it could be
APLOG_ERR(as in Debian package), or at leastAPLOG_NOTICE/APLOG_WARNING.We would then easily have these messages in the error log, without having to set
LogLeveltodebug.Goal is then to ban (with fail2ban) hosts generating too frequently these messages. For example, a slowloris attacker will generate such messages.
Thank you very much 👍
Ben