Fixed IE10's DNT setting being ignored #2

OscarGodson · 2012-09-09T23:20:56Z

No description provided.

mxriverlynn · 2012-09-09T23:25:14Z

+1

the arrogance and absurdity of Apache ignoring IE10 users is truly mind-boggling

zacparker · 2012-09-09T23:29:42Z

+1

liquidboy · 2012-09-09T23:34:15Z

+1

hizanberg · 2012-09-09T23:44:21Z

-1

Expecting a lot of small minded devs to vote +1 here, instead of petitioning a change to IE10's policy to abide by the intent of the standards.

OscarGodson · 2012-09-09T23:50:16Z

@hizanberg Actually we all like the way IE10 is handling it.

interscaperob-duplicate · 2012-09-09T23:53:19Z

+1

Expecting many self-righteous devs to vote -1 here, because they don't like the way the standard is being interpreted by browser vendors, and think injecting code in a message execution engine is the right way to fix the problem.

There is no possible way a company, for example, would never want to override the will of the individual users and not allow their employees to be tracked for advertising purposes. And the people that aren't idiots, that actually took the time to read whet the "Express" settings said and still chose to click it anyways... we should just throw their choice out the window, right?

hizanberg · 2012-09-09T23:55:42Z

@OscarGodson Who cares what you think it should be? You have no more right to set the defaults preference of all IE10 users anymore than Microsoft does.

OscarGodson · 2012-09-09T23:58:26Z

@hizanberg You realize thats why we're all upset right? Because Apache is making the choice for IE10 users...? Sort of an ironic comment you just made. Also, IE10 isn't setting a default. It's just a preselected value. The user ultimately has to make a choice to accept the preselected value or not. So, in this case, only Apache is trying to set a default for everyone on IE10. Not me, us, or Microsoft.

interscaperob-duplicate · 2012-09-10T00:00:12Z

@hizanberg Your statement assumes that all IE10 users are morons, and that no one will read the Express settings, and that people just click next until prompted otherwise. Some will, but not all, and to throw away the choice for the people that were OK with the default and chose it, just because some people click "next" without understanding the consequences, is foolish. It's like throwing out the results of a Presidential Election because some people didn't vote the way you wanted them to.

hizanberg · 2012-09-10T00:05:20Z

@OscarGodson yes you're ironically upset at a change of a default server setting in Apache (that doesn't do anything yet) to render's IE10's DNT meaningless since most of the times it wont have been a conscience user preference.

@interscaperob I'm assuming defaults will always get selected regardless of what they are, hence the user populace isn't making the choice the way it is. You're delusional if you think defaults aren't important or aren't a way for vendors to abuse for their own anti-competitive purposes. If you don't care what the defaults should be, you should be petitioning IE to change their policy so you are able to opt-in to DNT and it will actually get respected.

interscaperob-duplicate · 2012-09-10T00:09:24Z

@hizanberg "render's IE10's DNT meaningless since most of the times it wont have been a conscience user preference."

This is based on an assumption of user behavior and not any actual empirical data. if there was no text on the setup page, that would be one thing. It is there, it is clear, and it satisfies the requirements. And the part that you fail to understand is that setting it the other way IS STILL A DEFAULT. It's just one that advertisers just happen to like.

hizanberg · 2012-09-10T00:13:28Z

@interscaperob An explicit choice is never made when its bundled as part of the Default settings.
For various reasons (i.e. being the easiest option) regardless of what the default options are, most users will go with the defaults, at which point the browser vendor is making the decision.

It doesn't even matter what text is on the setup page, since most users wont even read it. It needs to be on a stand-alone screen (or dialog) with only that question only and no default option pre-selected for it to be a conscience user decision. Otherwise going to user preferences and checking a checkbox works too, since at that point most users will be deciding for themselves then.

interscaperob-duplicate · 2012-09-10T01:08:55Z

You understand that if the original value is DNT:0, that is still a default, right? It's a bit, and it is not nullable. It is either on or off. If it ships off... IT IS STILL A DEFAULT.

Correction: The header can not be sent. No header still equals off, which still equals "track me". Having the set value is still the result of an explicit user interaction. The spec does not say it has to be the only option on the page. It does however say that the header should not be interfered with by third parties.

covener · 2012-09-10T01:24:34Z

When there's no preference expressed, there's no header. That's the third state.

imanavg · 2012-09-10T05:56:06Z

@hizanberg - Please don't spread lies. It is proven over and over again that IE is actually compliant, since it is giving the option to user to make a choice i.e. whether choose express settings where it shows that DNT would be enabled or select customize to modify the settings. The user IS making the CHOICE.

So please don't spread FUD or try to create a political war due to your personal bias on the issue.

PS: Your point about user not reading the text is as stupid as it can get. The standard asks for an option to be provided and it is provided. It also lists that an implementation may bundle DNT as a part of other option such as privacy high and MS is doing that, so I fail to understand your point. Please keep your personal belief out of OSS, it would serve you well.

The text from the draft:
"For example, a user might select a check-box in their user agent's configuration, install a plug-in or extension that is specifically designed to add a tracking preference expression, or make a choice for privacy that then implicitly includes a tracking preference (e.g., Privacy settings: high). Likewise, a user might install or configure a proxy to add the expression to their own outgoing requests. For each of these cases, we say that a tracking preference is enabled."

OscarGodson · 2012-09-10T08:01:59Z

@mythz

There is no proof that IE is compliant (and no you saying so doesn't make it so).

Is this enough proof they're making it an option that is clearly marked? Not only that, you're free to change it at this time as well.

the fact that the person who helped form the standards, is saying that they're not compliant should at least ring some warning bells

We don't care if they are or are not. That's not the issue. Although we all agree their definition of compliance agrees with Microsofts implementation, we have more of an issue that they're overriding user's choice in IE10 as a form of "punishment" to Microsoft. That's not how open source and open standards works and it's wrong. Open standards are not laws, that's the entire point of them. They're supposed to be followed and grown organically, not forcefully.

sschocke · 2012-09-10T08:11:32Z

+1

As many have said, the web server is not the place for this. Also, the fact that it is basically hidden away as a default in Apache's configuration means those same idiots that don't read the "Express Settings" page will also use Apache with it's defaults on, and that damages the standard.

I have no problem with the idea behind this... Write a blog post or something about Microsoft's evil choice in your opinion... include in it a sample of how you can force DNT=0 for IE10 User Agents, and advertise it to everyone that will listen... Make forcing DNT=0 an EXPLICIT sysadmins choice!

But then again, @royfielding, @hizanberg, etc are all too scared no sysadmin will actually do it, and user privacy may actually get respected... or at least, that's how it seems from my point of view.

EvilHom3r · 2012-09-10T11:28:37Z

+1

gabranth · 2012-09-10T11:36:19Z

@OscarGodson isn't that image showing that Microsoft is complying with the DNT draft as its not set its only turned on when the end users chooses with the express setting that clearly stats that it will be enabled or turned off with customized settings

BrazilianJoe · 2012-09-10T14:01:30Z

+1.

I can see it has been shown that MS complies to the standard and provides the necessary choice for the user. It also presents it in a way that it is easier for the user to have less invasive settings, which is largely seen as a good thing by the consumers themselves.

As a user, sysadmin and web developer, working with Windows, Mac and Linux, I can't see how this patch saves any trouble from any part involved, much on the contrary. It needlessly meddles with the information sent by the browser. The amount of comments already shows that this was a bad move/commit, and politically motivated.

If MS had it set off by default, there would be screaming about how MS does not protect the users by 'hiding DNT in obscure custom settings'. Damned if you do, damned if you don't. All this Microsoft-bashing in open source is getting old already.

londospark · 2012-09-10T16:13:53Z

IE 10 is compliant with the spec and should not be treated in any other manner. MS has made it clear that DNT will be switched on and has made it clear that one may opt out and how. I hope this change is committed.

Fever905 · 2012-09-10T18:58:18Z

+1
Grow a freaking brain wouldja? And this was written by someone who works for Adobe whose clients and partners BENEFIT from the Do Not Track being disabled? Huge conflict of interest here. The choice is made in the express setup. Its clearly marked there.

mamund · 2012-09-10T19:02:56Z

Here is the key sentence in the draft:

"The basic principle is that a tracking preference expression is only transmitted when it reflects a deliberate choice by the user."[1]

Here are some possible ways to model this choice:

Tracking is allowed. Do you want to change this? yes( ) no ( ) (optional)
Tracking is now allowed. Do you want to change this? yes( ) no ( ) (optional)
Do you want to allow tracking? yes ( ) no ( ) (required)

Some here assert that granting a user the ability to skip taking an action ("We will set option Z to value X unless you tell us otherwise") meets the definition of "deliberate choice by the user." @royfielding (and others) do not see it the same way.

Until this point is cleared up, the work is stalled. My suggestion is to modify the document to define "deliberate choice by the user" in a way that removes the current ambiguity and still allows vendors freedom to craft their UX as they wish while knowing they are within the proposed standard.

[1] http://www.w3.org/2011/tracking-protection/drafts/tracking-dnt.html#determining

imanavg · 2012-09-11T07:22:49Z

@mamund - You are focusing only on partial text. Here is the detailed text at the same link:

"We do not specify how tracking preference choices are offered to the user or how the preference is enabled: each implementation is responsible for determining the user experience by which a tracking preference is enabled. For example, a user might select a check-box in their user agent's configuration, install an extension or add-on that is specifically designed to add a tracking preference expression, or make a choice for privacy that then implicitly includes a tracking preference (e.g., Privacy settings: high). The user-agent might ask the user for their preference during startup, perhaps on first use or after an update adds the tracking protection feature. Likewise, a user might install or configure a proxy to add the expression to their own outgoing requests."

kristiaanvandeneynde · 2012-09-11T09:40:45Z

@imanavg You just quoted why you're wrong. They key words are "The user-agent", not "The operating system". Unless the screenshot that @OscarGodson showed above is shown in Internet Explorer 10 on first use, Microsoft is ignoring the spec.

You cannot know that the person who installed the OS on a computer will also be the actual user of that pc.
Therefore, the DNT settings set during OS install cannot ever be trusted to be the actual user's preference.

That being said, the code that was added to Apache to block out IE10 also disrespects the DNT standard.

In short:
Yes, IE10 should be ignored when DNT is set by the OS.
No, IE10 shouldn't be ignored as a whole. Find a better way to solve this.

unikitty37 · 2012-09-11T10:43:26Z

+1.

Apache has no way of knowing whether or not that DNT: 1 is set because somebody handed me a pre-installed system where they just clicked through the options, or whether I went into the settings and verified that Do Not Track was set as I expected.

Just as Google had no right to override my "no third-party cookies" setting on Safari just because it happened to be the default, Apache has no right to override my "do not track" setting on Internet Explorer just because it happened to be the default.

imanavg · 2012-09-11T15:12:18Z

@kristiaanvandeneynde - What are you talking about, how many users do you think have more than one user in case of PC? Yes for may be a few scenarios it would result in an administrator selected default, but that does not mean that 90% of users who have single account should be penalized especially since they made the choice to have DNT on.

This is highly irresponsible on the part of Apache foundation. Shame on them for abusing OSS for their personal agenda.

kristiaanvandeneynde · 2012-09-11T15:14:45Z

@imanavg You'd be surprised how many people buy preconfigured computers or have their IT-savvy son/daughter/brother/dog configure it for them because they don't (want to) know the first thing about computers.

And do you have any factual data to back up your claim that 90% of the population configures its own pc? I think your numbers are way off...

ghost · 2012-09-11T18:03:53Z

-1, until MS revert their code, this PR should not be merged.

ghost · 2012-09-12T13:26:10Z

Seems you are all quite ignorant about how Roy arrived at the patch! It was discussed and agreed on by the powers that be in he Apache project. It was not a one-man-decision.

imanavg · 2012-09-12T14:23:00Z

@Drak - it was Roy's decision, a stupid one at best, not a cumulative one. The title of his commit proves his political stand. His patch and his politics has no place in OSS. And you keep your MS hatred to yourself, OSS is a better place without people filled with hatred similar to Islamic Jihadist.

ghost · 2012-09-12T14:31:00Z

@imanavg it is you who is behaving like a extremist, and you who is transferring your hatred and whatever else onto those who disagree with you. I never said I hate MS, I in fact use it over linux for my desktop. However, their browser leaves a lot to be desired and in a free world, I am able to says so. In your world, evidently those who disagree with you are terrorists who deserve the sharp end of an Apache helicopter attack? I support Apache's decision, and I would ask you to respect that fact without slinging mud.

imanavg · 2012-09-12T14:40:04Z

@Drak - All your are doing is coming here and sharing your opinion without any knowledge of the subject or any logical points as to why you think the original patch was the right choice.

On the other hand, we have proven over and over again that IE is actually compliant, since it is giving the option to user to make a choice i.e. whether choose express settings where it shows that DNT would be enabled or select customize to modify the settings. The user IS making the CHOICE.

What Apache is doing is ignoring the choice I as a user would make by selecting Express Setup option and you support that? Ask yourself, would you have supported the same commit, if it was against Chrome or FF? Sorry, your comment history here shows your bias, no matter how objective you pretend to be.

The text from the draft:
"For example, a user might select a check-box in their user agent's configuration, install a plug-in or extension that is specifically designed to add a tracking preference expression, or make a choice for privacy that then implicitly includes a tracking preference (e.g., Privacy settings: high). Likewise, a user might install or configure a proxy to add the expression to their own outgoing requests. For each of these cases, we say that a tracking preference is enabled."

ziyan · 2012-09-14T19:01:48Z

+1

kristiaanvandeneynde · 2012-09-18T09:31:30Z

@imanavg You haven't proven jack. Please stop patronizing people for expressing their opinion. Ad hominem attacks are the tool of those that have no real case to defend.

Because the IE10 DNT setting is set during Windows 8 install, you can never truly know whether a user intended to switch on DNT or not. Ergo Apache is making the right move here, albeit hypocritical. It's about sending a message.

If Microsoft's move is left unchecked, then what stops advertisers from simply not respecting DNT at all? After all, they could use IE10's settings to prove their right, saying: "We're willing to cooperate, but Microsoft forced our hand."

imanavg · 2012-09-23T12:21:17Z

@kristiaanvandeneynde Yes right, go read the damn spec and then argue. If I install windows 8 and I click express setting, why is Apache ignoring it? This proves that it is Apache that is in violation not respecting my settings than anything else.

Stop being a hater in life and you would feel much better.

kristiaanvandeneynde · 2012-09-23T15:53:25Z

@imanavg There's a reason your posts got deleted from a381ff3.

You're so full of yourself that you just insult anyone who disagrees with you.
There is a term for that: nerd rage. Please crawl back under the bridge from whence you came, troll.

I choose to no longer waste my time on you.
Blocking you and unfollowing this thread based on 95b9a0f.
Also reporting your comment in which you compare anyone who disagrees with you with jihadists.

MPMs event and worker both need a dedicated pool to handle the creation of the threads (listener, workers) and synchronization objects (queues, pollset, mutexes...) in the start_threads() thread, with at least the lifetime of the connections they handle, and thus survive pchild destruction (notably in ONE_PROCCESS mode, but SIG_UNGRACEFUL is concerned too). For instance, without this fix, the below backtrace can happen in ONE_PROCCESS mode and a signal/^C is received (with active connections): Thread 1 "httpd" received signal SIGSEGV, Segmentation fault. (gdb) bt #0 <BOOM> #1 0x00007ffff7c7e016 in apr_file_write (thefile=0x0, ...) ^ NULL (cleared) at file_io/unix/readwrite.c:230 #2 0x00007ffff7c7e4a7 in apr_file_putc (ch=1 '\001', thefile=0x0) ^ NULL (cleared) at file_io/unix/readwrite.c:377 #3 0x00007ffff7c8da4a in apr_pollset_wakeup (pollset=0x55555568b870) ^ already destroyed by pchild at poll/unix/pollset.c:224 #4 0x00007ffff7fc16c7 in decrement_connection_count (cs_=0x7fff08000ea0) at event.c:811 #5 0x00007ffff7c83e15 in run_cleanups (cref=0x7fffe4002b78) at memory/unix/apr_pools.c:2672 #6 0x00007ffff7c82c2f in apr_pool_destroy (pool=0x7fffe4002b58) ^ master_conn at memory/unix/apr_pools.c:1007 #7 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x7fff08000c28) ^ ptrans at memory/unix/apr_pools.c:1004 #8 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x555555638698) ^ pconf at memory/unix/apr_pools.c:1004 #9 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x555555636688) ^ pglobal at memory/unix/apr_pools.c:1004 #10 0x00005555555f4709 in ap_terminate () at unixd.c:522 #11 0x00007ffff6dbc8f1 in __run_exit_handlers (...) at exit.c:108 #12 0x00007ffff6dbc9ea in __GI_exit (status=<optimized out>) at exit.c:139 #13 0x00007ffff7fc1616 in clean_child_exit (code=0) at event.c:774 ^ pchild already destroyed here #14 0x00007ffff7fc5ae4 in child_main (child_num_arg=0, child_bucket=0) at event.c:2869 ... While at it, add comments about the lifetimes of MPMs pools and their objects, and give each pool a tag (e.g. "pchild" accordingly to other MPMs). git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1835845 13f79535-47bb-0310-9956-ffa450edef68

MPMs event and worker both need a dedicated pool to handle the creation of the threads (listener, workers) and synchronization objects (queues, pollset, mutexes...) in the start_threads() thread, with at least the lifetime of the connections they handle, and thus survive pchild destruction (notably in ONE_PROCCESS mode, but SIG_UNGRACEFUL is concerned too). For instance, without this fix, the below backtrace can happen in ONE_PROCCESS mode and a signal/^C is received (with active connections): Thread 1 "httpd" received signal SIGSEGV, Segmentation fault. (gdb) bt #0 <BOOM> #1 0x00007ffff7c7e016 in apr_file_write (thefile=0x0, ...) ^ NULL (cleared) at file_io/unix/readwrite.c:230 #2 0x00007ffff7c7e4a7 in apr_file_putc (ch=1 '\001', thefile=0x0) ^ NULL (cleared) at file_io/unix/readwrite.c:377 #3 0x00007ffff7c8da4a in apr_pollset_wakeup (pollset=0x55555568b870) ^ already destroyed by pchild at poll/unix/pollset.c:224 #4 0x00007ffff7fc16c7 in decrement_connection_count (cs_=0x7fff08000ea0) at event.c:811 #5 0x00007ffff7c83e15 in run_cleanups (cref=0x7fffe4002b78) at memory/unix/apr_pools.c:2672 #6 0x00007ffff7c82c2f in apr_pool_destroy (pool=0x7fffe4002b58) ^ master_conn at memory/unix/apr_pools.c:1007 #7 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x7fff08000c28) ^ ptrans at memory/unix/apr_pools.c:1004 #8 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x555555638698) ^ pconf at memory/unix/apr_pools.c:1004 #9 0x00007ffff7c82c12 in apr_pool_destroy (pool=0x555555636688) ^ pglobal at memory/unix/apr_pools.c:1004 #10 0x00005555555f4709 in ap_terminate () at unixd.c:522 #11 0x00007ffff6dbc8f1 in __run_exit_handlers (...) at exit.c:108 #12 0x00007ffff6dbc9ea in __GI_exit (status=<optimized out>) at exit.c:139 #13 0x00007ffff7fc1616 in clean_child_exit (code=0) at event.c:774 ^ pchild already destroyed here #14 0x00007ffff7fc5ae4 in child_main (child_num_arg=0, child_bucket=0) at event.c:2869 ... While at it, add comments about the lifetimes of MPMs pools and their objects, and give each pool a tag (e.g. "pchild" accordingly to other MPMs). (follow up for event_pollset in r1835846). git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1835845 13f79535-47bb-0310-9956-ffa450edef68

When enabling client authentication for proxy (SSLProxyMachineCertificateFile), the client certificate callback function ssl_callback_proxy_cert uses another reference count locking type then one that is used by the caller function when trying to free the private key afterwards by using EVP_PKEY_free. This can lead to a race-condition on pkey->references resulting in a double free error. On my system, the error occurs sporadically when threaded health checking (mod_watchdog) forces two threads competing for the client's private key. For example, see following two backtraces of a coredump where thread 1 and thread 15 both run into CRYPTO_free(). Actually, the private key should never be freed during run-time nor should two threads ever enter CRYPTO_free() concurrently. (gdb) t 1 [Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))] #0 0xf7f3f329 in __kernel_vsyscall () (gdb) bt #0 0xf7f3f329 in __kernel_vsyscall () #1 0xf7cec9e7 in raise () from /lib32/libc.so.6 #2 0xf7cedfb9 in abort () from /lib32/libc.so.6 #3 0xf7d2a14d in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 (gdb) t 15 [Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))] #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 (gdb) bt #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 #1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6 #2 0xf7cd89ab in ?? () from /lib32/libc.so.6 #3 0xf7d2a148 in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 Many thanks to Armin for finding this. Github: closes #129 Submitted by: Armin Abfalterer (arminabf) Reviewed by: ylavic git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1879179 13f79535-47bb-0310-9956-ffa450edef68

EVP_PKEY_up_ref(): fix ref count locking type for proxy EVP pkey When enabling client authentication for proxy (SSLProxyMachineCertificateFile), the client certificate callback function ssl_callback_proxy_cert uses another reference count locking type then one that is used by the caller function when trying to free the private key afterwards by using EVP_PKEY_free. This can lead to a race-condition on pkey->references resulting in a double free error. On my system, the error occurs sporadically when threaded health checking (mod_watchdog) forces two threads competing for the client's private key. For example, see following two backtraces of a coredump where thread 1 and thread 15 both run into CRYPTO_free(). Actually, the private key should never be freed during run-time nor should two threads ever enter CRYPTO_free() concurrently. (gdb) t 1 [Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))] #0 0xf7f3f329 in __kernel_vsyscall () (gdb) bt #0 0xf7f3f329 in __kernel_vsyscall () #1 0xf7cec9e7 in raise () from /lib32/libc.so.6 #2 0xf7cedfb9 in abort () from /lib32/libc.so.6 #3 0xf7d2a14d in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 (gdb) t 15 [Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))] #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 (gdb) bt #0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6 #1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6 #2 0xf7cd89ab in ?? () from /lib32/libc.so.6 #3 0xf7d2a148 in ?? () from /lib32/libc.so.6 #4 0xf7d2fd27 in ?? () from /lib32/libc.so.6 #5 0xf7d3047d in ?? () from /lib32/libc.so.6 #6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434 #7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406 #8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475 #9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426 #10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832 #12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231 #13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008 #14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245 #15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760 #16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590 #17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728 #18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590 #19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664 #20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806 #21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870 #22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266 #23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142 #24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0 #25 0xf7da862e in clone () from /lib32/libc.so.6 Many thanks to Armin for finding this. Github: closes #129 Submitted by: Armin Abfalterer (arminabf) Reviewed by: ylavic Follow up to r1879179: CHANGES entry. Reviewed by: ylavic, jorton, rpluem git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1879224 13f79535-47bb-0310-9956-ffa450edef68

…illed. There shouldn't be any worker thread active when pchild is destroyed (thus each thread's pool), so register workers_pool_cleanup as a pre_cleanup of pchild. This is to avoid races like the below stacktrace, where slot_run() threads are still running when clean_child_exit() is called. Thread 23 (Thread 0x7f4865b79800 (LWP 3740)): #0 0x00007f4864dec449 in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f4865020117 in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2629 #2 pool_clear_debug (pool=pool@entry=0x558a5297e4a0, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1830 #3 0x00007f486501ffee in pool_destroy_debug (pool=0x558a5297e4a0, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #4 0x00007f48650200f0 in pool_clear_debug (pool=pool@entry=0x558a52a41070, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1827 #5 0x00007f486501ffee in pool_destroy_debug (pool=0x558a52a41070, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #6 0x00007f486502085c in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1957 #7 0x0000558a52326cfc in clean_child_exit (code=0) at event.c:757 #8 0x0000558a52327969 in child_main (child_num_arg=child_num_arg@entry=1, child_bucket=child_bucket@entry=0) at event.c:2926 #9 0x0000558a52327ce5 in make_child (s=0x558a52c9f840, slot=slot@entry=1, bucket=0) at event.c:2992 #10 0x0000558a52327d4c in startup_children (number_to_start=2, number_to_start@entry=3) at event.c:3015 #11 0x0000558a523289ac in event_run (_pconf=<optimized out>, plog=0x558a5273ce00, s=0x558a52c9f840) at event.c:3374 #12 0x0000558a5233e91e in ap_run_mpm (pconf=0x558a5270cbe0, plog=0x558a5273ce00, s=0x558a52c9f840) at mpm_common.c:100 #13 0x0000558a5231b763 in main (argc=<optimized out>, argv=<optimized out>) at main.c:844 Thread 2 (Thread 0x7f4840b70700 (LWP 3836)): #0 0x00007f4864dec9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f486501f65d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68 #2 0x00007f484e14ae4a in get_next (slot=0x558a528d5fe0) at h2_workers.c:209 #3 slot_run (thread=0x558a52828b30, wctx=0x558a528d5fe0) at h2_workers.c:228 #4 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 1 (Thread 0x7f4841b72700 (LWP 3834)): #0 0x00007f4864a2ce97 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f4864a2e801 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f4865020865 in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1955 #3 0x00007f486502b536 in apr_thread_exit (thd=thd@entry=0x558a52ba8980, retval=retval@entry=0) at threadproc/unix/thread.c:206 #4 0x00007f484e14aec6 in slot_run (thread=0x558a52ba8980, wctx=0x558a528d6060) at h2_workers.c:248 #5 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 While at it, rename server_pool as pchild in h2_workers_create(), to make it clear which pool it is. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883675 13f79535-47bb-0310-9956-ffa450edef68

…illed. There shouldn't be any worker thread active when pchild is destroyed (thus each thread's pool), so register workers_pool_cleanup as a pre_cleanup of pchild. This is to avoid races like the below stacktrace, where slot_run() threads are still running when clean_child_exit() is called. Thread 23 (Thread 0x7f4865b79800 (LWP 3740)): #0 0x00007f4864dec449 in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f4865020117 in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2629 #2 pool_clear_debug (pool=pool@entry=0x558a5297e4a0, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1830 #3 0x00007f486501ffee in pool_destroy_debug (pool=0x558a5297e4a0, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #4 0x00007f48650200f0 in pool_clear_debug (pool=pool@entry=0x558a52a41070, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1827 #5 0x00007f486501ffee in pool_destroy_debug (pool=0x558a52a41070, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #6 0x00007f486502085c in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1957 #7 0x0000558a52326cfc in clean_child_exit (code=0) at event.c:757 #8 0x0000558a52327969 in child_main (child_num_arg=child_num_arg@entry=1, child_bucket=child_bucket@entry=0) at event.c:2926 #9 0x0000558a52327ce5 in make_child (s=0x558a52c9f840, slot=slot@entry=1, bucket=0) at event.c:2992 #10 0x0000558a52327d4c in startup_children (number_to_start=2, number_to_start@entry=3) at event.c:3015 #11 0x0000558a523289ac in event_run (_pconf=<optimized out>, plog=0x558a5273ce00, s=0x558a52c9f840) at event.c:3374 #12 0x0000558a5233e91e in ap_run_mpm (pconf=0x558a5270cbe0, plog=0x558a5273ce00, s=0x558a52c9f840) at mpm_common.c:100 #13 0x0000558a5231b763 in main (argc=<optimized out>, argv=<optimized out>) at main.c:844 Thread 2 (Thread 0x7f4840b70700 (LWP 3836)): #0 0x00007f4864dec9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f486501f65d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68 #2 0x00007f484e14ae4a in get_next (slot=0x558a528d5fe0) at h2_workers.c:209 #3 slot_run (thread=0x558a52828b30, wctx=0x558a528d5fe0) at h2_workers.c:228 #4 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 1 (Thread 0x7f4841b72700 (LWP 3834)): #0 0x00007f4864a2ce97 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f4864a2e801 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f4865020865 in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1955 #3 0x00007f486502b536 in apr_thread_exit (thd=thd@entry=0x558a52ba8980, retval=retval@entry=0) at threadproc/unix/thread.c:206 #4 0x00007f484e14aec6 in slot_run (thread=0x558a52ba8980, wctx=0x558a528d6060) at h2_workers.c:248 #5 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1884170 13f79535-47bb-0310-9956-ffa450edef68

mod_http2: Rename server_pool as pchild in h2_workers_create() To clarify which parent pool the workers threads have. And add a comment about workers_pool_cleanup()'s role and when it runs. No functional change. mod_http2: stop/wait the workers threads before their pool is killed. There shouldn't be any worker thread active when pchild is destroyed (thus each thread's pool), so register workers_pool_cleanup as a pre_cleanup of pchild. This is to avoid races like the below stacktrace, where slot_run() threads are still running when clean_child_exit() is called. Thread 23 (Thread 0x7f4865b79800 (LWP 3740)): #0 0x00007f4864dec449 in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f4865020117 in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2629 #2 pool_clear_debug (pool=pool@entry=0x558a5297e4a0, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1830 #3 0x00007f486501ffee in pool_destroy_debug (pool=0x558a5297e4a0, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #4 0x00007f48650200f0 in pool_clear_debug (pool=pool@entry=0x558a52a41070, file_line=0x558a5237456b "event.c:757") at memory/unix/apr_pools.c:1827 #5 0x00007f486501ffee in pool_destroy_debug (pool=0x558a52a41070, file_line=<optimized out>) at memory/unix/apr_pools.c:1915 #6 0x00007f486502085c in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1957 #7 0x0000558a52326cfc in clean_child_exit (code=0) at event.c:757 #8 0x0000558a52327969 in child_main (child_num_arg=child_num_arg@entry=1, child_bucket=child_bucket@entry=0) at event.c:2926 #9 0x0000558a52327ce5 in make_child (s=0x558a52c9f840, slot=slot@entry=1, bucket=0) at event.c:2992 #10 0x0000558a52327d4c in startup_children (number_to_start=2, number_to_start@entry=3) at event.c:3015 #11 0x0000558a523289ac in event_run (_pconf=<optimized out>, plog=0x558a5273ce00, s=0x558a52c9f840) at event.c:3374 #12 0x0000558a5233e91e in ap_run_mpm (pconf=0x558a5270cbe0, plog=0x558a5273ce00, s=0x558a52c9f840) at mpm_common.c:100 #13 0x0000558a5231b763 in main (argc=<optimized out>, argv=<optimized out>) at main.c:844 Thread 2 (Thread 0x7f4840b70700 (LWP 3836)): #0 0x00007f4864dec9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f486501f65d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68 #2 0x00007f484e14ae4a in get_next (slot=0x558a528d5fe0) at h2_workers.c:209 #3 slot_run (thread=0x558a52828b30, wctx=0x558a528d5fe0) at h2_workers.c:228 #4 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 1 (Thread 0x7f4841b72700 (LWP 3834)): #0 0x00007f4864a2ce97 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f4864a2e801 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f4865020865 in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1955 #3 0x00007f486502b536 in apr_thread_exit (thd=thd@entry=0x558a52ba8980, retval=retval@entry=0) at threadproc/unix/thread.c:206 #4 0x00007f484e14aec6 in slot_run (thread=0x558a52ba8980, wctx=0x558a528d6060) at h2_workers.c:248 #5 0x00007f4864de66db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f4864b0f88f in clone () from /lib/x86_64-linux-gnu/libc.so.6 Submitted by: ylavic Reviewed by: ylavic, jorton, covener git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1884318 13f79535-47bb-0310-9956-ffa450edef68

mod_proxy_http2: thread safety with MPM prefork, still.. The allocator of pchild has no mutex with MPM prefork, but we need one for h2 workers threads synchronization. Even though mod_http2 shouldn't be used with prefork, better be safe than sorry, so forcibly set the mutex in h2_child_init() if it doesn't exist. This prevents the below situation: AddressSanitizer: heap-use-after-free on address 0x6250003ea938 at pc 0x7fe229f40f3c bp 0x7fe22146dd30 sp 0x7fe22146dd28 WRITE of size 8 at 0x6250003ea938 thread T4 #0 0x7fe229f40f3b in apr_pool_destroy memory/unix/apr_pools.c:1015 `-> if ((*pool->ref = pool->sibling) != NULL) #1 0x7fe229f6ef1a in apr_thread_exit threadproc/unix/thread.c:206 #2 0x7fe223a26671 in slot_run /home/yle/src/apache/httpd/trunk.ro/modules/http2/h2_workers.c:248 #3 0x7fe229f6ebcc in dummy_worker threadproc/unix/thread.c:142 #4 0x7fe229ecbea6 in start_thread nptl/pthread_create.c:477 #5 0x7fe229df9d4e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfdd4e) 0x6250003ea938 is located 56 bytes inside of 8192-byte region [0x6250003ea900,0x6250003ec900) freed by thread T6 here: #0 0x7fe22a1ecb6f in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.6+0xa9b6f) #1 0x7fe229f3fe38 in allocator_free memory/unix/apr_pools.c:507 #2 0x7fe229f4107b in apr_pool_destroy memory/unix/apr_pools.c:1043 #3 0x7fe229f6ef1a in apr_thread_exit threadproc/unix/thread.c:206 #4 0x7fe223a26671 in slot_run /home/yle/src/apache/httpd/trunk.ro/modules/http2/h2_workers.c:248 #5 0x7fe229f6ebcc in dummy_worker threadproc/unix/thread.c:142 #6 0x7fe229ecbea6 in start_thread nptl/pthread_create.c:477 mod_proxy_http2: follow up to r1883704. For event/worker MPMs, pchild uses pconf's allocator, so its is NULL. Submitted by: ylavic Reviewed by: ylavic, jorton, covener git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1885138 13f79535-47bb-0310-9956-ffa450edef68

When the session pool is destroyed, so is the beam's pool so we don't want to run the beam cleanup twice. ASan is reporting something like this: ================================================================= ==81201==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000080ce8 at pc 0x7fdc78962cc9 bp 0x7fdc731ff4f0 sp 0x7fdc731ff4e8 READ of size 8 at 0x603000080ce8 thread T11 #0 0x7fdc78962cc8 in recv_buffer_cleanup /home/yle/src/ylavic/httpd/modules/http2/h2_bucket_beam.c:279 apache#1 0x7fdc78962fdc in beam_cleanup /home/yle/src/ylavic/httpd/modules/http2/h2_bucket_beam.c:306 apache#2 0x7fdc7896300c in beam_pool_cleanup /home/yle/src/ylavic/httpd/modules/http2/h2_bucket_beam.c:313 apache#3 0x7fdc7c5a8239 in run_cleanups memory/unix/apr_pools.c:2689 apache#4 0x7fdc7c5a50f9 in pool_clear_debug memory/unix/apr_pools.c:1867 apache#5 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#6 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 apache#7 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#8 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 apache#9 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#10 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 apache#11 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#12 0x7fdc7c5a5827 in apr_pool_destroy_debug memory/unix/apr_pools.c:2014 apache#13 0x7fdc789aeaa5 in h2_session_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_session.c:1934 apache#14 0x7fdc7896a20e in h2_c1_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:188 apache#15 0x7fdc7896b538 in h2_c1_hook_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:308 apache#16 0x5596139aeb28 in ap_run_pre_close_connection /home/yle/src/ylavic/httpd/server/connection.c:45 apache#17 0x5596139af353 in ap_prep_lingering_close /home/yle/src/ylavic/httpd/server/connection.c:128 apache#18 0x5596139af3f2 in ap_start_lingering_close /home/yle/src/ylavic/httpd/server/connection.c:154 apache#19 0x7fdc7835bdf0 in process_lingering_close /home/yle/src/ylavic/httpd/server/mpm/event/event.c:1999 apache#20 0x7fdc78359ccb in process_socket /home/yle/src/ylavic/httpd/server/mpm/event/event.c:1540 apache#21 0x7fdc783608d7 in worker_thread /home/yle/src/ylavic/httpd/server/mpm/event/event.c:2756 apache#22 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 apache#23 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 apache#24 0x7fdc7c337bde in clone (/lib/x86_64-linux-gnu/libc.so.6+0xfcbde) 0x603000080ce8 is located 8 bytes inside of 32-byte region [0x603000080ce0,0x603000080d00) freed by thread T11 here: #0 0x7fdc7c887f07 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:122 apache#1 0x7fdc7c5a5420 in pool_clear_debug memory/unix/apr_pools.c:1906 apache#2 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#3 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 apache#4 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 apache#5 0x7fdc7c5a5827 in apr_pool_destroy_debug memory/unix/apr_pools.c:2014 apache#6 0x7fdc789aeaa5 in h2_session_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_session.c:1934 apache#7 0x7fdc7896a20e in h2_c1_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:188 apache#8 0x7fdc7896b538 in h2_c1_hook_pre_close /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:308 apache#9 0x5596139aeb28 in ap_run_pre_close_connection /home/yle/src/ylavic/httpd/server/connection.c:45 apache#10 0x5596139af353 in ap_prep_lingering_close /home/yle/src/ylavic/httpd/server/connection.c:128 apache#11 0x5596139af3f2 in ap_start_lingering_close /home/yle/src/ylavic/httpd/server/connection.c:154 apache#12 0x7fdc7835bdf0 in process_lingering_close /home/yle/src/ylavic/httpd/server/mpm/event/event.c:1999 apache#13 0x7fdc78359ccb in process_socket /home/yle/src/ylavic/httpd/server/mpm/event/event.c:1540 apache#14 0x7fdc783608d7 in worker_thread /home/yle/src/ylavic/httpd/server/mpm/event/event.c:2756 apache#15 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 apache#16 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 previously allocated by thread T11 here: #0 0x7fdc7c8882b8 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 apache#1 0x7fdc7c5a4d00 in pool_alloc memory/unix/apr_pools.c:1787 apache#2 0x7fdc7c5a507a in apr_palloc_debug memory/unix/apr_pools.c:1828 apache#3 0x7fdc7c4d8160 in apr_brigade_create buckets/apr_brigade.c:90 apache#4 0x7fdc7c4d82d8 in apr_brigade_split_ex buckets/apr_brigade.c:107 apache#5 0x7fdc78967f7c in h2_beam_receive /home/yle/src/ylavic/httpd/modules/http2/h2_bucket_beam.c:729 apache#6 0x7fdc789b65f0 in buffer_output_receive /home/yle/src/ylavic/httpd/modules/http2/h2_stream.c:847 apache#7 0x7fdc789bb655 in h2_stream_read_output /home/yle/src/ylavic/httpd/modules/http2/h2_stream.c:1372 apache#8 0x7fdc789aa155 in on_stream_output /home/yle/src/ylavic/httpd/modules/http2/h2_session.c:1313 apache#9 0x7fdc789956ba in mplx_pollset_poll /home/yle/src/ylavic/httpd/modules/http2/h2_mplx.c:1299 apache#10 0x7fdc7898deb8 in h2_mplx_c1_poll /home/yle/src/ylavic/httpd/modules/http2/h2_mplx.c:532 apache#11 0x7fdc789ae04b in h2_session_process /home/yle/src/ylavic/httpd/modules/http2/h2_session.c:1863 apache#12 0x7fdc78969b0f in h2_c1_run /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:138 apache#13 0x7fdc7896b302 in h2_c1_hook_process_connection /home/yle/src/ylavic/httpd/modules/http2/h2_c1.c:286 apache#14 0x5596139ae4b6 in ap_run_process_connection /home/yle/src/ylavic/httpd/server/connection.c:43 apache#15 0x7fdc78358d67 in process_socket /home/yle/src/ylavic/httpd/server/mpm/event/event.c:1353 apache#16 0x7fdc783608d7 in worker_thread /home/yle/src/ylavic/httpd/server/mpm/event/event.c:2756 apache#17 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 apache#18 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 Thread T11 created by T2 here: #0 0x7fdc7c7baa22 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208 apache#1 0x7fdc7c5d4534 in apr_thread_create threadproc/unix/thread.c:228 apache#2 0x7fdc7836273d in start_threads /home/yle/src/ylavic/httpd/server/mpm/event/event.c:3035 apache#3 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 apache#4 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 Thread T2 created by T0 here: #0 0x7fdc7c7baa22 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208 apache#1 0x7fdc7c5d4534 in apr_thread_create threadproc/unix/thread.c:228 apache#2 0x7fdc78363d9f in child_main /home/yle/src/ylavic/httpd/server/mpm/event/event.c:3262 apache#3 0x7fdc7836483b in make_child /home/yle/src/ylavic/httpd/server/mpm/event/event.c:3421 apache#4 0x7fdc78364b89 in startup_children /home/yle/src/ylavic/httpd/server/mpm/event/event.c:3444 apache#5 0x7fdc78368abc in event_run /home/yle/src/ylavic/httpd/server/mpm/event/event.c:3932 apache#6 0x5596139b6d18 in ap_run_mpm /home/yle/src/ylavic/httpd/server/mpm_common.c:101 apache#7 0x55961399098b in main /home/yle/src/ylavic/httpd/server/main.c:880 apache#8 0x7fdc7c2627ec in __libc_start_main ../csu/libc-start.c:332 SUMMARY: AddressSanitizer: heap-use-after-free /home/yle/src/ylavic/httpd/modules/http2/h2_bucket_beam.c:279 in recv_buffer_cleanup Shadow bytes around the buggy address: 0x0c0680008140: fa fa 00 00 00 00 fa fa fd fd fd fa fa fa fd fd 0x0c0680008150: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c0680008160: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x0c0680008170: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x0c0680008180: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa =>0x0c0680008190: fd fd fd fa fa fa fd fd fd fa fa fa fd[fd]fd fd 0x0c06800081a0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x0c06800081b0: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c06800081c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c06800081d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c06800081e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==81201==ABORTING

When the session pool is destroyed, so is the beam's pool so we don't want to run the beam cleanup twice. ASan is reporting something like this (APR_POOL_DEBUG): ================================================================= ==81201==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000080ce8 at pc 0x7fdc78962cc9 bp 0x7fdc731ff4f0 sp 0x7fdc731ff4e8 READ of size 8 at 0x603000080ce8 thread T11 #0 0x7fdc78962cc8 in recv_buffer_cleanup ~httpd/modules/http2/h2_bucket_beam.c:279 #1 0x7fdc78962fdc in beam_cleanup ~httpd/modules/http2/h2_bucket_beam.c:306 #2 0x7fdc7896300c in beam_pool_cleanup ~httpd/modules/http2/h2_bucket_beam.c:313 #3 0x7fdc7c5a8239 in run_cleanups memory/unix/apr_pools.c:2689 #4 0x7fdc7c5a50f9 in pool_clear_debug memory/unix/apr_pools.c:1867 #5 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #6 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 #7 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #8 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 #9 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #10 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 #11 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #12 0x7fdc7c5a5827 in apr_pool_destroy_debug memory/unix/apr_pools.c:2014 #13 0x7fdc789aeaa5 in h2_session_pre_close ~httpd/modules/http2/h2_session.c:1934 #14 0x7fdc7896a20e in h2_c1_pre_close ~httpd/modules/http2/h2_c1.c:188 #15 0x7fdc7896b538 in h2_c1_hook_pre_close ~httpd/modules/http2/h2_c1.c:308 #16 0x5596139aeb28 in ap_run_pre_close_connection ~httpd/server/connection.c:45 #17 0x5596139af353 in ap_prep_lingering_close ~httpd/server/connection.c:128 #18 0x5596139af3f2 in ap_start_lingering_close ~httpd/server/connection.c:154 #19 0x7fdc7835bdf0 in process_lingering_close ~httpd/server/mpm/event/event.c:1999 #20 0x7fdc78359ccb in process_socket ~httpd/server/mpm/event/event.c:1540 #21 0x7fdc783608d7 in worker_thread ~httpd/server/mpm/event/event.c:2756 #22 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 #23 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 #24 0x7fdc7c337bde in clone (/lib/x86_64-linux-gnu/libc.so.6+0xfcbde) 0x603000080ce8 is located 8 bytes inside of 32-byte region [0x603000080ce0,0x603000080d00) freed by thread T11 here: #0 0x7fdc7c887f07 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:122 #1 0x7fdc7c5a5420 in pool_clear_debug memory/unix/apr_pools.c:1906 #2 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #3 0x7fdc7c5a5179 in pool_clear_debug memory/unix/apr_pools.c:1880 #4 0x7fdc7c5a562e in pool_destroy_debug memory/unix/apr_pools.c:1965 #5 0x7fdc7c5a5827 in apr_pool_destroy_debug memory/unix/apr_pools.c:2014 #6 0x7fdc789aeaa5 in h2_session_pre_close ~httpd/modules/http2/h2_session.c:1934 #7 0x7fdc7896a20e in h2_c1_pre_close ~httpd/modules/http2/h2_c1.c:188 #8 0x7fdc7896b538 in h2_c1_hook_pre_close ~httpd/modules/http2/h2_c1.c:308 #9 0x5596139aeb28 in ap_run_pre_close_connection ~httpd/server/connection.c:45 #10 0x5596139af353 in ap_prep_lingering_close ~httpd/server/connection.c:128 #11 0x5596139af3f2 in ap_start_lingering_close ~httpd/server/connection.c:154 #12 0x7fdc7835bdf0 in process_lingering_close ~httpd/server/mpm/event/event.c:1999 #13 0x7fdc78359ccb in process_socket ~httpd/server/mpm/event/event.c:1540 #14 0x7fdc783608d7 in worker_thread ~httpd/server/mpm/event/event.c:2756 #15 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 #16 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 previously allocated by thread T11 here: #0 0x7fdc7c8882b8 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x7fdc7c5a4d00 in pool_alloc memory/unix/apr_pools.c:1787 #2 0x7fdc7c5a507a in apr_palloc_debug memory/unix/apr_pools.c:1828 #3 0x7fdc7c4d8160 in apr_brigade_create buckets/apr_brigade.c:90 #4 0x7fdc7c4d82d8 in apr_brigade_split_ex buckets/apr_brigade.c:107 #5 0x7fdc78967f7c in h2_beam_receive ~httpd/modules/http2/h2_bucket_beam.c:729 #6 0x7fdc789b65f0 in buffer_output_receive ~httpd/modules/http2/h2_stream.c:847 #7 0x7fdc789bb655 in h2_stream_read_output ~httpd/modules/http2/h2_stream.c:1372 #8 0x7fdc789aa155 in on_stream_output ~httpd/modules/http2/h2_session.c:1313 #9 0x7fdc789956ba in mplx_pollset_poll ~httpd/modules/http2/h2_mplx.c:1299 #10 0x7fdc7898deb8 in h2_mplx_c1_poll ~httpd/modules/http2/h2_mplx.c:532 #11 0x7fdc789ae04b in h2_session_process ~httpd/modules/http2/h2_session.c:1863 #12 0x7fdc78969b0f in h2_c1_run ~httpd/modules/http2/h2_c1.c:138 #13 0x7fdc7896b302 in h2_c1_hook_process_connection ~httpd/modules/http2/h2_c1.c:286 #14 0x5596139ae4b6 in ap_run_process_connection ~httpd/server/connection.c:43 #15 0x7fdc78358d67 in process_socket ~httpd/server/mpm/event/event.c:1353 #16 0x7fdc783608d7 in worker_thread ~httpd/server/mpm/event/event.c:2756 #17 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 #18 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 Thread T11 created by T2 here: #0 0x7fdc7c7baa22 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208 #1 0x7fdc7c5d4534 in apr_thread_create threadproc/unix/thread.c:228 #2 0x7fdc7836273d in start_threads ~httpd/server/mpm/event/event.c:3035 #3 0x7fdc7c5d3e57 in dummy_worker threadproc/unix/thread.c:153 #4 0x7fdc7c441d7f in start_thread nptl/pthread_create.c:481 Thread T2 created by T0 here: #0 0x7fdc7c7baa22 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208 #1 0x7fdc7c5d4534 in apr_thread_create threadproc/unix/thread.c:228 #2 0x7fdc78363d9f in child_main ~httpd/server/mpm/event/event.c:3262 #3 0x7fdc7836483b in make_child ~httpd/server/mpm/event/event.c:3421 #4 0x7fdc78364b89 in startup_children ~httpd/server/mpm/event/event.c:3444 #5 0x7fdc78368abc in event_run ~httpd/server/mpm/event/event.c:3932 #6 0x5596139b6d18 in ap_run_mpm ~httpd/server/mpm_common.c:101 #7 0x55961399098b in main ~httpd/server/main.c:880 #8 0x7fdc7c2627ec in __libc_start_main ../csu/libc-start.c:332 SUMMARY: AddressSanitizer: heap-use-after-free ~httpd/modules/http2/h2_bucket_beam.c:279 in recv_buffer_cleanup Shadow bytes around the buggy address: 0x0c0680008140: fa fa 00 00 00 00 fa fa fd fd fd fa fa fa fd fd 0x0c0680008150: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c0680008160: fd fd fd fd fa fa fd fd fd fd fa fa fd fd fd fd 0x0c0680008170: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x0c0680008180: fd fd fa fa fd fd fd fd fa fa fd fd fd fa fa fa =>0x0c0680008190: fd fd fd fa fa fa fd fd fd fa fa fa fd[fd]fd fd 0x0c06800081a0: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd 0x0c06800081b0: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c06800081c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c06800081d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c06800081e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==81201==ABORTING git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897868 13f79535-47bb-0310-9956-ffa450edef68

Fixed IE10's DNT setting being ignored

b6f2923

Humbedooh closed this in 309142b Jan 4, 2014

notroj added a commit to notroj/httpd that referenced this pull request Nov 8, 2019

Fix parse error apache#2?

9dc36ec

arminabf mentioned this pull request Jun 25, 2020

fix race condition leading to double free error #129

Closed

Fixed IE10's DNT setting being ignored #2

Fixed IE10's DNT setting being ignored #2

Uh oh!

Conversation

OscarGodson commented Sep 9, 2012

Uh oh!

mxriverlynn commented Sep 9, 2012

Uh oh!

zacparker commented Sep 9, 2012

Uh oh!

liquidboy commented Sep 9, 2012

Uh oh!

hizanberg commented Sep 9, 2012

Uh oh!

OscarGodson commented Sep 9, 2012

Uh oh!

interscaperob-duplicate commented Sep 9, 2012

Uh oh!

hizanberg commented Sep 9, 2012

Uh oh!

OscarGodson commented Sep 9, 2012

Uh oh!

interscaperob-duplicate commented Sep 10, 2012

Uh oh!

hizanberg commented Sep 10, 2012

Uh oh!

interscaperob-duplicate commented Sep 10, 2012

Uh oh!

hizanberg commented Sep 10, 2012

Uh oh!

interscaperob-duplicate commented Sep 10, 2012

Uh oh!

covener commented Sep 10, 2012

Uh oh!

imanavg commented Sep 10, 2012

Uh oh!

OscarGodson commented Sep 10, 2012

Uh oh!

sschocke commented Sep 10, 2012

Uh oh!

EvilHom3r commented Sep 10, 2012

Uh oh!

gabranth commented Sep 10, 2012

Uh oh!

BrazilianJoe commented Sep 10, 2012

Uh oh!

londospark commented Sep 10, 2012

Uh oh!

Fever905 commented Sep 10, 2012

Uh oh!

mamund commented Sep 10, 2012

Uh oh!

imanavg commented Sep 11, 2012

Uh oh!

kristiaanvandeneynde commented Sep 11, 2012

Uh oh!

unikitty37 commented Sep 11, 2012

Uh oh!

imanavg commented Sep 11, 2012

Uh oh!

kristiaanvandeneynde commented Sep 11, 2012

Uh oh!

ghost commented Sep 11, 2012

Uh oh!

ghost commented Sep 12, 2012

Uh oh!

imanavg commented Sep 12, 2012

Uh oh!

ghost commented Sep 12, 2012

Uh oh!

imanavg commented Sep 12, 2012

Uh oh!

ziyan commented Sep 14, 2012

Uh oh!

kristiaanvandeneynde commented Sep 18, 2012

Uh oh!

imanavg commented Sep 23, 2012

Uh oh!

kristiaanvandeneynde commented Sep 23, 2012

Uh oh!