add proxy.config.http.cache.max_open_write_retry_timeout parameter

[traeak]

The current cache write lock retries work by spinning N times at ms intervals, defined by proxy.config.http.cache.max_open_write_retries (default 1) and proxy.config.http.cache.open_read_retry_time (default 10ms).

While performing synthetic load testing with an origin that still accepts connections but stalls, we encountered larger than expected transaction ttms. This was caused by the retry time space going far beyond the configured 10ms.

Using the following setting I was getting too much variation in time spent spinning on the write lock. Anywhere from 1.5s to 8s.

CONFIG proxy.config.http.cache.max_open_write_retries INT 150

So I added this millisecond timeout/deadline:

CONFIG proxy.config.http.cache.max_open_write_retry_timeout INT 1500

When set this timeout is used in preference to the sloppy open_write_retries * open_read_retry_time

[ezelkow1]

[approve ci rocky]

[bryancall]

@SolidWallOfCode is going to try to take a look at this

[bryancall]

@cmcfarlen is going to take a look at this

[cmcfarlen]

Check the logic on the read retry assert.

[cmcfarlen]

lgtm after removing the backward logic.

[traeak]

We've been running with this patch in prod but disabled for a bit now. The last squash had some changes from when I ran tests using docker containers, i removed a small optimization that missed evaluating an open_write increment.

[bryancall]

Based on @cmcfarlen review