fix: pin Farm client to HTTP/1.1 and cap concurrent uploads

[basvandijk]

Problem

Farm's web server (warp-tls) advertises HTTP/2 SETTINGS_MAX_CONCURRENT_STREAMS=64 and tears down the whole connection (RST_STREAM + GOAWAY \"exceeds max concurrent\") when a burst of multiplexed streams exceeds that limit — strict behavior vs. RFC 9113 §5.1.2, which prescribes rejecting only the offending stream.

The blocking reqwest client used by the test driver multiplexes all parallel requests over a single HTTP/2 connection. When a test starts many VMs at once (e.g. the 120-node xnet_slo_120_subnets_staging_test), the request burst outruns the server's SETTINGS frame on high-RTT paths and Farm kills the connection, failing all in-flight requests at once:

ERRO ... sending a request to Farm failed: reqwest::Error { kind: Request, ...,
source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Http2,
Error { kind: GoAway(b\"exceeds max concurrent\", REFUSED_STREAM, Remote) })) }
WARN ... Starting VM failed with: Retried too many times: sending a request to Farm

→ setup fails with zero VMs started. Low-RTT environments usually win the race, which is why this mostly bites when running system tests from devenvs.

Diagnosis detail: all failures land in the same millisecond (one shared connection dying), and ss shows only a single connection to Farm during a 120-VM burst.

Fix

• Force HTTP/1.1 on the Farm client: one TCP connection per concurrent request, no stream-concurrency limit to violate.
• Cap concurrent upload_file calls at 32 with a small blocking semaphore (RAII permit held across the whole retry loop). Under HTTP/1.1 this bounds parallel TCP connections; it also keeps the code safe should the HTTP/1.1 pin ever be removed (32 < 64 leaves headroom for other requests sharing the client).

Verification

Ran //rs/tests/message_routing/xnet:xnet_slo_120_subnets_staging_test (120 single-node subnets) from a dev container:

• Without this change: setup fails — all 120 VM-start requests die with the GoAway error above, 0 VMs started.
• With this change: setup starts all 120 VMs reliably and the test runs to completion (passed in combination with fix: evict dead pooled XNet connections via HTTP/2 keep-alive pings #10455).

[Copilot]

Pull request overview

This PR mitigates Farm connection teardown failures during large parallel VM setups by preventing HTTP/2 stream multiplexing against Farm’s strict concurrent-stream handling and by bounding upload concurrency in the test driver.

Changes:

• Force the Farm reqwest::blocking client to use HTTP/1.1 to avoid HTTP/2 multiplexed bursts exceeding Farm’s SETTINGS_MAX_CONCURRENT_STREAMS.
• Add a small blocking counting semaphore shared across Farm clones to cap concurrent upload_file calls at 32, holding the permit across the full retry loop.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nmattia]

Nice! couple questions to help me understand better

[nmattia]

I'm a bit confused by this comment. I see below that we now enforce HTTP1.1, so what do we mean by "..., so stay well below [the HTTP2 max streams]"?

[nmattia]

is this ever called within tokio? this will be blocking the whole thread, correct?