HTTP: The definition of `server.address` differs across various metrics

[antonfirsov]

Area(s)

HTTP

What's the problem?

The HTTP client request metrics spec (e.g. for http.client.request.duration) and the HTTP client span spec defines server.address as

Host identifier of the "URI origin" HTTP request is sent to.

while in the spec of other metrics (e.g. http.client.connection.duration), and the attribute registry for "Server" it is defined as

Server domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name.

This leads to a problem deciding whether the Host header should take precedence over the URI origin. The current wording seems to slightly suggest that request span (and metrics) should use the URI origin while the connection metrics should use the Host header but this doesn't seem to be neither intentional nor clear.

Context: in dotnet/runtime#117049 a user reported a problem with .NET's HttpClient metrics which currently report the URI origin instead of the Host header. While changing this to report the Host header instead seems better from practical POV, the expected behavior is not clear from the current state of the semconv spec.

Describe the solution you'd like

• Uniformize the specs or make it clear in comments that the expected attribute values differ intentionally across various metrics/spans.
• In either case, it should be clear if the Host header has precedence over the URI origin (for each metric).
• Personally, I would prefer the Host header whenever available since it adds the host information back when an IP address is being used in the URI. There is network.peer.address for IP addresses anyways.

[lmolkova]

@antonfirsov is it the case when HttpRequestMessage.RequestUri.Host is different than the Host header? I would imagine when it gets to sending physical request, the HttpRequestMessage.RequestUri.Host is actually ignored, is this right?

From the HTTP RFC perspective, the only valid case when there are multiple authorities is HTTP 1.1 proxy case https://www.rfc-editor.org/rfc/rfc9112.html#section-3.2 when the request line has the absolute URL. In other cases, even when abolsute-url is present, it should be consistent with the host.

The RFC 9112 says:

When an origin server receives a request with an absolute-form of request-target, the origin server MUST ignore the received Host header field (if any) and instead use the host information of the request-target. Note that if the request-target does not have an authority component, an empty Host header field will be sent in this case

Looking into the original issue (dotnet/runtime#117049), it doesn't seem to be the proxy case, so I'm a bit lost how/why Host and target are different.

So a few thoughts/questions

• HttpRequestMessage.RequestUri is not really URI origin from HTTP RFC standpoint, right? We need to figure out what happens over then wire.
• The host header is the source of truth only if there is no authority in the request-target,
• Perhaps we should link request-target instead of URI-origin in semconv?
• Different request-target and Host header is where opt-in http.request.header.host attribute would be helpful. It's already there on spans, but would be cool to add it to HTTP client metrics

[antonfirsov]

The primary case when request.RequestUri.IdnHost != request.Headers.Host is when an IP address is being used in the URI. In this case it is necessary to specify the Host header so it can be used for TLS. (e.g. https://15.197.167.90, with the Host header set to opentelemetry.io). Some users prefer to implement load-balancing by manually choosing the IP address from the list returned by DNS. By logging the IP the host information gets lost.

[lmolkova]

@antonfirsov in this case semconv currently (in the stable part) prescribes to use 15.197.167.90 in the server.address. It's consistent with HTTP RFC which says to ignore Host header. So I don't think we should (or could) change it.

But we can add http.request.header.{key} as opt-in on metrics - this would preserve the host info for those who prefer to do this.

[Tragetaschen]

I'm a bit confused

In this case it is necessary to specify the Host header so it can be used for TLS.

That's probably nitpicky: The Host header is only sent after a TLS connection has been established. It's value on the client side, though, is used for the SNI extension as well so the server can chose the right certificate, but even that behavior could be overridden.

It's consistent with HTTP RFC which says to ignore Host header.

Are you referring to the above RFC 9112? Then this has no relevance since the scenario here is not about a client talking to a proxy and sending an absolute URI in the request line in combination with a Host header. It's a normal client-to-server HTTP request with the URL's path in the request line and a hostname in the Host header.

The presented use case is a gRPC client that runs DNS resolution of a given hostname and then constructs for each resolved IP a URL with IP address as hostname. The identity is still the original hostname, but the individual URLs on the client don't reflect that anymore. Instead, the hostname from the URL (the IP) is exclusively used to establish TCP (or QUIC) connections. Both the SNI extension and the Host header use the original hostname to denote the server's identity so the server (or a reverse proxy in front of it) can properly determine the target of the request.

Or put into less words: whether the URL is https://opentelemetry.io or https://15.197.167.90 has no influence on the octets that are sent over the wire in the actual request.

[lmolkova]

Are you referring to the above RFC 9112? Then this has no relevance since the scenario here is not about a client talking to a proxy and sending an absolute URI in the request line in combination with a Host header. It's a normal client-to-server HTTP request with the URL's path in the request line and a hostname in the Host header.

@Tragetaschen in this case, the only thing that knows about different hosts is .NET - HttpRequestMessage.RequestUri contains the full form of the URI but the full form never goes over the wire for that specific request. As far as otel semconv are concerned, the :authority header should be used on telemetry.

But in some edge cases (HTTP 1.1 + full url in the request-line) the request-line would win over Host header.

@antonfirsov could you confirm?

[antonfirsov]

@lmolkova as @Tragetaschen pointed out, the absolute-form is only used when the request is being sent to a proxy. In most cases the origin-form will be used and a Host header well be added by the client, as required by the RFC. So if you send a GET request to http(s)://opentelemetry.org,

GET / HTTP/1.1
Host: opentelemetry.org

will be sent to 15.197.167.90:port. If the request object also contains a Host header, the .NET implementation will only use the URI Origin for TCP connection establishment. In case of HTTP/2, the Host header will be used as :authority when present, if not, it will be taken from the URI. I expect other implementations to behave similarly.

Coming back to your questions:

HttpRequestMessage.RequestUri is not really URI origin from HTTP RFC standpoint, right?

The term "URI Origin" is useful when reasoning about URI-s in isolation, but I do not find it useful when it comes to an HTTP request going over the wire. There are too many details complicating the picture, see the example I described above.

The host header is the source of truth only if there is no authority in the request-target,

this could only happen in the proxy case (absolute-form).

Perhaps we should link request-target instead of URI-origin in semconv?

The request-target component does not necessarily contain a host name (eg. it doesn't in the origin-form). Moreover, RFC 9112 only covers HTTP/1.1, but not HTTP/2 and HTTP/3. Figuring out an exact definition for server.address and linking it to RFC-s would lead to a long and complicated definition. Instead, I think the best approach would be to allow implementations to emit whatever identifies the origin host for a given outgoing request. From that sense I like the wording in the attribute registry. I don't see why is there an alternative wording using the URI origin term in other places.

Different request-target and Host header is where opt-in http.request.header.host attribute would be helpful.

.NET currently doesn't have a generic mechanism to control opt-in attributes, and without that I don't think would implement those.

Conclusion

It's up to semconv to define what information should be in server.address, but I don't think there is anything in the RFCs that imposes constraints. From practical POV it would be great to have a definition that allows implementations to use the Host header. This would only make the resulting metrics richer when HTTP clients are being fed with an IP URI + Host header, since the metrics would contain both the IP address (network.peer.address) and the hostname (server.address).

[lmolkova]

Thanks for the context.

It seems we're all on the same page that instrumentation and conventions should use contents of the host/authority header (unless it's a proxy or other esoteric, but possible, scenario involving absolute-form target).

I'm going to break this problem down into 3 pieces:

• clarifying URI origin where it's used - I agree it's confusing today. We need a more precise explanation on how to populate server.address | port. I created Replace confusing URI origin with more specific guidance #2463 to address it. PTAL
• clarifying connection metrics. They are not stable and we have more wiggle room, but connection and request-level metrics can't be exactly the same - source of connection-level host:port is different.
• adding opt-in header attributes to metrics - it's a nice to have, not required to support, etc

[antonfirsov]

Regarding connection metrics: the current wording expects that in case a connection is made to a proxy, server.address SHOULD represent the server address behind any intermediaries.

This doesn't make sense with implementations that share the same proxy connection for all requests going through the proxy, since in such cases the server address behind any intermediaries is a trait of the request not the connection. As a result, we cannot implement this in .NET, see discussion under dotnet/runtime#117540 (comment).

Would it make sense to relax the wording?

[lmolkova]

@antonfirsov could you please send a pr with a proposal?