Remove the dependency on grpc-go's experimental API (e.g. resolver & balancer)

[ahrtr]

What would you like to be added?

etcd is depending on some experimental APIs of grpc-go, such as resolver. Once these APIs change in future, it might break etcd. So we should try to remove the dependency on any experimental APIs; instead, let's try to use stable APIs.

References:

• transport: new stream with actual server name grpc/grpc-go#5748 (comment)
• transport: new stream with actual server name grpc/grpc-go#5748 (comment)
• panic: runtime error: comparing uncomparable type map[string]interface {} #19700 (comment)

We should have

• a summary on existing design/implementation first
• proposal(s) to replace them, and with pros & cons. invite grpc's experts to review.
• evaluate the impact on etcd itself and users of etcd client sdk.

Mitigations:

• Replace resolver.State.Addresses with resolver.State.Endpoint.Addresses #19780
• Deprecate usage of gresolver.Address.Metadata #19706
  • feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785
  • chore(client/v3/naming): deprecate Metadata field in Endpoint struct #19825
• mark the gRPC naming and discovery feature as experimental, see also panic: runtime error: comparing uncomparable type map[string]interface {} #19700 (comment)
  • Mark the gRPC naming and discovery feature as experimental website#990

Long-term solution?

• use the GCP gRPC connection pool ?
  • panic: runtime error: comparing uncomparable type map[string]interface {} #19700 (comment)

Related followup:

• An endpoint may have multiple addresses: i.e. IPv4 and IPv6. Currently we only assume each endpoint only has one address

  • panic: runtime error: comparing uncomparable type map[string]interface {} #19700 (comment)
  • Replace resolver.State.Addresses with resolver.State.Endpoint.Addresses #19780

• Remove the deprecated grpc.DialContext #19772

Why is this needed?

Improve stability

[dfawley]

grpc-go TL here. I'm happy to work with you to find / provide a stable solution, but I would first need to know more about your use cases & requirements.

[tjungblu]

just to randomly add what I've found in #14501 - once we reconsider the balancer/resolver, we should also add some circuit breaking. There were some ideas by Easwar in grpc/grpc-go#5672.

[ahrtr]

Sorry for the late response.

The use cases & requirement seems to be simple. Users may specify multiple endpoints on the client side (see example below), and etcd clientv3 needs to support client side load balancing.

$ ./etcdctl --endpoints=http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379 get k1 

Currently the target used when calling grpc.DialContext is a string in format etcd-endpoints://<pointer>/<authority>, e.g. etcd-endpoints://0xc00036e1e0/127.0.0.1:2379. etcd implements a customized resolver , which is passed to gRPC via grpc.WithResolvers. The loadBalancingPolicy is round_robin.

Some historical background:

1. Previously etcd 3.4 implemented a custom load balancer, and it was changed to gRPC's Resolver and load balancing in 3.5.0 in clientv3: Replace balancer with upstream grpc solution #12671.
2. But unfortunately clientv3: Replace balancer with upstream grpc solution #12671 introduced a critical issue etcd client v3 v3.5.0 sends invalid :authority header field value #13192 due to invalid :authority header, which was fixed in Fix http2 authority header in single endpoint scenario #13359

Based on https://github.com/grpc/grpc/blob/master/doc/naming.md, it seems the gRPC resolver plugin (although the API is still experimental) is still the right direction? If not, could you please advise the correct approach? cc @dfawley

[ahrtr]

@dfawley could you advise how to replace the experimental API mentioned above?

[ahrtr]

One more dependency on experimental API: grpcServer.ServeHTTP

[dfawley]

Also sorry for the late rely

Based on grpc/grpc@master/doc/naming.md, it seems the gRPC resolver plugin (although the API is still experimental) is still the right direction?

For this you could instead create (N) gRPC clients, one for each address, and do round robin dispatch to them by wrapping them in another object that implements its own grpc.ClientConnInterface and has a pool of the grpc.ClientConns created. This is what is done by Google's cloud client libraries to spread load across multiple connections.

What would be your concerns with such an approach?

[ahrtr]

What would be your concerns with such an approach?

Thanks for the response. The only concern is what (etcd or gRPC?) is the best place to implement it. It looks like a generic client side load balancer solution, does it make more sense to get it wrapped in grpc/grpc-go instead of etcd?

Also some APIs (such as resolver.Resolve, balancer.Balancer , etc) were added as experimental APIs about 6 years ago, they are still experimental APIs today. Have you considered to graduate them to stable APIs or deprecate/remove them and provide alternative solution/APIs?

cc @mitake @ptabor @serathius @spzala @liggitt @wojtek-t @lavalamp for opinions.

[dfawley]

I was just digging around and saw that C-core has the following name resolver (https://github.com/grpc/grpc/blob/master/doc/naming.md#name-syntax):

• ipv4:address[:port][,address[:port],...] -- IPv4 addresses
  • Can specify multiple comma-delimited addresses of the form address[:port]:
    • address is the IPv4 address to use.
    • port is the port to use. If not specified, 443 is used.

We could also explore making this a first-class name resolver. Would that take care of all of your need for resolver and balancer APIs?