Protobufs + gRPC

[liamzebedee]

Exploring using protocol buffers and gRPC.

• Rewrite all the current RPC messages as protobuf types and services.
• Build a simple PoC server/peer which does the same stuff as netpeer.go

Currently the RPC interface is implemented as a simple HTTP server, with a set of RPC message types that are serialised using JSON, and peers just send messages and some messages have responses.

Some immediate benefit to using protobufs:

• Compact encoding - protobufs does "packing", meaning it can basically take a uint32, determine the number fits into a smaller uint8, and do that automatically.
• Fast encoding/decoding speed.

Some benefits to using gRPC:

• Events. Right now, peers can gossip new blocks and transactions. Usually RPC is just request-response, but gRPC also includes a type called "bidirectional streams". This can basically be used to disseminate events. I like it - it's the exact pattern but formulated with strong types (ie. it's its own class of message). Right now this isn't exactly clear from reading the code. Also the bidirectional streams use HTTP/2 streaming which makes it really efficient too (?).
• I like the service PeerService { definition format. Makes the wire protocol more readable.
• Some other stuff here:
  • https://grpc.io/blog/principles/
  • Flow control
  • https://martinfowler.com/articles/distributed-objects-microservices.html

Things I still gotta work out:

• translating between protobuf messages and the "fixed-size" type definitions - protobuf doesn't support [32]byte, it only has bytes. So you have to translate that.
• canonical encoding for block hashing. protobufs do not have support for canonical encoding.

[moshababo]

protobuf (de)serialization isn't deterministic, meaning that a given memory layout can have multiple valid serialized representations. This is fine for the node's API RPC & database-related serialization, but less good for p2p networking, and bad for crypto-related serialization (hashing / signing).

It can be turned deterministic by applying a stricter set of rules (example: node/libs/protobuf/src/proto_fmt.rs), but I'm not sure if such implementation is available in Go.

Also, using gRPC for p2p networking can be problematic. Although most things are doable on the the HTTP/2 connection, It's probably better to either go with plain TCP for full control, or use libp2p.

[vrypan]

May be wort reaching out to the Farcaster team. They use gRPC and libp2p and the network size is ~10,000 nodes.

[liamzebedee]

Thanks @moshababo

protobuf (de)serialization isn't deterministic, meaning that a given memory layout can have multiple valid serialized representations. This is fine for the node's API RPC & database-related serialization, but less good for p2p networking, and bad for crypto-related serialization (hashing / signing).

I'm aware of the serialisation aspects, though haven't heard about the impact on networking? I would guess that one aspect of the dynamically sized encoding is that it might impact benchmarks re: networking.

[liamzebedee]

Also, using gRPC for p2p networking can be problematic. Although most things are doable on the the HTTP/2 connection, It's probably better to either go with plain TCP for full control, or use libp2p.

Can you elaborate / link some references? libp2p definitely on my radar, for their TURN/NAT holepunching.

[liamzebedee]

Ugh I have accidentally closed this while deleting some old branches, github UX is a bit shite. Will try fix, still curious on comments above.