Bad metadata for TransactionPaymentApi.query_info call in Metadata v15.

[ypopovych]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

While working on the Swift SDK I found that the new RuntimeApi metadata has inconsistency for the TransactionPaymentApi.query_info call (and query_fee_details call too).

Right now the call is described as:

TransactionPaymentApi_query_info {
    uxt: sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic(Vec<u8>),
    len: u32
}

UncheckedExtrinsic has the sequence type inside, which should be encoded with a size prefix. But in reality, call expects uxt to be encoded without prefix, as fixed data. This inconsistency prevents the dynamic encoding of this runtime call.

Steps to reproduce

No response

[bkchr]

cc @lexnv

[lexnv]

Hey,

Thanks for raising this question!

The runtime transaction API indeed expects a raw encoded UnchcekdExtrinsic without any length prefix.
The parameter types of those runtime calls are exposed without any information regarding their encoded content.
As such any information on custom encoding is lost in the process of populating the metadata.

For example, the UncheckedExtrinsic is represented in substrate as

substrate/primitives/runtime/src/generic/unchecked_extrinsic.rs

Lines 43 to 54 in 9a6d706

	#[derive(PartialEq, Eq, Clone)]
	pub struct UncheckedExtrinsic<Address, Call, Signature, Extra>
	where
	Extra: SignedExtension,
	{
	/// The signature, address, number of extrinsics have come before from
	/// the same signer and an era describing the longevity of this transaction,
	/// if this is a signed extrinsic.
	pub signature: Option<(Address, Signature, Extra)>,
	/// The function that should be called.
	pub function: Call,
	}

However, the logic of encoding / decoding the extrinsic is specific to the chain and a custom scale_info::TypeDef is propagated for this type instead

substrate/primitives/runtime/src/generic/unchecked_extrinsic.rs

Lines 83 to 86 in 9a6d706

	// Because of the custom encoding, we can only accurately describe the encoding as an
	// opaque `Vec<u8>`. Downstream consumers will need to manually implement the codec to
	// encode/decode the `signature` and `function` fields.
	.composite(Fields::unnamed().field(|f| f.ty::<Vec<u8>>()))

This is because we include custom information in the first byte of the extrinsic (signed or nor; version) and apply custom hashing logic for the signature type.

The current metadata version cannot describe appropriately these custom operations.

We have a similar issue while implementing the runtime API calls in subxt. Our subxt-codegen would generate a static interface similar to:

#[derive(TypeInfo)]
pub struct UncheckedExtrinsic<_0, _1, _2, _3>(
    pub ::std::vec::Vec<::core::primitive::u8>,
    #[codec(skip)] pub ::core::marker::PhantomData<(_1, _0, _2, _3)>,
);

Since we want to preserve the generic types as close as possible to Substrate, we have altered the representation scale type from a Sequence to a Tuple. This will result in an encoding error for the static code.

We could provide a type wrapper directly from substrate, say struct AlreadyEncoded(pub Vec<u8>) to signal to the metadata users that those bytes are supposed to be passed through, but that feels like complicating the format more than simplifying it.

In subxt in particular, I believe the path forward would be to create a custom UncheckedEncodedExtrinsic type (for example), which does not encode the bytes. Then, substitute the UncheckedExtrinsic type with the newly created one.

I believe this should be handled by the Runtime API generator (ie user), similarly to how we require the user SDKs to have specific knowledge about encoding an extrinsic to be submitted to the chain.

A crazy idea is that we probably could explore in a future V16 the possibility of having custom logic included in the metadata:

• present a minimal WASM blob of code for metadata types specific operations
• include the function name responsible for the given type into scale_info::Type
• SDKs would have the possibility of implementing things natively (similar to how things are done now)
• SDKs that don't want to be highly optimized would opt for parsing / loading / calling the WASM blob
  That would allow signing with custom logic, building and encoding the extrinsic, an identity encoding fn etc

@jsdw @bkchr would love to hear your thoughts 🙏

[jsdw]

In subxt in particular, I believe the path forward would be to create a custom UncheckedEncodedExtrinsic type

Yup, I think that's the way to go at the mo in Subxt; create/reuse a custom type eg pub struct Encoded(Vec<u8>) and use type substitution in the subxt macro to swap out sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic with this type. This is the usual approach that we have to take when there is some "custom" logic that needs to be adhered to that is not available via the TypeInfo/metadata.

Either myself of @lexnv can have a go at showing a quick workaround for this here tomorrow; it shouldn't be too much effort. And then we'll likely add a baked-in type substitution for the next release to more broadly fix the issue.

[jsdw]

@ypopovych we merged a fix in Subxt (see above PR) that will make it into the next release in a few days.

If you wanted a workaround today, you could either:

1. Added a custom type like this: https://github.com/paritytech/subxt/pull/1076/files#diff-db70f5a4eb384b30976367b6cd082c92e9331e4c959e2b1d6722f00df148cae0R21, and use type substitution on the macro (https://docs.rs/subxt/latest/subxt/attr.subxt.html#substitute_typepath---with--) to make use of it, or
2. Much simpler, use api.runtime_apis().at_latest().call_raw and encode the value like we did in this test: https://github.com/paritytech/subxt/pull/1076/files#diff-ad3e47e2dc26a93cf6dcf51a964d06a4c98e0243f5a916b8f19bb600ef3511b4R85