Prefetch COSE endorsements of previous identities

[maxtropets]

This PR changes the way CCF pulls up COSE endorsements for previous service identities (one of #7401).

Why?

• To allow instantly verify any incoming COSE receipt against the current ledger in the future, therefore one more step to self-transparency.

What's changed?

• On-demand endorsement fetching was triggered by historical queries adapter (or explicitly by the app writer) and needed N triggers, depending of the length of the chain (amortised O(1) though), where N = recovery_count.
• With this change, it starts asynchronously, waits (periodically pulls) until the previous identity has been endorsed during DR, and proceeds to preload the whole chain available.
• populate_cose_service_endorsements now depends on it, rather then re-loading the chain from the historical cache.

Q. Why wait until the current identity endorses the previous one?

• it keeps code simpler
• it does not interfere with mid-recovery historical state requests, if any

Q. Why not fail the node if prefetching went wrong?

• it hangs forever in FetchStatus::Fail to play safe, and although user can't make a historical request (with COSE endorsements), the node keeps functioning, although producing FAIL logs once when fetch has failed, and any time it's called to get the chain
• I think it's more sane, as in ill-formed chain does not mean the current identity is bad, and screaming about error but allow node to live looks fine to me

Q. Why fetching status is atomic, but everything else isn't?

• This's the barrier to prevent interaction with the subsystem until it's done with modifications of internal state
• Once completed, no more modifications expected, so can read endorsements asynchronously.

Q. Where are the tests?

• Tested well on current recovery tests with cose_endorsements endpoints for previous epochs.
• More tests around this will inevitably come with follow-up work to this, e.g. COSE receipts (+verification) endpoint exposed, etc.

[Copilot]

Pull request overview

This PR refactors the COSE endorsement fetching mechanism from on-demand to asynchronous prefetching. The goal is to enable instant verification of incoming COSE receipts against the current ledger by preloading the entire chain of previous service identity endorsements during disaster recovery.

Key changes:

• Moves endorsement fetching logic from historical queries adapter into the NetworkIdentitySubsystem
• Implements asynchronous prefetching that starts in the subsystem constructor and retries until complete
• Changes populate_cose_service_endorsements to depend on the prefetched chain rather than on-demand loading

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/node/rpc/network_identity_subsystem.h	Adds new async prefetching implementation with validation logic for endorsement chains
src/node/historical_queries_utils.cpp	Removes on-demand fetching logic and updates to use prefetched endorsements from subsystem
src/node/historical_queries_adapter.cpp	Updates function call to pass network identity subsystem instead of state cache
src/node/tx_receipt_impl.h	Changes endorsements type from raw vector to CoseEndorsementsChain type alias
include/ccf/network_identity_interface.h	Adds new interface methods and types for fetching status and endorsement chains
include/ccf/historical_queries_utils.h	Updates function signature to accept network identity subsystem
src/enclave/enclave.h	Updates subsystem construction to pass historical cache dependency

[achamayou]

I think it's more sane, as in ill-formed chain does not mean the current identity is bad, and screaming about error but allow node to live looks fine to me

Missing links are inconclusive, some ledger files could be missing etc. But there are ill-formed chains that are definitely very bad, like the current identity pointing a previous at a seqno that directly conflicts with what we fetch. I think we need to consider shutting down if that happens.