Problem
When binding priorities are all the same (the default of 0), the platform levels reads across bindings and journals so that it proceeds in ascending wall-clock (publication time) order. This means that when a new binding with a large historical backlog is introduced — for example, adding a new L1 rollup into an L2 derivation — all of that historical data is ordered before any current data from other bindings, because its publication time is earlier.
This caused a real issue with the catalog-stats L2 derivation: introducing a new test data plane's L1 binding caused the L2 derivation to stall on current data while it worked through the historical backlog, because the leveled reads prioritized the older publication times.
Read priority exists for exactly this use case — marking a binding as lower priority so the derivation prefers other bindings first and works through the backlog second. However, using it today requires raising the priority of all other bindings, because priority is a non-negative integer and the default is 0. You can't express "this one binding is lower priority than normal."
Proposal
Allow negative values for the priority field on derivation transforms and materialization bindings. This way, a user (or operator) can set priority: -1 on a single binding to express that it is lower-priority than bindings using the default (0), without needing to touch every other binding.
This should be backwards-compatible with all current usages, since existing values > 0 continue to be higher priority and the default of 0 is unchanged.
Expected behavior
priority: -1 (or any negative value) is accepted in derivation transform and materialization binding configs- Negative-priority bindings are read after all default (
0) and positive-priority bindings
- Existing behavior for
priority >= 0 is unchanged
Problem
When binding priorities are all the same (the default of
0), the platform levels reads across bindings and journals so that it proceeds in ascending wall-clock (publication time) order. This means that when a new binding with a large historical backlog is introduced — for example, adding a new L1 rollup into an L2 derivation — all of that historical data is ordered before any current data from other bindings, because its publication time is earlier.This caused a real issue with the catalog-stats L2 derivation: introducing a new test data plane's L1 binding caused the L2 derivation to stall on current data while it worked through the historical backlog, because the leveled reads prioritized the older publication times.
Read priority exists for exactly this use case — marking a binding as lower priority so the derivation prefers other bindings first and works through the backlog second. However, using it today requires raising the priority of all other bindings, because priority is a non-negative integer and the default is
0. You can't express "this one binding is lower priority than normal."Proposal
Allow negative values for the
priorityfield on derivation transforms and materialization bindings. This way, a user (or operator) can setpriority: -1on a single binding to express that it is lower-priority than bindings using the default (0), without needing to touch every other binding.This should be backwards-compatible with all current usages, since existing values
> 0continue to be higher priority and the default of0is unchanged.Expected behavior
priority: -1(or any negative value) is accepted in derivation transform and materialization binding configs0) and positive-priority bindingspriority >= 0is unchanged