|
| 1 | +# Committed Resource Reservation System |
| 2 | + |
| 3 | +The committed resource reservation system manages capacity commitments, i.e. strict reservation guarantees usable by projects. |
| 4 | +When customers pre-commit to resource usage, Cortex reserves capacity on hypervisors to guarantee availability. |
| 5 | +The system integrates with Limes (via the LIQUID protocol) to receive commitments, expose usage and capacity data, and provides acceptance/rejection feedback. |
| 6 | + |
| 7 | +## File Structure |
| 8 | + |
| 9 | +```text |
| 10 | +internal/scheduling/reservations/commitments/ |
| 11 | +├── config.go # Configuration (intervals, API flags, secrets) |
| 12 | +├── controller.go # Reconciliation of reservations |
| 13 | +├── syncer.go # Periodic sync task with Limes, ensures local state matches Limes' commitments |
| 14 | +├── reservation_manager.go # Reservation CRUD operations |
| 15 | +├── api.go # HTTP API initialization |
| 16 | +├── api_change_commitments.go # Handle commitment changes from Limes and updates local reservations accordingly |
| 17 | +├── api_report_usage.go # Report VM usage per project, accounting to commitments or PAYG |
| 18 | +├── api_report_capacity.go # Report capacity per AZ |
| 19 | +├── api_info.go # Readiness endpoint with versioning (of underlying flavor group configuration) |
| 20 | +├── capacity.go # Capacity calculation from Hypervisor CRDs |
| 21 | +├── usage.go # VM-to-commitment assignment logic |
| 22 | +├── flavor_group_eligibility.go # Validates VMs belong to correct flavor groups |
| 23 | +└── state.go # Commitment state helper functions |
| 24 | +``` |
| 25 | + |
| 26 | +## Operations |
| 27 | + |
| 28 | +### Configuration |
| 29 | + |
| 30 | +| Helm Value | Description | |
| 31 | +|------------|-------------| |
| 32 | +| `committedResourceEnableChangeCommitmentsAPI` | Enable/disable the change-commitments endpoint | |
| 33 | +| `committedResourceEnableReportUsageAPI` | Enable/disable the usage reporting endpoint | |
| 34 | +| `committedResourceEnableReportCapacityAPI` | Enable/disable the capacity reporting endpoint | |
| 35 | +| `committedResourceRequeueIntervalActive` | How often to revalidate active reservations | |
| 36 | +| `committedResourceRequeueIntervalRetry` | Retry interval when knowledge not ready | |
| 37 | +| `committedResourceChangeAPIWatchReservationsTimeout` | Timeout waiting for reservations to become ready while processing commitment changes via API | |
| 38 | +| `committedResourcePipelineDefault` | Default scheduling pipeline | |
| 39 | +| `committedResourceFlavorGroupPipelines` | Map of flavor group to pipeline name | |
| 40 | +| `committedResourceSyncInterval` | How often the syncer reconciles Limes commitments to Reservation CRDs | |
| 41 | + |
| 42 | +Each API endpoint can be disabled independently. The periodic sync task can be disabled by removing it (`commitments-sync-task`) from the list of enabled tasks in the `cortex-nova` Helm chart. |
| 43 | + |
| 44 | +### Observability |
| 45 | + |
| 46 | +Alerts and metrics are defined in `helm/bundles/cortex-nova/alerts/nova.alerts.yaml`. Key metric prefixes: |
| 47 | +- `cortex_committed_resource_change_api_*` - Change API metrics |
| 48 | +- `cortex_committed_resource_usage_api_*` - Usage API metrics |
| 49 | +- `cortex_committed_resource_capacity_api_*` - Capacity API metrics |
| 50 | + |
| 51 | +## Architecture Overview |
| 52 | + |
| 53 | +```mermaid |
| 54 | +flowchart LR |
| 55 | + subgraph State |
| 56 | + Res[(Reservation CRDs)] |
| 57 | + end |
| 58 | + |
| 59 | + ChangeAPI[Change API] |
| 60 | + UsageAPI[Usage API] |
| 61 | + Syncer[Syncer Task] |
| 62 | + Controller[Controller] |
| 63 | + Scheduler[Scheduler API] |
| 64 | + |
| 65 | + ChangeAPI -->|CRUD| Res |
| 66 | + Syncer -->|CRUD| Res |
| 67 | + UsageAPI -->|read| Res |
| 68 | + Res -->|watch| Controller |
| 69 | + Controller -->|update spec/status| Res |
| 70 | + Controller -->|placement request| Scheduler |
| 71 | +``` |
| 72 | + |
| 73 | +Reservations are managed through the Change API, Syncer Task, and Controller reconciliation. The Usage API provides read-only access to report usage data back to Limes. |
| 74 | + |
| 75 | +### Change-Commitments API |
| 76 | + |
| 77 | +The change-commitments API receives batched commitment changes from Limes. A request can contain multiple commitment changes across different projects and flavor groups. The semantic is **all-or-nothing**: if any commitment in the batch cannot be fulfilled (e.g., insufficient capacity), the entire request is rejected and rolled back. |
| 78 | + |
| 79 | +Cortex performs CRUD operations on local Reservation CRDs to match the new desired state: |
| 80 | +- Creates new reservations for increased commitment amounts |
| 81 | +- Deletes existing reservations |
| 82 | +- Cortex preserves existing reservations that already have VMs allocated when possible |
| 83 | + |
| 84 | +### Syncer Task |
| 85 | + |
| 86 | +The syncer task runs periodically and fetches all commitments from Limes. It syncs the local Reservation CRD state to match Limes' view of commitments. |
| 87 | + |
| 88 | +### Controller (Reconciliation) |
| 89 | + |
| 90 | +The controller watches Reservation CRDs and performs reconciliation: |
| 91 | + |
| 92 | +1. **For new reservations** (no target host assigned): |
| 93 | + - Calls Cortex for scheduling to find a suitable host |
| 94 | + - Assigns the target host and marks the reservation as Ready |
| 95 | + |
| 96 | +2. **For existing reservations** (already have a target host): |
| 97 | + - Validates that allocated VMs are still on the expected host |
| 98 | + - Updates allocations if VMs have migrated or been deleted |
| 99 | + - Requeues for periodic revalidation |
| 100 | + |
| 101 | +### Usage API |
| 102 | + |
| 103 | +This API reports for a given project the total committed resources and usage per flavor group. For each VM, it reports whether the VM accounts to a specific commitment or PAYG. This assignment is deterministic and may differ from the actual Cortex internal assignment used for scheduling. |
| 104 | + |
0 commit comments