Feature Request: Quota Bypass for Rolling Updates in Machine Deployments

[ronissac88]

What happened?

When implementing Project Quotas, we're running into a problem where rolling restarts of machine deployments fail due to quota validation. The issue occurs because during a rolling update, the machine controller temporarily creates new machines before deleting old ones, causing a brief period where the total resource usage exceeds the quota limits.

Error Message:

failed to sync Machineset replicas: admission webhook "machines.cluster.k8c.io" denied the request: requested CPU "6" would exceed current quota (quota/used "20"/"18")

Root Cause:

• Machine controller uses a rolling update strategy with maxSurge to maintain availability
• New machines are created before old machines are deleted
• Quota validation happens at admission webhook level before the old machines are removed
• This creates a temporary overage that fails quota validation

Expected behavior

The machine controller should be able to perform rolling updates without failing quota validation when:

1. The replica count of the MachineDeployment is not being increased (i.e., it's a rolling restart/update, not a scale-up)
2. The temporary overage is within the maxSurge limits of the rolling update strategy
3. The total resource usage will return to normal levels once the rolling update completes

Proposed Solution:

• Add a quota bypass mechanism for machines created during rolling updates
• Use an annotation kubermatic.io/bypass-quota-validation=true on machines created during rolling updates
• Modify the admission webhook to detect quota-related errors and bypass validation when the annotation is present
• Ensure the bypass only applies to quota-related errors, not other validation failures

How to reproduce the issue?

1. Setup a MachineDeployment with quota limits:

   apiVersion: "cluster.k8s.io/v1alpha1"
   kind: MachineDeployment
   metadata:
     name: test-deployment
   spec:
     replicas: 3
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxSurge: 1
         maxUnavailable: 1
     template:
       spec:
         providerSpec:
           value:
             cloudProvider: "azure"
             cloudProviderSpec:
               vmSize: "Standard_D2s_v3"  # 2 vCPUs per machine

2. Set up resource quotas:

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: compute-quota
   spec:
     hard:
       requests.cpu: "6"  # Total of 3 machines * 2 CPUs = 6 CPUs

3. Trigger a rolling update by changing the machine template (e.g., update the OS image or kubelet version)

4. Observe the failure when the machine controller tries to create the 4th machine (3 existing + 1 new for rolling update)

Additional details

Current Flow:

MachineDeployment Update → MachineSet Scaling → Machine Creation → Quota Check → FAIL

Proposed Flow:

MachineDeployment Update → MachineSet Scaling → Machine Creation (with bypass annotation) → Quota Check → BYPASS → SUCCESS

[csengerszabo]

Internal reference: 8382

[kron4eg]

Should we add a new webhook to remove the kubermatic.io/bypass-quota-validation=true annotation? 😂 I'm pretty sure running at 100% of the quota will make troubles at any system. My "solution" would be to increase the quota, or decrease the replicas -1 before the rollout.

[steled]

@kron4eg I think a manual approach maybe works for 3 projects but not for >10.
To manually update temporarily the project quota for over 10 projects doesn't feel right.
Also decreasing the replicas -1 does not feel very well.

It would be great to have a programmatic solution that new nodes during a md rolling restart are not counted into the project quota.

[kron4eg]

@steled I meant that userclusters / projects in general should not run at the 100% quota. Or if they need more, then the quota is not correctly set.

[steled]

For some customers, we do not sell on demand, they just buy a specific node count.
And I think it should be possible to run on 100% quota but still be able to do a rolling restart of machine deployment wihtout temporarily updating the project quota.

I'm also running into a problem that I can't restart machine deployment even when the project quota has a buffer of 1 additional node.
I have a machine deployment with 3 nodes (2 CPUs, 2048 MB RAM) and a project quota of 8 CPUs, 9 GB RAM.

[kron4eg]

apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineDeployment
metadata:
  name: test-deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1

such a rollingUpdate config will replace Machines one by one, first deleting the Machine (and draining the corresponding Node) and then creating the new Machine, once it can bypass the Quota.

EDIT 1:
assuming running at 100% quota capacity

[steled]

Unfortunately does not work correctly.
I updated a test cluster as you described:

apiVersion: cluster.k8s.io/v1alpha1
kind: MachineDeployment
metadata:
  name: adoring-hypatia
spec:
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

I than set a project quota (nearly 100%):

And started a rolling update (Restart Machine Deployment via web ui).
For some tries it worked but currently it's failing:

[kron4eg]

Are you sure there is no other ResourceQuota that targets the current Project?