OCPBUGS-57348: add MCO operator manifest for boot image management by patrickdillon · Pull Request #9783 · openshift/installer

patrickdillon · 2025-06-12T15:50:54Z

Adds manifest generation for MCO configuration. Currently the manifest is only generated when compute node custom boot images are specified, in order to disable MCO management of those boot images.

For example, aws install config with compute ami set:

compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    aws:
      amiID: ami-08f1807771f4e468b
  replicas: 3

Produces the following manifest

cat test/manifests/cluster-mco-02-config.yml

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
spec:
  logLevel: Normal
  operatorLogLevel: Normal
  managedBootImages:
    machineManagers:
      - resource: machinesets
        apiGroup: machine.openshift.io
        selection:
          mode: None

openshift-ci-robot · 2025-06-12T15:51:05Z

@patrickdillon: This pull request references Jira Issue OCPBUGS-57348, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @gpei

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Adds manifest generation for MCO configuration. Currently the manifest is only generated when custom boot images are specified, in order to disable MCO management of those boot images.

For example, aws install config with control plane ami set:

controlPlane:
 architecture: amd64
 hyperthreading: Enabled
 name: master
 platform:
   aws:
     amiID: ami-xxxxxxx
 replicas: 3

Produces the following manifest

cat test/manifests/cluster-mco-02-config.yml

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
 creationTimestamp: null
 name: cluster
 namespace: openshift-machine-config-operator
spec:
 forceRedeploymentReason: ""
 managedBootImages:
   machineManagers:
   - apiGroup: machine.openshift.io
     resource: machinesets
     selection:
       mode: Partial
       partial:
         machineResourceSelector:
           matchLabels:
             machine.openshift.io/cluster-api-machine-role: worker
 managementState: ""
 nodeDisruptionPolicy:
   files: null
   sshkey:
     actions: null
   units: null
 observedConfig: null
 unsupportedConfigOverrides: null
status:
 managedBootImagesStatus:
   machineManagers: null
 nodeDisruptionPolicyStatus:
   clusterPolicies:
     sshkey:
       actions: null

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pkg/asset/manifests/mco.go

openshift-ci-robot · 2025-06-15T20:35:13Z

@patrickdillon: This pull request references Jira Issue OCPBUGS-57348, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @gpei

Details

In response to this:

Adds manifest generation for MCO configuration. Currently the manifest is only generated when compute node custom boot images are specified, in order to disable MCO management of those boot images.

For example, aws install config with compute ami set:
compute:
- architecture: amd64
 hyperthreading: Enabled
 name: worker
 platform:
   aws:
     amiID: ami-08f1807771f4e468b
 replicas: 3
Produces the following manifest
cat test/manifests/cluster-mco-02-config.yml
apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
 name: cluster
spec:
 logLevel: Normal
 operatorLogLevel: Normal
 managedBootImages:
   machineManagers:
     - resource: machinesets
       apiGroup: machine.openshift.io
       selection:
         mode: None

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

patrickdillon · 2025-06-15T20:41:34Z

Ok I have reworked the PR, because cluster-bootstrap was not able to successfully apply the manifest created in the original implementation. I confirmed in testing what @djoshy suspected: when golang serializes the empty values of the mco configuration, it leaves a lot of empty values, and API server validation fails on them.

So in order to create the minimally specified manifest (in the updated description above), I switched the implementation to use a go template file. I have added this as a fixup commit ATM and will squash it later, once we align on this approach.

patrickdillon · 2025-06-15T23:45:41Z

I have the lint fixes committed locally but will wait to push them until I get more feedback. This is a substantial change from the first iteration, so I wanted to get it up for review. I also still need to rework the mco tests; which I think I can mostly reuse.

Local testing shows the configuration I would expect in the cluster:

% oc describe machineconfiguration cluster 
Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  operator.openshift.io/v1
Kind:         MachineConfiguration
Metadata:
  Creation Timestamp:  2025-06-15T23:39:28Z
  Generation:          1
  Resource Version:    2513
  UID:                 40bfcf3b-fd93-4066-9006-f9e37cc0808e
Spec:
  Log Level:  Normal
  Managed Boot Images:
    Machine Managers:
      API Group:  machine.openshift.io
      Resource:   machinesets
      Selection:
        Mode:          None
  Operator Log Level:  Normal
Events:                <none>

djoshy · 2025-06-16T13:28:22Z

This approach looks good to me from the MCO POV. Sorry about the operator API weirdness, it's caused us some issues too(we have to do server side apply in same cases 😓 ).

Adds manifest generation for MCO configuration. Currently the manifest is only generated when custom boot images are specified, in order to disable MCO management of those boot images. The manifest generation uses a golang template as testing revealed that API server validation would not permit the manifests generated from serializing the golang structs, which would be more consistent with how we generate manifests for other openshift operators. As golang will populate the zero value for any non-pointer struct this triggered validation, where the API server expected certain required fields for these zero-value structs. Using a template allows us to bypass this problem. Fixes OCPBUGS-57348

wking · 2025-06-16T19:50:29Z

pkg/asset/templates/content/manifests/cluster-mco-config.go

+)
+
+const (
+	mcoConfigTemplateFileName = "cluster-mco-02-config.yaml.template"


Where does the eventual generated manifest's filename sort vs. the installer-generated MachineSet manifests? Do we want a 90_ prefix or something to ensure we sort the MachineConfiguration first, so the MCO knows to leave MachineSets alone before the MachineSets are created in the cluster?

Good question. The resulting filename is indeed cluster-mco-02-config.yaml and therefore sorts after the 99_ prefixed machinesets. I am running a local install to gather the bootkube logs and see if I can get a sense of when it is created.

My only hesitation is that none of the other cluster operator manifests have a numerical prefix... but I don't find that convincing.

Renamed in dd2b388

djoshy · 2025-06-16T19:56:30Z

/lgtm

The MCO manifests and tests seems sane to me. I'm not as familiar with the manifest generation flow, so I will defer that to the installer folks. Thanks for putting this together!

patrickdillon · 2025-06-17T16:06:41Z

/retest-required

openshift-ci · 2025-06-17T16:34:08Z

@patrickdillon: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`dd2b388`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

patrickdillon · 2025-06-18T14:35:32Z

#9793 has merged for unit tests

/retest-required

patrickdillon · 2025-06-18T14:37:01Z

/label tide/merge-method-squash
for fixups

patrickdillon · 2025-06-18T14:37:11Z

@djoshy would you mind bumping lgtm?

patrickdillon · 2025-06-18T14:37:23Z

/cherry-pick release-4.19

openshift-cherrypick-robot · 2025-06-18T14:37:26Z

@patrickdillon: once the present PR merges, I will cherry-pick it on top of release-4.19 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

djoshy · 2025-06-18T14:38:24Z

/lgtm

patrickdillon · 2025-06-19T11:34:41Z

/approve

openshift-ci · 2025-06-19T11:35:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [patrickdillon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gpei · 2025-06-19T12:02:43Z

/label qe-approved

openshift-ci-robot · 2025-06-19T15:09:59Z

@patrickdillon: Jira Issue OCPBUGS-57348: All pull requests linked via external trackers have merged:

openshift/installer#9783

Jira Issue OCPBUGS-57348 has been moved to the MODIFIED state.

Details

In response to this:

Adds manifest generation for MCO configuration. Currently the manifest is only generated when compute node custom boot images are specified, in order to disable MCO management of those boot images.

For example, aws install config with compute ami set:
compute:
- architecture: amd64
 hyperthreading: Enabled
 name: worker
 platform:
   aws:
     amiID: ami-08f1807771f4e468b
 replicas: 3
Produces the following manifest
cat test/manifests/cluster-mco-02-config.yml
apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
 name: cluster
spec:
 logLevel: Normal
 operatorLogLevel: Normal
 managedBootImages:
   machineManagers:
     - resource: machinesets
       apiGroup: machine.openshift.io
       selection:
         mode: None

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot · 2025-06-19T15:10:56Z

@patrickdillon: new pull request created: #9797

Details

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2025-06-19T17:16:54Z

[ART PR BUILD NOTIFIER]

Distgit: ose-installer
This PR has been included in build ose-installer-container-v4.20.0-202506191613.p0.gb241c4e.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-06-19T17:22:54Z

[ART PR BUILD NOTIFIER]

Distgit: ose-baremetal-installer
This PR has been included in build ose-baremetal-installer-container-v4.20.0-202506191613.p0.gb241c4e.assembly.stream.el9.
All builds following this will include this PR.

openshift-bot · 2025-06-19T18:04:56Z

[ART PR BUILD NOTIFIER]

Distgit: ose-installer-artifacts
This PR has been included in build ose-installer-artifacts-container-v4.20.0-202506191613.p0.gb241c4e.assembly.stream.el9.
All builds following this will include this PR.

If custom boot images are specified, the installer generates an MCO manifest to disable MCO boot image management [0]. On AWS, previously edge compute machine pool was not considered for custom AMI. This commit ensures custom AMI is considered in all compute machine pools. References: [0] openshift#9783

openshift-ci bot requested review from andfasano, gpei and rwsu June 12, 2025 15:51

patrickdillon commented Jun 12, 2025

View reviewed changes

pkg/asset/manifests/mco.go Outdated Show resolved Hide resolved

patrickdillon force-pushed the OCPBUGS-57348-bootimg-mgmt branch from 97f0735 to 7f5f88f Compare June 12, 2025 15:54

djoshy reviewed Jun 12, 2025

View reviewed changes

pkg/asset/manifests/mco.go Show resolved Hide resolved

djoshy reviewed Jun 12, 2025

View reviewed changes

pkg/asset/manifests/mco.go Outdated Show resolved Hide resolved

patrickdillon force-pushed the OCPBUGS-57348-bootimg-mgmt branch from 7f5f88f to e04874e Compare June 12, 2025 20:10

wking reviewed Jun 13, 2025

View reviewed changes

pkg/asset/manifests/mco.go Outdated Show resolved Hide resolved

wking reviewed Jun 13, 2025

View reviewed changes

pkg/asset/manifests/mco.go Outdated Show resolved Hide resolved

patrickdillon force-pushed the OCPBUGS-57348-bootimg-mgmt branch from e04874e to 2ccb6ec Compare June 15, 2025 20:33

djoshy mentioned this pull request Jun 16, 2025

OCPBUGS-57348: Cluster manages bootimages despite explicit bootimages in installconfig openshift/machine-config-operator#5120

Closed

patrickdillon force-pushed the OCPBUGS-57348-bootimg-mgmt branch from 2ccb6ec to 8931141 Compare June 16, 2025 18:58

wking reviewed Jun 16, 2025

View reviewed changes

openshift-ci bot assigned djoshy Jun 16, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2025

fixup! pkg/asset/manifests: add MCO operator manifest

808d592

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2025

fixup! pkg/asset/manifests: add MCO operator manifest

dd2b388

openshift-ci bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jun 18, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 18, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 19, 2025

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Jun 19, 2025

openshift-merge-bot bot merged commit b241c4e into openshift:main Jun 19, 2025
17 of 18 checks passed

openshift-cherrypick-robot mentioned this pull request Jun 19, 2025

[release-4.19] OCPBUGS-57796: add MCO operator manifest for boot image management #9797

Merged

tthvo mentioned this pull request Jun 19, 2025

OCPBUGS-57803: MCO: consider custom AMI for edge compute machine pool #9798

Merged

djoshy mentioned this pull request Feb 27, 2026

OCPBUGS-77440: Image in worker machineset is auto-updated when user specifies the osImage #10341

Open

Conversation

patrickdillon commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 15, 2025

Uh oh!

patrickdillon commented Jun 15, 2025

Uh oh!

patrickdillon commented Jun 15, 2025

Uh oh!

djoshy commented Jun 16, 2025

Uh oh!

wking Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

patrickdillon Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

patrickdillon Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

djoshy commented Jun 16, 2025

Uh oh!

patrickdillon commented Jun 17, 2025

Uh oh!

openshift-ci bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrickdillon commented Jun 18, 2025

Uh oh!

patrickdillon commented Jun 18, 2025

Uh oh!

patrickdillon commented Jun 18, 2025

Uh oh!

patrickdillon commented Jun 18, 2025

Uh oh!

openshift-cherrypick-robot commented Jun 18, 2025

Uh oh!

djoshy commented Jun 18, 2025

Uh oh!

patrickdillon commented Jun 19, 2025

Uh oh!

openshift-ci bot commented Jun 19, 2025

Uh oh!

gpei commented Jun 19, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 19, 2025

Uh oh!

openshift-cherrypick-robot commented Jun 19, 2025

Uh oh!

openshift-bot commented Jun 19, 2025

Uh oh!

openshift-bot commented Jun 19, 2025

Uh oh!

openshift-bot commented Jun 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

patrickdillon commented Jun 12, 2025 •

edited

Loading

openshift-ci bot commented Jun 17, 2025 •

edited

Loading