Single-version generated CRDs cannot be installed

[muvaf]

Probably I missed something but I can't get a single-version CRD to be installed on my cluster.

This is a project that has 2 kinds generated by kubebuilder v2: https://github.com/muvaf/kubebuilder-init

Check it out and run the following:

GO111MODULE=on controller-gen crd paths=./api/... output:stdout | kubectl create -f -

You'll get:

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "examplekinds.mygroup.muvaf.github.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc00396d000), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version schemas may not all be set to identical values (top-level validation should be used instead)
Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "otherexamplekinds.mygroup.muvaf.github.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc00386f970), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version schemas may not all be set to identical values (top-level validation should be used instead)

I see that it hits the following error on https://github.com/kubernetes/apiextensions-apiserver/blob/master/pkg/apis/apiextensions/validation/validation.go#L453 which signals it may be a bug on the api-server but I'm not really familiar with that codebase. Any help is appreciated!

Environment:

$ minikube version
minikube version: v1.2.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

[muvaf]

Just tested it with k8s 1.14.5. The creation command succeeds but all validations are omitted.

The yaml file I use for creation:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: null
  name: eksclusters.compute.aws.crossplane.io
spec:
  group: compute.aws.crossplane.io
  names:
    kind: EKSCluster
    plural: eksclusters
  scope: ""
  version: v1alpha1
  versions:
  - additionalPrinterColumns:
    - JSONPath: .status.bindingPhase
      name: STATUS
      type: string
    - JSONPath: .status.state
      name: STATE
      type: string
    - JSONPath: .status.clusterName
      name: CLUSTER-NAME
      type: string
    - JSONPath: .status.endpoint
      name: ENDPOINT
      type: string
    - JSONPath: .spec.classRef.name
      name: CLUSTER-CLASS
      type: string
    - JSONPath: .spec.location
      name: LOCATION
      type: string
    - JSONPath: .spec.reclaimPolicy
      name: RECLAIM-POLICY
      type: string
    - JSONPath: .metadata.creationTimestamp
      name: AGE
      type: date
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: EKSCluster is the Schema for the resources API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: EKSClusterSpec specifies the configuration for an EKS cluster.
            properties:
              claimRef:
                description: ObjectReference contains enough information to let you
                  inspect or modify the referred object.
                properties:
                  apiVersion:
                    description: API version of the referent.
                    type: string
                  fieldPath:
                    description: 'If referring to a piece of an object instead of
                      an entire object, this string should contain a valid JSON/Go
                      field access statement, such as desiredState.manifest.containers[2].
                      For example, if the object reference is to a container within
                      a pod, this would take on a value like: "spec.containers{name}"
                      (where "name" refers to the name of the container that triggered
                      the event) or if no container name is specified "spec.containers[2]"
                      (container with index 2 in this pod). This syntax is chosen
                      only to have some well-defined way of referencing a part of
                      an object. TODO: this design is not final and this field is
                      subject to change in the future.'
                    type: string
                  kind:
                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
                    type: string
                  name:
                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
                    type: string
                  namespace:
                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
                    type: string
                  resourceVersion:
                    description: 'Specific resourceVersion to which this reference
                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
                    type: string
                  uid:
                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
                    type: string
                type: object
              classRef:
                description: ObjectReference contains enough information to let you
                  inspect or modify the referred object.
                properties:
                  apiVersion:
                    description: API version of the referent.
                    type: string
                  fieldPath:
                    description: 'If referring to a piece of an object instead of
                      an entire object, this string should contain a valid JSON/Go
                      field access statement, such as desiredState.manifest.containers[2].
                      For example, if the object reference is to a container within
                      a pod, this would take on a value like: "spec.containers{name}"
                      (where "name" refers to the name of the container that triggered
                      the event) or if no container name is specified "spec.containers[2]"
                      (container with index 2 in this pod). This syntax is chosen
                      only to have some well-defined way of referencing a part of
                      an object. TODO: this design is not final and this field is
                      subject to change in the future.'
                    type: string
                  kind:
                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
                    type: string
                  name:
                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
                    type: string
                  namespace:
                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
                    type: string
                  resourceVersion:
                    description: 'Specific resourceVersion to which this reference
                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
                    type: string
                  uid:
                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
                    type: string
                type: object
              cliInput:
                description: CLIInput --cli-input-json  (string) Performs service
                  operation based on the JSON string provided. The JSON string follows
                  the format provided by  --gen- erate-cli-skeleton.  If  other  arguments  are  provided
                  on the command line, the CLI values will override the JSON-provided
                  values. It is  not possible to pass arbitrary binary values using
                  a JSON-provided value as the string will be taken literally.
                type: string
              clientRequestToken:
                description: ClientRequestToken --client-request-token (string) Unique,
                  case-sensitive identifier you provide to ensure the  idempo- tency
                  of the request.
                type: string
              clusterVersion:
                description: ClusterVersion --kubernetes-version (string) The desired
                  Kubernetes version for your cluster. If you do not spec- ify a value
                  here, the latest version  available  in  Amazon  EKS  is used.
                type: string
              generateCLISkeleton:
                description: GenerateCLISkeleton --generate-cli-skeleton (string)
                  Prints a  JSON  skeleton  to  standard output without sending an
                  API request. If provided with no value or the value input, prints
                  a sample input JSON that can be used as an argument for  --cli-input-json.  If
                  provided with the value output, it validates the command inputs
                  and returns a sample output JSON for that command.
                type: string
              mapRoles:
                description: "MapRoles - each item maps an aws role to kubernetes
                  groups Default role that allows nodes access to communicate with
                  master is autogenerated when a node pool comes online. Additional
                  map roles configuration are documented in the link below https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/master/README.md
                  \n # map arn:aws:iam::000000000000:role/KubernetesAdmin to cluster
                  admin - rolearn: arn:aws:iam::000000000000:role/KubernetesAdmin
                  \  username: kubernetes-admin    groups:      - system:masters"
                items:
                  description: 'MapRole maps an aws role to kubernetes groups see:
                    https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
                    https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/master/README.md'
                  properties:
                    groups:
                      items:
                        type: string
                      type: array
                    rolearn:
                      type: string
                    username:
                      type: string
                  required:
                  - groups
                  - rolearn
                  - username
                  type: object
                type: array
              mapUsers:
                description: "MapUsers - each item maps an aws users to kubernetes
                  groups Additional map users configuration options are documented
                  in the link below https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/master/README.md
                  \n # map user IAM user Alice in 000000000000 to user \"alice\" in
                  group \"system:masters\" - userarn: arn:aws:iam::000000000000:user/Alice
                  \ username: alice  groups:    - system:masters"
                items:
                  description: 'MapUser maps an aws user to kubernetes groups see:
                    https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
                    https://github.com/kubernetes-sigs/aws-iam-authenticator/blob/master/README.md'
                  properties:
                    groups:
                      items:
                        type: string
                      type: array
                    userarn:
                      type: string
                    username:
                      type: string
                  required:
                  - groups
                  - userarn
                  - username
                  type: object
                type: array
              providerRef:
                description: ObjectReference contains enough information to let you
                  inspect or modify the referred object.
                properties:
                  apiVersion:
                    description: API version of the referent.
                    type: string
                  fieldPath:
                    description: 'If referring to a piece of an object instead of
                      an entire object, this string should contain a valid JSON/Go
                      field access statement, such as desiredState.manifest.containers[2].
                      For example, if the object reference is to a container within
                      a pod, this would take on a value like: "spec.containers{name}"
                      (where "name" refers to the name of the container that triggered
                      the event) or if no container name is specified "spec.containers[2]"
                      (container with index 2 in this pod). This syntax is chosen
                      only to have some well-defined way of referencing a part of
                      an object. TODO: this design is not final and this field is
                      subject to change in the future.'
                    type: string
                  kind:
                    description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
                    type: string
                  name:
                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names'
                    type: string
                  namespace:
                    description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/'
                    type: string
                  resourceVersion:
                    description: 'Specific resourceVersion to which this reference
                      is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency'
                    type: string
                  uid:
                    description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids'
                    type: string
                type: object
              reclaimPolicy:
                description: A ReclaimPolicy determines what should happen to managed
                  resources when their bound resource claims are deleted.
                type: string
              region:
                description: Region for EKS Cluster
                enum:
                - us-west-2
                - us-east-1
                - eu-west-1
                type: string
              roleARN:
                description: 'RoleARN --role-arn The Amazon Resource Name (ARN) of
                  the IAM role that provides permis- sions for Amazon EKS to make
                  calls to other AWS  API  operations  on your  behalf.  For more
                  information, see Amazon EKS Service IAM Role in the * Amazon EKS
                  User Guide * . TODO: we could simplify this to roleName.'
                type: string
              securityGroupIds:
                description: 'SecurityGroupIds Syntax: securityGroupIds=string,string,'
                items:
                  type: string
                type: array
              subnetIds:
                description: 'SubnetIds Syntax: subnetIds=string,string,'
                items:
                  type: string
                type: array
              vpcId:
                description: "ResourcesVPCConfig --resources-vpc-config (structure)
                  The VPC subnets and security groups  used  by  the  cluster  control
                  plane.  Amazon  EKS VPC resources have specific requirements to
                  work properly with Kubernetes. For more information, see Cluster
                  VPC Con- siderations  and Cluster Security Group Considerations
                  in the Amazon EKS User Guide . You must specify at  least  two  subnets.
                  \ You  may specify  up  to  5  security groups, but we recommend
                  that you use a dedicated security group for your cluster control
                  plane. \n VpcID of EKS cluster"
                type: string
              workerNodes:
                description: WorkerNodes configuration for cloudformation
                properties:
                  bootstrapArguments:
                    description: 'BootstrapArguments Arguments to pass to the bootstrap
                      script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami
                      Default: ""'
                    type: string
                  clusterControlPlaneSecurityGroup:
                    description: ClusterControlPlaneSecurityGroup The security group
                      of the cluster control plane.
                    type: string
                  keyName:
                    description: KeyName The EC2 Key Pair to allow SSH access to the
                      instances
                    type: string
                  nodeAutoScalingGroupMaxSize:
                    description: 'NodeAutoScalingGroupMaxSize Maximum size of Node
                      Group ASG. Default: 3'
                    type: integer
                  nodeAutoScalingGroupMinSize:
                    description: NodeAutoScalingGroupMinSize Minimum size of Node
                      Group ASG. default 1
                    type: integer
                  nodeGroupName:
                    description: NodeGroupName Unique identifier for the Node Group.
                    type: string
                  nodeImageId:
                    description: NodeImageId The EC2 Key Pair to allow SSH access
                      to the instances defaults to region standard AMI
                    type: string
                  nodeInstanceType:
                    description: NodeInstanceType EC2 instance type for the node instances
                    enum:
                    - t2.small
                    - t2.medium
                    - t2.large
                    - t2.xlarge
                    - t2.2xlarge
                    - t3.nano
                    - t3.micro
                    - t3.small
                    - t3.medium
                    - t3.large
                    - t3.xlarge
                    - t3.2xlarge
                    - m3.medium
                    - m3.large
                    - m3.xlarge
                    - m3.2xlarge
                    - m4.large
                    - m4.xlarge
                    - m4.2xlarge
                    - m4.4xlarge
                    - m4.10xlarge
                    - m5.large
                    - m5.xlarge
                    - m5.2xlarge
                    - m5.4xlarge
                    - m5.12xlarge
                    - m5.24xlarge
                    - c4.large
                    - c4.xlarge
                    - c4.2xlarge
                    - c4.4xlarge
                    - c4.8xlarge
                    - c5.large
                    - c5.xlarge
                    - c5.2xlarge
                    - c5.4xlarge
                    - c5.9xlarge
                    - c5.18xlarge
                    - i3.large
                    - i3.xlarge
                    - i3.2xlarge
                    - i3.4xlarge
                    - i3.8xlarge
                    - i3.16xlarge
                    - r3.xlarge
                    - r3.2xlarge
                    - r3.4xlarge
                    - r3.8xlarge
                    - r4.large
                    - r4.xlarge
                    - r4.2xlarge
                    - r4.4xlarge
                    - r4.8xlarge
                    - r4.16xlarge
                    - x1.16xlarge
                    - x1.32xlarge
                    - p2.xlarge
                    - p2.8xlarge
                    - p2.16xlarge
                    - p3.2xlarge
                    - p3.8xlarge
                    - p3.16xlarge
                    - r5.large
                    - r5.xlarge
                    - r5.2xlarge
                    - r5.4xlarge
                    - r5.12xlarge
                    - r5.24xlarge
                    - r5d.large
                    - r5d.xlarge
                    - r5d.2xlarge
                    - r5d.4xlarge
                    - r5d.12xlarge
                    - r5d.24xlarge
                    - z1d.large
                    - z1d.xlarge
                    - z1d.2xlarge
                    - z1d.3xlarge
                    - z1d.6xlarge
                    - z1d.12xlarge
                    type: string
                  nodeVolumeSize:
                    description: 'NodeVolumeSize Node volume size in GB Default: 20'
                    type: integer
                required:
                - keyName
                - nodeInstanceType
                type: object
              writeConnectionSecretToRef:
                description: LocalObjectReference contains enough information to let
                  you locate the referenced object inside the same namespace.
                properties:
                  name:
                    description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                      TODO: Add other useful fields. apiVersion, kind, uid?'
                    type: string
                type: object
            required:
            - providerRef
            - region
            - roleARN
            - securityGroupIds
            - subnetIds
            - vpcId
            - workerNodes
            type: object
          status:
            description: EKSClusterStatus schema of the status of eks cluster
            properties:
              bindingPhase:
                description: Phase represents the binding phase of the resource.
                enum:
                - Unbindable
                - Unbound
                - Bound
                type: string
              cloudformationStackId:
                description: CloudFormationStackID Stack-id
                type: string
              conditions:
                description: Conditions of the managed resource.
                items:
                  description: A Condition that may apply to a managed resource.
                  properties:
                    lastTransitionTime:
                      description: LastTransitionTime is the last time this condition
                        transitioned from one status to another.
                      format: date-time
                      type: string
                    message:
                      description: A Message containing details about this condition's
                        last transition from one status to another, if any.
                      type: string
                    reason:
                      description: A Reason for this condition's last transition from
                        one status to another.
                      type: string
                    status:
                      description: Status of this condition; is it currently True,
                        False, or Unknown?
                      type: string
                    type:
                      description: Type of this condition. At most one of each condition
                        type may apply to a managed resource at any point in time.
                      type: string
                  required:
                  - lastTransitionTime
                  - reason
                  - status
                  - type
                  type: object
                type: array
              endpoint:
                description: Endpoint for cluster
                type: string
              resourceName:
                description: ClusterName identifier
                type: string
              state:
                description: State of the cluster (see status constants above)
                type: string
            type: object
        type: object
    served: true
    storage: true
    subresources: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

What I get when I run kubectl get crd eksclusters.compute.aws.crossplane.io -o yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-08-09T09:47:49Z"
  generation: 1
  name: eksclusters.compute.aws.crossplane.io
  resourceVersion: "1038"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/eksclusters.compute.aws.crossplane.io
  uid: bf9dec33-ba8a-11e9-8d71-4ee85ae2447d
spec:
  conversion:
    strategy: None
  group: compute.aws.crossplane.io
  names:
    kind: EKSCluster
    listKind: EKSClusterList
    plural: eksclusters
    singular: ekscluster
  scope: Namespaced
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true
status:
  acceptedNames:
    kind: EKSCluster
    listKind: EKSClusterList
    plural: eksclusters
    singular: ekscluster
  conditions:
  - lastTransitionTime: "2019-08-09T09:47:49Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1alpha1

[zhsj]

I think it's been fixed by #322

[pires]

I'm running into this as well if I set crdVersions=v1. Support for CRD v1 was introduced in #356 which I'm thinking introduced a regression that indeed had been fixed by #322.

If I don't set crdVersions=v1, it just works.

I am using controller-gen v0.2.4 and kubebuilder v2.2.0.

[pires]

Interestingly enough, i see this problem only when testing my controller with envtest, not with kubectl apply --validate -f.

kubectl is v1.16.3 and I'm running kind v0.6.0 (Kubernetes v1.16.3). Tests are run with USE_EXISTING_CLUSTER=true.

[pires]

Here's a sample with crd:crdVersions=v1,trivialVersions=false: https://github.com/pires/kubebuilder-issue/blob/master/config/crd/bases/example.pires.github.com_stuffs.yaml#L18-L23.

[pires]

@muvaf my workaround is to not install the CRDs with envtest and instead do it with kubectl before running the tests. If you built your project with kubebuilder, it should be as easy as adding in the Makefile the goal install as a dependency of goal tests.

With that said, I'm thinking the problem lies in envtest, since it fails in both the default mode (running its own etcd and kube-apiserver) and relying on an existing cluster (USE_EXISTING_CLUSTER=true).

[muvaf]

@pires Thanks for the suggestion! I haven't run into this issue since we haven't needed to migrate to CRD v1 version. Is there a technical reason that you want to pass crdVersions=v1 explicitly? As far as I see, api-server would happily manage the underlying conversions.

[pires]

You need this in order to enable the generation of certain markers, like kubebuilder:validation:default.

[pires]

As per @DirectXMan12 comment, it seems I have no issue at all and this is expected #322 (comment) when doing CRD v1 alone.