Operator Lifecycle Manager – CustomResourceDefinitions

Docs: ClusterServiceVersion

Mon, 01 Jan 0001 00:00:00 +0000

A ClusterServiceVersion (CSV) represents a particular version of a running operator on a cluster. It includes metadata such as name, description, version, repository link, labels, icon, etc. It declares owned/required CRDs, cluster requirements, and install strategy that tells OLM how to create required resources and set up the operator as a deployment.

OLM requires you to provide metadata about your operator in order to ensure that it can be kept running safely on a cluster, and to provide information about how updates should be applied as you publish new versions of your operator.

This is very similar to packaging software for a traditional operating system - think of the packaging step for OLM as the stage at which you make your rpm, deb, or apk bundle.

Writing your Operator Manifests

OLM uses an api called ClusterServiceVersion (CSV) to describe a single instance of a version of an operator. This is the main entrypoint for packaging an operator for OLM.

There are two important ways to think about the CSV:

Like an rpm or deb, it collects metadata about the operator that is required to install it onto the cluster.
Like a Deployment that can stamp out Pods from a template, the ClusterServiceVersion describes a template for the operator Deployment and can stamp them out.

This is all in service of ensuring that when a user installs an operator from OLM, they can understand what changes are happening to the cluster, and OLM can ensure that installing the operator is a safe operation.

Example ClusterServiceVersion

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
 annotations:
 name: memcached-operator.v0.10.0
spec:
 # metadata
 description: This is an operator for memcached.
 displayName: Memcached Operator
 keywords:
 - memcached
 - app
 maintainers:
 - email: corp@example.com
 name: Some Corp
 maturity: alpha
 provider:
 name: Example
 url: www.example.com
 version: 0.10.0
 minKubeVersion: 1.16.0

 # operator scope
 installModes:
 - supported: true
 type: OwnNamespace
 - supported: true
 type: SingleNamespace
 - supported: false
 type: MultiNamespace
 - supported: true
 type: AllNamespaces

 # installation
 install:
 # strategy indicates what type of deployment artifacts are used
 strategy: deployment
 # spec for the deployment strategy is a list of deployment specs and required permissions - similar to a pod template used in a deployment
 spec:
 permissions:
 - serviceAccountName: memcached-operator
 rules:
 - apiGroups:
 - ""
 resources:
 - pods
 verbs:
 - '*'
 # the rest of the rules
 # permissions required at the cluster scope
 clusterPermissions:
 - serviceAccountName: memcached-operator
 rules:
 - apiGroups:
 - ""
 resources:
 - serviceaccounts
 verbs:
 - '*'
 # the rest of the rules
 deployments:
 - name: memcached-operator
 spec:
 replicas: 1
 # the rest of a deployment spec

 # apis provided by the operator
 customresourcedefinitions:
 owned:
 # a list of CRDs that this operator owns
 # name is the metadata.name of the CRD (which is of the form <plural>.<group>)
 - name: memcacheds.cache.example.com
 # version is the spec.versions[].name value defined in the CRD
 version: v1alpha1
 # kind is the CamelCased singular value defined in spec.names.kind of the CRD.
 kind: Memcached
 required:
 # a list of CRDs that this operator requires
 # see field descriptions above
 - name: others.example.com
 version: v1alpha1
 kind: Other

Release Field

The release field is an optional field in the CSV spec that specifies the packaging version of an operator. It allows operator authors to distinguish between different builds of the same operator version.

When to Use the Release Field

Use the release field when you need to make changes to the CSV that don’t affect the operator’s functionality, such as:

Fixing typos in descriptions
Adding or amending annotations or labels
Amending examples or documentation
Producing different builds for different environments

Format and Validation

The release field must satisfy the following criteria:

Composed of dot-separated identifiers containing only alphanumerics and hyphens
Maximum length of 20 characters
Cannot contain build metadata (the + character and everything after it)

Examples: 1, alpha, alpha.1, beta-1, rc.2.0

Example

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
 name: memcached-operator-v0.10.0-1
spec:
 version: 0.10.0
 release: "1" # Optional packaging version
 displayName: Memcached Operator
 description: This is an operator for memcached.
 # ... rest of CSV spec

Bundle Naming Convention with Release

When a release field is specified, the bundle name must follow the convention:

<package-name>-v<version>-<release>

Examples:

memcached-operator-v0.10.0-1
memcached-operator-v0.10.0-alpha.1
memcached-operator-v0.10.0-rc.2

Version Ordering

Bundles are ordered using a composite version that considers both version and release:

Bundles are first ordered by semantic version
For bundles with the same version, those with a release are ordered before those without a release
Bundles with the same version and both having releases are ordered from highest to lowest release version

graph TD A[Bundle Comparison] --> B{Same Version?} B -->|No| C[Order by Version] B -->|Yes| D{Both have Release?} D -->|No| E{Has Release?} D -->|Yes| F[Order by Release] E -->|Bundle A| G[A before B] E -->|Bundle B| H[B before A] style A fill:#e1f5ff style C fill:#c3e6cb style F fill:#c3e6cb style G fill:#c3e6cb style H fill:#c3e6cb

Example versions in ascending order:

memcached-operator.v0.9.0 # version 0.9.0, no release
memcached-operator.v0.10.0 # version 0.10.0, no release
memcached-operator-v0.10.0-1 # version 0.10.0, release "1"
memcached-operator-v0.10.0-2 # version 0.10.0, release "2"
memcached-operator-v0.10.0-alpha # version 0.10.0, release "alpha"

Build Metadata Restriction

The release field cannot contain build metadata. The following is invalid:

# ❌ INVALID - release cannot contain build metadata (the + character)
version: 0.10.0
release: "1+fffdb0e"

# ✅ VALID - build metadata can be in version when no release is present
version: 0.10.0+fffdb0e
# no release field

Docs: CatalogSource

Mon, 01 Jan 0001 00:00:00 +0000

A CatalogSource represents a store of metadata that OLM can query to discover and install operators and their dependencies.

There are three primary types of CatalogSource(spec.sourceType):

grpc with an image reference: OLM will pull the image and run a pod that has an api endpoint that can be queried for the metadata in the store
grpc with an address field: OLM will attempt to contact the grpc api at the given address. This should not be used in most cases.
internal or configmap: OLM will parse the configmap’s data and spin up a pod that can serve the grpc api over it.

Example - OperatorHub.io

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: operatorhubio-catalog
 namespace: olm
spec:
 sourceType: grpc
 image: quay.io/operatorhubio/catalog:latest
 displayName: Community Operators
 publisher: OperatorHub.io

This defines a CatalogSource for OperatorHub.io’s content. The name of the CatalogSource is used as input to a Subscription, which tells OLM where to look to find a requested operator:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-operator
 namespace: olm
spec:
 channel: stable
 name: my-operator
 source: operatorhubio-catalog

Docs: OperatorCondition

Mon, 01 Jan 0001 00:00:00 +0000

An OperatorCondition is CustomResourceDefinition that creates a communication between OLM and an operator it manages. Operators may write to the Status.Conditions array to modify OLM management the operator.

Here’s an example of an OperatorCondition CustomResource:

apiVersion: operators.coreos.com/v1
kind: OperatorCondition
metadata:
 name: foo-operator
 namespace: operators
spec:
 overrides:
 - type: Upgradeable # Allows the cluster admin to change operator's Upgrade readiness to True
 status: "True"
 reason: "upgradeIsSafe" # optional
 message: "The cluster admin wants to make the operator eligible for an upgrade." # optional
status:
 conditions:
 - type: Upgradeable
 status: "False"
 reason: "migration"
 message: "The operator is performing a migration."
 lastTransitionTime: "2020-08-24T23:15:55Z"

Docs: Subscription

Mon, 01 Jan 0001 00:00:00 +0000

A Subscription represents an intention to install an operator. It is the CustomResource that relate an operator to a CatalogSource. Subscriptions describe which channel of an operator package to subscribe to, and whether to perform updates automatically or manually. If set to automatic, the Subscription ensures OLM will manage and upgrade the operator to ensure the latest version is always running in the cluster.

Here’s an example of a Subscription definition:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-operator
 namespace: operators
spec:
 channel: stable
 name: my-operator
 source: my-catalog
 sourceNamespace: operators

This Subscription object defines the name and namespace of the operator, as well as the catalog from which the operator data can be found. The channel (such as alpha, beta, or stable) helps determine which stream of the operator should be installed from the CatalogSource.

Manually Approving Upgrades via Subscriptions

By default, OLM will automatically approve updates to an operator as new versions become available via a CatalogSource. When creating a subscription, it is possible to disable automatic updates by setting the installPlanApproval field to Manual like so:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-operator
 namespace: operators
spec:
 channel: stable
 name: my-operator
 source: my-catalog
 sourceNamespace: operators
 installPlanApproval: Manual

Setting the installPlanApproval field to manual will prevent OLM from automatically installing the operator. As such, you will need to approve the installPlan which can be done with the following commands:

kubectl -n operators get installplans
NAME CSV APPROVAL APPROVED
install-bfmxd my-operator.v0.1.0 Manual false

$ kubectl -n operators patch installplan install-bfmxd -p '{"spec":{"approved":true}}' --type merge
installplan.operators.coreos.com/install-bfmxd patched

$ kubectl -n operators get installplans
NAME CSV APPROVAL APPROVED
install-bfmxd my-operator.v0.1.0 Manual true

Now that the install-bfmxd installPlan is in the approved state, OLM will install the operator defined by the my-operator.v0.1.0 CSV.

When the CatalogSource is updated with a newer version of that operator in the channel you selected, a new installPlan will be created in the namespace that you installed the operator to, as shown below:

$ kubectl -n operators get installplans
NAME CSV APPROVAL APPROVED
install-bfmxd my-operator.v0.1.0 Manual true
install-svojy my-operator.v0.2.0 Manual false

From here, you can approve install-svojy using the patch command shown earlier.

With the new installPlan in the approve state, the my-operator.v0.2.0 CSV will be deployed to the cluster and if the CSV reaches the Succeeded state the old CSV will be deleted. If the new CSV fails to reach the Succeeded state, both CSVs will continue to exist and it is up to the user to resolve the failure. In either case, OLM will not delete old installPlans as they act as a record of CSVs that were installed on your cluster.

How do I know when an update is available for an operator

It is possible to identify when there is a newer version of an operator available by inspecting the status of the operator’s subscription. The value associated with the currentCSV field is the newest version that is known to OLM, and installedCSV is the version that is installed on the cluster.

Docs: InstallPlan

Mon, 01 Jan 0001 00:00:00 +0000

Warning

This page is under construction

An InstallPlan defines a set of resources to be created in order to install or upgrade to a specific version of a ClusterService defined by a CSV.

Docs: OperatorGroup

Mon, 01 Jan 0001 00:00:00 +0000

An OperatorGroup is an OLM resource that provides rudimentary multitenant configuration to OLM installed operators.

Overview

An OperatorGroup selects a set of target namespaces in which to generate required RBAC access for its member operators.
The set of target namespaces is provided via a comma-delimited string stored in the olm.targetNamespaces annotation. This annotation is applied to member operator’s ClusterServiceVersion (CSV) instances and is projected into their deployments. It is accessible to operator containers using The Downward API
An operator is said to be a member of an OperatorGroup if its CSV exists in the same namespace as the OperatorGroup and its CSV’s InstallModes support the set of namespaces targeted by the OperatorGroup
In order to transition, a CSV must be an active member of an OperatorGroup that has no provided API conflicts with intersecting OperatorGroups

Membership

An operator defined by CSV csv-a is said to be a member of OperatorGroup op-a in namespace ns-a if both of the following hold:

op-a is the only OperatorGroup in ns-a
csv-a’s InstallModes support op-a’s target namespace set

TooManyOperatorGroups

If there exists more than one OperatorGroup in a single namespace, any CSV created in that namespace will transition to a failure state with reason TooManyOperatorGroups. CSVs in a failed state for this reason will transition to pending once the number of OperatorGroups in their namespaces reaches one.

InstallModes and Supported OperatorGroups

An InstallMode consists of an InstallModeType field and a boolean Supported field. A CSV’s spec can contain a set of InstallModes of four distinct InstallModeTypes:

OwnNamespace: If supported, the operator can be a member of an OperatorGroup that selects its own namespace
SingleNamespace: If supported, the operator can be a member of an OperatorGroup that selects one namespace
MultiNamespace: If supported, the operator can be a member of an OperatorGroup that selects more than one namespace
AllNamespaces: If supported, the operator can be a member of an OperatorGroup that selects all namespaces (target namespace set is the empty string “”)

Note: If a CSV’s spec omits an entry of InstallModeType, that type is considered unsupported unless support can be inferred by an existing entry that implicitly supports it.

UnsupportedOperatorGroup

If a CSV’s InstallModes do not support the target namespace selection of the OperatorGroup in its namespace, the CSV will transition to a failure state with reason UnsupportedOperatorGroup. CSVs in a failed state for this reason will transition to pending once either the OperatorGroups’s target namespace selection changes to a supported configuration, or the CSV’s InstallModes are modified to support the OperatorGroup’s target namespace selection.

Target Namespace Selection

Select the set of namespaces by specifying a label selector with the spec.selector field:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: my-group
 namespace: my-namespace
spec:
 selector:
 cool.io/prod: "true"

or by explicitly naming target namespaces with the spec.targetNamespaces field:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: my-group
 namespace: my-namespace
spec:
 targetNamespaces:
 - my-namespace
 - my-other-namespace
 - my-other-other-namespace

Note: If both spec.targetNamespaces and spec.selector are defined, spec.selector is ignored.

Additionally, a global OperatorGroup (which selects all namespaces) is specified by omitting both spec.selector and spec.targetNamespaces:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: my-group
 namespace: my-namespace

The resolved set of selected namespaces is surfaced in an OperatorGroup’s status.namespaces field. A global OperatorGroup’s status.namespace is of length 1 and contains the empty string, "", which signals a consuming operator that it should watch all namespaces.

Note: The consuming operator must know to treat "" as an all namespace configuration.

OperatorGroup CSV Annotations

Member CSVs of an OperatorGroup get the following annotations:

olm.operatorGroup=<group-name>
- Contains the name of the OperatorGroup
olm.operatorGroupNamespace=<group-namespace>
- Contains the namespace of the OperatorGroup
olm.targetNamespaces=<target-namespaces>
- Contains a comma-delimited string listing the OperatorGroup’s target namespace selection. This annotation is projected onto the pod template of a CSV’s deployments and can be consumed by a pod instance via The Downward API

Note: All annotations except olm.targetNamespaces are included with copied CSVs. Omitting the olm.targetNamespaces annotation on copied CSVs prevents the names of target namespaces from being leaked between tenants.

Provided APIs Annotation

Information about what GroupVersionKindss (GVK) are provided by an OperatorGroup are surfaced in an olm.providedAPIs annotation. The annotation’s value is a string consisting of a set of <Kind>.<version>.<group>s delimited with commas. The GVKs of CRDs and APIServices provided by all active member CSVs of an OperatorGroup are included.

Here’s an example of an OperatorGroup with a single active member CSV providing the PackageManifests resource:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 annotations:
 olm.providedAPIs: PackageManifest.v1.packages.operators.coreos.com
 name: olm-operators
 namespace: local
 ...
spec:
 selector: {}
 serviceAccount:
 metadata:
 creationTimestamp: null
 targetNamespaces:
 - local
status:
 lastUpdated: 2019-02-19T16:18:28Z
 namespaces:
 - local

RBAC

When an OperatorGroup is created, 3 ClusterRoles each containing a single AggregationRule are generated:

<operatorgroup-name>-admin
- ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name> label
<operatorgroup-name>-edit
- ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name> label
<operatorgroup-name>-view
- ClusterRoleSelector set to match the olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name> label

When a CSV becomes an active member of an OperatorGroup and is not in a failed state with reason InterOperatorGroupOwnerConflict, the following RBAC resources are generated:

For each provided API resource from a CRD:
- A <kind>.<group>-<version>-admin ClusterRole is generated with the * verb on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-admin: true and olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name>
- A <kind>.<group>-<version>-edit ClusterRole is generated with the create, update, patch, delete verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-edit: true and olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name>
- A <kind>.<group>-<version>-view ClusterRole is generated with the get, list, watch verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>
- A <kind>.<group>-<version>-view-crdview ClusterRole is generated with the get verb on apiextensions.k8s.io customresourcedefinitions <crd-name> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>
For each provided API resource from an APIService:
- A <kind>.<group>-<version>-admin ClusterRole is generated with the * verb on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-admin: true and olm.opgroup.permissions/aggregate-to-admin: <operatorgroup-name>
- A <kind>.<group>-<version>-edit ClusterRole is generated with the create, update, patch, delete verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-edit: true and olm.opgroup.permissions/aggregate-to-edit: <operatorgroup-name>
- A <kind>.<group>-<version>-view ClusterRole is generated with the get, list, watch verbs on <group> <kind> with aggregation labels rbac.authorization.k8s.io/aggregate-to-view: true and olm.opgroup.permissions/aggregate-to-view: <operatorgroup-name>
If |target namespaces| == 1 and contains *:
- A ClusterRole and corresponding ClusterRoleBinding are generated for each permission defined in the CSV’s permissions field. All resources generated are given the olm.owner: <csv-name> and olm.owner.namespace: <csv-namespace> labels
Else for each target namespace:
- All Roles and RoleBindings in the operator namespace with the olm.owner: <csv-name> and olm.owner.namespace: <csv-namespace> labels are copied into the target namespace.

Copied CSVs

OLM will create copies of all active member CSVs of an OperatorGroup in each of that OperatorGroup’s target namespaces. The purpose of a Copied CSV is to tell users of a target namespace that a specific operator is configured to watch resources created there. Copied CSVs have a status reason Copied and are updated to match the status of their source CSV. The olm.targetNamespaces annotation is stripped from copied CSVs before they are created on the cluster. Omitting the target namespace selection avoids an unnecessary information leak. Copied CSVs are deleted when their source CSV no longer exists or the operator group their source CSV belongs to no longer targets the copied CSV’s namespace.

Static OperatorGroups

An OperatorGroup is static if it’s spec.staticProvidedAPIs field is set to true. As a result, OLM does not modify the OperatorGroups’s olm.providedAPIs annotation, which means that it can be set in advance. This is useful when a user wishes to use an OperatorGroup to prevent resource contention in a set of namespaces, but does not have active member CSVs that provide the APIs for those resources.

Here’s an example of an OperatorGroup that “protects” prometheus resources in all namespaces with the something.cool.io/cluster-monitoring: "true" annotation:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: cluster-monitoring
 namespace: cluster-monitoring
 annotations:
 olm.providedAPIs: Alertmanager.v1.monitoring.coreos.com,Prometheus.v1.monitoring.coreos.com,PrometheusRule.v1.monitoring.coreos.com,ServiceMonitor.v1.monitoring.coreos.com
spec:
 staticProvidedAPIs: true
 selector:
 matchLabels:
 something.cool.io/cluster-monitoring: "true"

OperatorGroup Intersection

OperatorGroup Intersection Terminology

Two OperatorGroups are said to be intersecting if the intersection of their target namespace sets is not the empty set
Two OperatorGroups are said to have intersecting provided APIs if they are intersecting and the intersection of their provided API sets (defined by olm.providedAPIs annotations) is not the empty set

What Can Go Wrong?

OperatorGroups with intersecting provided APIs can compete for the same resources in the set of intersecting namespaces.

Rules for Intersection

Each time an active member CSV syncs, OLM queries the cluster for the set of intersecting provided APIs between the CSV’s OperatorGroup and all others. OLM then checks if that set is the empty set:

If true and the CSV’s provided APIs are a subset of the OperatorGroup’s:
- Continue transitioning
If true and the CSV’s provided APIs are not a subset of the OperatorGroup’s:
- If the OperatorGroup is static:
  - Clean up any deployments that belong to the CSV
  - Transition the CSV to a failed state with status reason CannotModifyStaticOperatorGroupProvidedAPIs
- Else:
  - Replace the OperatorGroup’s olm.providedAPIs annotation with the union of itself and the CSV’s provided APIs
If false and the CSV’s provided APIs are not a subset of the OperatorGroup’s:
- Clean up any deployments that belong to the CSV
- Transition the CSV to a failed state with status reason InterOperatorGroupOwnerConflict
If false and the CSV’s provided APIs are a subset of the OperatorGroup’s:
- If the OperatorGroup is static:
  - Clean up any deployments that belong to the CSV
  - Transition the CSV to a failed state with status reason CannotModifyStaticOperatorGroupProvidedAPIs
- Else:
  - Replace the OperatorGroup’s olm.providedAPIs annotation with the difference between itself and the CSV’s provided APIs

Note: Failure states caused by OperatorGroups are non-terminal. Note: When checking intersection rules, an OperatorGroup’s namespace is always included as part of its selected target namespaces.

Each time an OperatorGroup syncs:

The set of provided APIs from active member CSV’s is calculated from the cluster (ignoring copied CSVs)
The cluster set is compared to olm.providedAPIs:
- If olm.providedAPIs contains any extraneous provided APIs:
  - olm.providedAPIs is pruned of any extraneous provided APIs (not provided on cluster)
All CSVs that provide the same APIs across all namespaces (including those removed) are requeued. This notifies conflicting CSVs in intersecting groups that their conflict has possibly been resolved, either through resizing or through deletion of the conflicting CSV.