Operator Lifecycle Manager – Advanced Tasks

Docs: Shipping an operator that includes Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

The WebhookDefinition ClusterServiceVersion Object

Operators may define validating, mutating, and conversion webhooks within its ClusterServiceVersion resource (CSV)

These webhooks are defined with the CSV’s WebhookDefinition array. The WebhookDescription object contains a union of the fields defined in the ValidatingWebhookConfiguration and MutatingWebhookConfiguration type with the exception of the NamespaceSelector, which is generated by OLM to match namespaces scoped by the OperatorGroup that the operator is deployed in.

In addition to these fields, OLM requires that you define:

The Type of the Webhook, which must be set to ValidatingAdmissionWebhook, MutatingAdmissionWebhook, or ConversionWebhook
The DeploymentName that OLM must mount the CA Cert information into, this should match the name of a deployment defined in the CSV

Creating an Operator with a Webhook

This document will not cover the steps required to create an operator that includes an admission or conversion webhook. If you are interested in creating such an operator, please refer to the following documentation provided by the Operator-SDK and Kubebuilder projects:

Deploying an operator with webhooks using OLM

OLM has a few constraints that must be considered when developing an operator featuring validating, mutating, or conversion webhooks.

Certificate Authority Requirements

OLM will create and mount a self-signed Certificate Authority (CA) to each deployment that contains a webhook as defined in the CSV. The certificates generated by OLM will expire after 2 years, at which point OLM will generate new certificates and redeploy the operator. The logic that generates and mounts the CA into the deployment was originally developed for the API Service lifecycle logic and has been expanded for use with webhooks.

OLM will mount the certificates at the default location that operators built with Kubebuilder and the Operator-SDK expect the certificates, specifically

The TLS Cert file will be mounted to the deployment at /tmp/k8s-webhook-server/serving-certs/tls.cert
The TLS Key file will be mounted to the deployment at /tmp/k8s-webhook-server/serving-certs/tls.key

Note: When OLM was released, the CAs meant for Webhooks were mounted to the same location as defined by the existing API Service Lifecycle code. In an effort to remain backwards compatible, the certificates are also mounted to the following locations:

The TLS Cert file will also be mounted to the deployment at /apiserver.local.config/certificates/apiserver.crt
The TLS Key file will also be mounted to the deployment at /apiserver.local.config/certificates/apiserver.key

OLM does not allow users to specify a mount location or name for the certificates. This issue should be addressed once OLM makes the decision to offload the management of CAs to projects such as Cert-Manager and the Service-CA Operator, this work can be tracked here.

Admission Webhook Rule Requirements

When developing an admission webhook please be aware that in an attempt to prevent operator from configuring the cluster into an unrecoverable state, OLM will place the CSV in the failed phase if the Rules defined in an admission webhook:

Intercept requests that target all groups
Intercept requests that target the operators.coreos.com group
Intercept requests that target the ValidatingWebhookConfigurations or MutatingWebhookConfigurations resources

Conversion Webhook Rules Requirements

CSVs featuring a conversion webhook may only support the AllNamespaces installMode
The CRD targeted by the Conversion webhook must have its spec.preserveUnknownFields field set to false or nil
The Conversion Webhook defined in the CSV must target an owned CRD

Example: Installing a operator featuring webhooks with OLM

This example will focus on installing an operator that the OLM team built for its e2e test portfolio, the webhook-operator. The webhook-operator implements a validating, mutating, and conversion webhook for the WebhookTest CRD.

Defining Conversion Webhooks in the ClusterServiceVersion

The full webhook operator CSV can be seen here, but we will highlight the important parts of the CSV below:

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
 name: webhook-operator.v0.0.1
spec:
 customresourcedefinitions:
 owned:
 - kind: WebhookTest
 name: webhooktests.webhook.operators.coreos.io # The CRDs target by the conversion webhook must exist here
 version: v1
 install:
 spec:
 deployments:
 - name: webhook-operator-webhook
 ...
 ...
 ...
 strategy: deployment
 installModes:
 - supported: false
 type: OwnNamespace
 - supported: false
 type: SingleNamespace
 - supported: false
 type: MultiNamespace
 - supported: true
 type: AllNamespaces
 webhookdefinitions:
 - type: ValidatingAdmissionWebhook
 admissionReviewVersions:
 - v1beta1
 - v1
 containerPort: 443
 targetPort: 4343
 deploymentName: webhook-operator-webhook
 failurePolicy: Fail
 generateName: vwebhooktest.kb.io
 rules:
 - apiGroups:
 - webhook.operators.coreos.io
 apiVersions:
 - v1
 operations:
 - CREATE
 - UPDATE
 resources:
 - webhooktests
 sideEffects: None
 webhookPath: /validate-webhook-operators-coreos-io-v1-webhooktest
 - type: MutatingAdmissionWebhook
 admissionReviewVersions:
 - v1beta1
 - v1
 containerPort: 443
 targetPort: 4343
 deploymentName: webhook-operator-webhook
 failurePolicy: Fail
 generateName: mwebhooktest.kb.io
 rules:
 - apiGroups:
 - webhook.operators.coreos.io
 apiVersions:
 - v1
 operations:
 - CREATE
 - UPDATE
 resources:
 - webhooktests
 sideEffects: None
 webhookPath: /mutate-webhook-operators-coreos-io-v1-webhooktest
 - type: ConversionWebhook
 admissionReviewVersions:
 - v1beta1
 - v1
 containerPort: 443
 targetPort: 4343
 deploymentName: webhook-operator-webhook
 generateName: cwebhooktest.kb.io
 sideEffects: None
 webhookPath: /convert
 conversionCRDs:
 - webhooktests.webhook.operators.coreos.io # Each CRD must have spec.PreserveUnknownFields property set to `false` or `nil`

Docs: Overriding CatalogSource Pod Scheduling Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Overriding CatalogSource Pod Scheduling Configuration

When given a CatalogSource of type grpc with spec.image defined, the catalog operator will create Pod that serves the content in spec.image. By default, this pod defines in its spec no tolerations, no priorityClassName, and only the following node selector: kubernetes.io/os=linux. These values can be overriden with CatalogSource’s spec.grpcPodConfig. For instance,

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: operatorhubio-catalog
 namespace: olm
spec:
 sourceType: grpc
 image: quay.io/operatorhubio/catalog:latest
 displayName: Community Operators
 publisher: OperatorHub.io

 # optional
 grpcPodConfig:
 # override nodeSelector
 # for more information on nodeSelectors see:
 # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
 # optional
 nodeSelector:
 custom_label: value

 # change priorityClassName
 # kubernetes ships with: system-cluster-critical and system-node-critical
 # setting it to empty ("") will assign the pod the default priority.
 # Other priority classes can be defined manually. For more information on priority classes see:
 # https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
 # optional
 priorityClassName: system-cluster-critical

 # override tolerations
 # for more information on taints and tolerations see:
 # https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration
 # optional
 tolerations:
 - key: "key1"
 operator: "Equal"
 value: "value1"
 effect: "NoSchedule"

As can be seen in the example, spec.grpcPodConfig and all of its attributes are optional. These attributes are: nodeSelector, priorityClassName, and tolerations. It should be noted that priorityClassName can be overriden to be "". This will give the pod the default priority. Any value outside system-cluster-critical, system-node-critical, and "" will need to correspond to a pre-existing and custom defined priorityClass.

Previously, the only pod scheduling parameter that could be overriden was the priorityClassName. This was done by adding the following annotation to the CatalogSource CR: operatorframework.io/priorityclass. For instance:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
 name: operatorhubio-catalog
 namespace: olm
 annotations:
 # kubernetes ships with: system-cluster-critical and system-node-critical
 # setting it to empty ("") will assign the pod the default priority.
 # Other priority classes can be defined manually. For more information on priority classes see:
 # https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
 # optional
 operatorframework.io/priorityclass: system-cluster-critical
spec:
 sourceType: grpc
 image: quay.io/operatorhubio/catalog:latest
 displayName: Community Operators
 publisher: OperatorHub.io

NOTE: If a CatalogSource CR defines both the annotation and spec.grpcPodConfig.priorityClassName, the annotation will take precedence over the configuration parameter.

Docs: Communicating Operator Conditions to OLM

Mon, 01 Jan 0001 00:00:00 +0000

Communicating Operator Conditions to OLM

As part of its role in managing the lifecycle of an operator, the Operator-Lifecycle-Manager (OLM) infers the state of an operator from the state of Kubernetes resources that define the operator. While this approach provides some level of assurance that an operator is in a given state, there are many instances where an operator may wish to communicate information to OLM that could not be inferred otherwise. This information can then be used by OLM to better manage the lifecycle of the operator.

OperatorConditions

OLM introduced a new CustomResourceDefinition called the OperatorCondition allowing operators to communicate conditions to OLM. There are a set of “OLM Supported Conditions” which influence OLM’s management of the operator when present in the OperatorCondition’s Status.Conditions array.

OLM Supported Conditions

The set of “OLM Supported Conditions” include:

The Upgradeable Condition

Upgradeable

The Upgradeable “OLM Supported Condition” prevents the existing CSV from being replaced by a newer version of the CSV. When the Upgradeable condition is set to False, OLM will:

Prevent a channel entry in a subscribed package that replaces the operator’s existing CSV from leaving the PendingPhase.

The Upgradeable condition might be useful when:

An operator is about to start a critical process and should not be upgraded until after the process is completed.
The operator is performing a migration of CRs that must be completed before the operator is ready to be upgraded.

Example Upgradeable OperatorCondition

apiVersion: operators.coreos.com/v1
kind: OperatorCondition
metadata:
 name: foo-operator
 namespace: operators
status:
 conditions:
 - type: Upgradeable # The name of the `Upgradeable` OLM Supported Condition.
 status: "False" # The operator is not ready to be upgraded.
 reason: "migration"
 message: "The operator is performing a migration."
 lastTransitionTime: "2020-08-24T23:15:55Z"

Given that the Upgradable Condition’s status is set to False, OLM will understand that it should not upgrade the operator.

Overriding OperatorConditions

There are times as a Cluster Admin that you may want to ignore an “OLM Supported Condition” reported by an Operator. For example, imagine that a known version of an operator always communicates that it is not upgradeable. In this instance, you may want to upgrade the operator despite the operator communicating that it is not upgradeable. This could be accomplished by overriding the OLM Supported Condition by adding the condition’s type and status to the spec.overrides array in the OperatorCondition CR:

“OLM Supported Conditions” can be overridden by Cluster Admins by appending the desired OperatorCondition to the opertor’s OperatorCondition’s Spec.Overrides Condition Array. When present, “OLM Supported Conditions” in the Spec.Overrides array will override the Conditions in the Status.conditions array, allowing Cluster Admins to deal with situations where an operator is incorrectly reporting a state to OLM.

Example Override

apiVersion: operators.coreos.com/v1
kind: OperatorCondition
metadata:
 name: foo-operator
 namespace: operators
spec:
 overrides:
 - type: Upgradeable # Allows the cluster admin to change operator's Upgrade readiness to True
 status: "True"
 reason: "upgradeIsSafe"
 message: "This is a known issue with the operator where it always reports that it cannot be upgraded."
status:
 conditions:
 - type: Upgradeable
 status: "False"
 reason: "migration"
 message: "The operator is performing a migration."
 lastTransitionTime: "2020-08-24T23:15:55Z"

Updating your operator to use OLM OperatorCondition

OLM will automatically create an OperatorCondition for each ClusterServiceVersion that it reconciles. All service accounts in the CSV will be granted the RBAC to interact with the OperatorCondition owned by the operator.

Operators deployed by OLM may then use the operator-library to set conditions on their operator.

Setting Defaults

In an effort to remain backwards compatible, OLM treats the absence of an OperatorConditions as opting out of the condition. Therefore, an operator that opts in to using OperatorConditions should set default conditions before the pod’s ready probe is set to true. This provides the operator with a grace period to update the condition to the correct state.

Docs: Configuring OLM

Mon, 01 Jan 0001 00:00:00 +0000

Configuring OLM

The Operator-Lifecycle-Manager (OLM) controller can be configured through an OLMConfig CustomResourceDefinition (CRD) named cluster. This document will outline what configurations OLM currently supports.

Disabling Copied CSVs

Background

When an operator is installed by OLM, a stripped down copy of its CSV is created in every namespace the operator is configured to watch. These stripped down CSVs are known as “Copied CSVs” and communicate to users which controllers are actively reconciling resource events in a given namespace. When operators are installed in the AllNamespace mode, a Copied CSV is created in every namespace on the cluster. On especially large clusters, with namespaces and installed operators tending in the hundreds or thousands, Copied CSVs consume an untenable amount of resources; e.g. OLM’s memory usage, cluster Etcd limits, networking, etc.

Usage

In an effort to support these larger cluster, OLM allows users to disable Copied CSVs for operators installed in the AllNamespace mode by setting the cluster olmConfig’s spec.features.disableCopiedCSVs field to true.

$ kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OLMConfig
metadata:
name: cluster
spec:
features:
disableCopiedCSVs: true # Disabled Copied CSVs for AllNamespace operators.
EOF
olmconfig.operators.coreos.com/cluster configured

When Copied CSVs are Disabled, OLM will capture this information in an event in the operator’s namespace, an example of the event can be seen below:

$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
85s Warning DisabledCopiedCSVs clusterserviceversion/my-csv.v1.0.0 CSV copying disabled for operators/my-csv.v1.0.0

When the cluster olmConfig’s spec.features.disableCopiedCSVs field is missing or set to false, OLM will recreate the Copied CSVs for all operators installed in the AllNamespace mode and deleted the previously mentioned events.

Additional information about this feature can be found in its original enhancement proposal.

Changing the Package Server Sync Interval

Background

After CatalogSources are created, they are synced by the OLM packageservers every 12 hours. This sync interval may be increased to reduce the CPU utilization of the packageserver and CatalogSources, or decreased to improve response times to available updates. This field represents a duration, but is limited to using hours, minutes and seconds.

Usage

$ kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1
kind: OLMConfig
metadata:
name: cluster
spec:
features:
packageServerSyncInterval: 1h30m
EOF
olmconfig.operators.coreos.com/cluster configured

When the cluster olmConfig’s spec.features.packageServerSyncInterval field is missing, OLM will use the default value of 12h.

Docs: Shipping an operator that supports Multiple Architectures

Mon, 01 Jan 0001 00:00:00 +0000

An operator’s target OS and architecture can be specified by labelling its respective ClusterServiceVersion.

Supported Labels

The following label convention defines the target OSes and architectures supported by an operator:

labels:
 operatorframework.io/arch.<GOARCH>: supported
 operatorframework.io/os.<GOOS>: supported

Where <GOARCH> and <GOOS> are one of the values listed here.

Multiple Architectures

Some operators may support multiple node architectures or OSes. In this case, multiple labels can be added. For example, an operator that support both windows and linux workloads will sport the following labels:

labels:
 operatorframework.io/os.windows: supported
 operatorframework.io/os.linux: supported

Defaults

If a ClusterServiceVersion does not include an os label, a target OS is assumed to be linux.

labels:
 operatorframework.io/os.linux: supported

If a ClusterServiceVersion does not include an arch label, a target architecture is assumed to be amd64.

labels:
 operatorframework.io/arch.amd64: supported

Filtering available operators by os or arch

Only windows:

kubectl get packagemanifests -l operatorframework.io/os.windows=supported

Caveats

Only the labels on the HEAD of the default channel are considered for filtering PackageManifests by label.

This means, for example, that providing an alternate architecture for an operator in the non-default channel is possible, but will not be available for filtering in the PackageManifest API.

Docs: Overriding Operator Pod Affinity Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Overriding Operator Pod Affinity Configuration

Pods can be configured to be scheduled in particular nodes using affinity and anti-affinity constraints. Namely, node affinity, and pod affinity and anti-affinity. These constraints can enable users to have high-grained control over where pods are scheduled in the cluster, e.g. to schedule them on nodes running with cheaper architectures, such as arm64, or to improve service resilience by ensuring pod replicas are never scheduled on the same node. In OLM, the Subscription API can be used to override operator pod’s affinity configuration, thus giving users the ability to override or define their own affinity settings for operator deployments.

The affinity settings for an operator deployment pod defined by the operator author can be overriden in Subscription.config.affinity, i.e.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-package
 namespace: my-namespace
spec:
 name: my-package
 source: my-operators
 sourceNamespace: operator-registries
 config:
 affinity:
 nodeAffinity:
 ...
 podAffinity:
 ...
 podAntiAffinity:
 ...

Example: Overriding/Defining Node Affinity

The operator nodeAffinity configuration can be overriden in the following way:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-package
 namespace: my-namespace
spec:
 name: my-package
 source: my-operators
 sourceNamespace: operator-registries
 config:
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64

Note that this will completely override the nodeAffinity configuration defined in the operator deployment pod spec defined by the author.

Example: Removing Operator Author defined Affinity

The empty object {} can be used to remove any affinity definition already defined in the operator deployment pod spec, e.g.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-package
 namespace: my-namespace
spec:
 name: my-package
 source: my-operators
 sourceNamespace: operator-registries
 config:
 affinity: {}

If equivalent to no affinity configuration. And,

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: my-package
 namespace: my-namespace
spec:
 name: my-package
 source: my-operators
 sourceNamespace: operator-registries
 config:
 affinity:
 podAffinity: {}
 podAntiAffinity: {}

Is equivalent to keeping the original nodeAffinity, while removing the original podAffinity and podAnitAffinity configurations.

Docs: Operator scoping with OperatorGroups

Mon, 01 Jan 0001 00:00:00 +0000

OLM runs with cluster-admin privileges and is capable of granting permissions to operators that it deploys. By default, an operator can specify any set of permission(s) in the CSV and OLM will consequently grant it to the operator. In effect, an operator can achieve cluster-scoped privilege(s) which may not always be desired.

OLM introduces the concept of OperatorGroups to enable cluster admins complete control over the permissions that OLM grants operators that it deploys.

An admin may create a single OperatorGroup in a given namespace. Any CSV created in that namespace is said to be a member operator of that OperatorGroup. With OperatorGroups, a cluster admin can:

Define the set of permissions that OLM may grant to member operators
Define the set of namespaces that OLM may grant namespaced permissions in.

Configuring OperatorGroups

Scoping Member Operators to Specific Namespaces

Using an OperatorGroup, a cluster admin can scope member operators’ namespaced permissions to specific namespaces in two ways:

Using a predefined list of namespaces
Using a label selector.

Defining a set of namespaces

The set of namespaces can be hardcoded setting the spec.targetNamespaces of an OperatorGroup like so:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: my-group
 namespace: my-namespace
spec:
 targetNamespaces:
 - my-namespace
 - my-other-namespace
 - my-other-other-namespace

In the example above, member operator will be scoped to the following namespaces:

my-namespace
my-other-namespace
my-other-other-namespace

Defining a set of namespaces with a label selector

Cluster admins may not know in advance which namespaces member operators should be scoped to. In this case, it may make sense to use a label selector to identify which namespaces permissions should be granted in. A namespace selector can be defined for an OpearatorGroup like so:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: my-group
 namespace: my-namespace
spec:
 selector:
 matchLabels:
 cool.io/prod: "true"

In the example above, member operators will be scoped to any namespaces with the cool.io/prod: true label. If no namespaces exist with the cool.io/prod: true label, OLM will fail to install any member operators.

Note: In the case that both a selector and a list of namespaces are provided, the selector is ignored.

TargetNamespaces and their relationship to InstallModes

When creating OperatorGroups it is important to keep in mind that an operator may not support all namespace configurations. For example, an operator that is designed to run at the cluster level shouldn’t be expected to work in an OperatorGroup that defines a single targetNamespace.

Operator authors are responsible for defining which InstallModes their operator supports within its ClusterServiceVersion (CSV). There are four InstallModes that an operator can support:

OwnNamespace: If supported, the operator can be configured to watch for events in the namespace it is deployed in.
SingleNamespace: If supported, the operator can be configured to watch for events in a single namespace that the operator is not deployed in.
MultiNamespace: If supported, the operator can be configured to watch for events in more than one namespace.
AllNamespaces: If supported, the operator can be configured to watch for events in all namespaces.

Note: If a CSV’s spec omits an entry of InstallModeType, that type is considered unsupported unless support can be inferred by an existing entry that implicitly supports it.

Cluster admins cannot override which InstallModes an operator supports, and so should understand how to create an OperatorGroup that supports each InstallMode. Let’s look at an example of an OperatorGroup implementing each type of installMode:

OwnNamespace

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: own-namespace-operator-group
 namespace: own-namespace
spec:
 targetNamespaces:
 - own-namespace

SingleNamespace

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: single-namespace-operator-group
 namespace: own-namespace
spec:
 targetNamespaces:
 - some-other-namespace

MultiNamespace

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: multi-namespace-operator-group
 namespace: own-namespace
spec:
 targetNamespaces:
 - own-namespace
 - some-other-namespace

AllNamespaces

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: all-namespaces-operator-group
 namespace: own-namespace

Scoping Member Operator Permissions

When creating an OperatorGroup, cluster admins may specify a ServiceAccount that defines the set of permissions that may be granted to all member operators. OLM will ensure that when an operator is installed its privileges are confined to that of the ServiceAccount specified.

As a result a cluster-admin can limit an operator to a pre-defined set of RBAC rules. The Operator will not be able to do anything that is not explicitly permitted by these permissions. This enables self-sufficient installation of Operators by non-cluster-admin users with a limited scope.

Defining a ServiceAccount for an OperatorGroup

A ServiceAccount may be defined within an OperatorGroup like so:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
 name: scoped-permissions-operator-group
 namespace: own-namespace
spec:
 serviceAccountName: member-operator-servicee-account
 targetNamespaces:
 - own-namespace

Any operator tied to this OperatorGroup will now be confined to the permission(s) granted to the specified ServiceAccount. If the operator asks for permission(s) that are outside the scope of the ServiceAccount the install will fail with appropriate error(s).

Configuring the End User Experience

Making Operators Available

Once a cluster admin has successfully created an OperatorGroup, he or she then has the opportunity to decide which operators should be offered as a part of that group. This is an important phase in configuring the cluster, as most users may not have the ability to install an operator into an OperatorGroup.

A cluster admin can add an operator to an OperatorGroup by creating a Subscription in the same namespace. An operator can be added and removed from an OperatorGroup at anytime.

Keep in mind that this process can be repeated for any number of OperatorGroups. This means that a cluster admin can decide for a set of operator to watch for events in one set of namespaces while defining a seperate set of operators that watch for events in a seperate set of namespaces.

Enabling a User to use an Operator

Once a cluster admin has decided which operators are available on a cluster, it is now time to decide which user(s) may take advantage of an available operator.

Users interact with operators by creating a resource in a namespace that an operator is watching. As such, by tuning RBAC privileges a cluster admin has complete control over the set of users that may interact with an operator along with the set of APIs that user may take advantage of.

For example, if an operator were available that offered two APIs, a cluster admin may provide a user with full RBAC privileges over one API but not grant the user any privileges with the second API.

Docs: Opting into UnsafeFailForward upgrades

Mon, 01 Jan 0001 00:00:00 +0000

Warning

This feature is not recommended in a majority of cases as enabling UnsafeFailForward upgrades no longer guarantees sane upgrade paths, possibly causing unrecoverable failures resulting in data loss. Only use this feature if you:

Know every operator installed in the namespace.
Have deep knowledge regarding the upgrade paths for each operator in the namespace.
Have control over the contents of the catalogs providing the operators in the namespace.
Do not want to manually upgrade your failed operators.

What “UnsafeFailForward” upgrades add

A failed installation/upgrade is typically caused by one of two scenarios:

A Failed CSV: The CSV is in the FAILED phase.
A Failed InstallPlan: Usually occurs because a resource listed in the InstallPlan fails to be created or updated. An InstallPlan may fail independently of its CSV and may fail to create the CSV.

By opting into “UnsafeFailForward” upgrades, OLM will allow you to recover from failed installations and upgrades by:

Allowing CSVs to move from the FAILED phase to the REPLACING phase.
Allowing OLM to calculate new InstallPlans for a set of installables if:
- The InstallPlan referenced by a Subscription is in the FAILED phase.
- The CatalogSource has been updated to include a new upgrade for one or more CSVs in the namespace.

Using UnsafeFailForward Upgrades

Since resolution is namespace scoped, the toggle for allowing “UnsafeFailForward” upgrades is namespace scoped as well. Accordingly, the namespace scoped resource, OperatorGroup, has the upgradeStrategy field.

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: foo
 namespace: bar
spec:
 # Possible values include "Default" or "TechPreviewUnsafeFailForward".
 upgradeStrategy: TechPreviewUnsafeFailForward

With the upgradeStrategy type set to TechPreviewUnsafeFailForward, OLM will allow operators to “UnsafeFailForward” in adherence with the principles discussed below. If the upgradeStrategy is unset or set to Default OLM will exhibit existing behavior.

Understanding “UnsafeFailForward” upgrades

Before using UnsafeFailForward upgrades, it is important to understand how OLM and the Resolver calculate what needs to be installed. When determining what to install, OLM provides the Resolver with the set of CSVs in a namespace. The Resolver treats these CSVs differently depending on whether or not they are claimed by a Subscription, where the Subscription lists the CSV’s name in its .status.currentCSV field.

If a CSV is claimed by a Subscription, the Resolver will allow it to be upgraded by a bundle that replaces it.
If a CSV is not claimed by a Subscription, it is assumed that the user installed the CSV themselves, that OLM should not upgrade the CSV, and that it must appear in the next set of installables.

OLM’s Resolver is deterministic, meaning that the set of installables will not change for a given set of arguments if no new upgrades have been declared in the existing CatalogSources. When an upgrade fails due to a CSV entering the FAILED phase, the CSV being replaced still exists and is not referenced by a Subscription. Today, OLM would send both Operator v1 and Operator v2 to the Resolver which will normally be unsatisfiable because Operator v1 is marked as required and Operator v2 cannot be upgraded further since it provides the same APIs as Operator v1. In order to support “failing forward”, OLM needs to omit CSVs in the REPLACING phase from the set of arguments sent to the Resolver when “UnsafeFailForward” upgrades are enabled.

When an InstallPlan failed

Let’s review what steps must be taken to recover from a failed InstallPlan today:

Operator v1 is being upgraded to Operator v2.
The InstallPlan for Operator v2 fails.
Operator v3 is added to the catalog and is defined as the upgrade for Operator v1.
The user deletes the InstallPlan created for Operator v2.
A new InstallPlan is generated for Operator v3 and the upgrade succeeds.

With “UnsafeFailForward” upgrades, OLM allows new InstallPlans to be generated if:

The failed InstallPlan is referenced by a Subscription in the namespace.
The CatalogSource has been updated to include a new upgrade for the set of arguments.

In practice, the fourth step from the previous workflow would be removed, meaning that the you simply need to update the CatalogSource and wait for the cluster to move past the failed install. With catalog polling enabled, you can even skip updating the CatalogSource directly by pushing a new catalog image to the same tag.

BEST PRACTICE NOTE: If a bundle is known to fail, it should be skipped in the upgrade graph using the skips or skipRange feature.

To further understand, let’s visualize this.

graph TD; A(Operavtor V1 installed) --> B(Starts upgrading to Operator V2) B --> |Install succeeds| F(Operator V2 is installed and succeeds) B --> |InstallPlan fails| C(Operator V3 added to catalog replacing V2) C --> |Upgrade blocked as V1 is in solution set still| D(Manually delete Operator V2 InstallPlan) D --> |New InstallPlan generated and approved| E(Operator V3 is installed and succeeds) B --> |InstallPlan fails with UnsafeFailForward upgrades| H(Operator V3 added to catalog replacing V2) H --> |New InstallPlan generated and approved| J(Operator V3 is installed and succeeds)

When one or more CSVs have failed

Let’s review what steps must be taken to recover from a failed CSV today:

Operator v1 is being upgraded to Operator v2.
The CSV for Operator v2 enters the FAILED phase.
Operator v3 is added to the catalog which replaces or skips Operator v2.
The Resolver cannot upgrade Operator v2 while including Operator v1 in the solution set, upgrade is blocked.
User manually deletes the existing CSVs, a new InstallPlan is generated and approved.
Operator v3 is installed and the upgrade succeeds.

To further understand, let’s visualize this.

graph TD; A(Operator V1 installed) --> B(Starts upgrading to Operator V2) B --> |Install succeeds| F(Operator V2 is installed and succeeds) B --> |CSV enters FAILED phase| C(Operator V3 added to catalog replacing V2) C --> |Upgrade blocked as V1 is in solution set still| D(Manually delete Operator V2 CSV) D --> |New InstallPlan generated and approved| E(Operator V3 is installed and succeeds) B --> |CSV enters FAILED phase with UnsafeFailForward upgrades| H(Operator V3 added to catalog replacing V2) H --> |New InstallPlan generated and approved| J(Operator V3 is installed and succeeds)

When you opt into “UnsafeFailForward” upgrades, OLM:

Does not include CSVs in the REPLACING phase in the set of arguments sent to the Resolver if the final CSV in the upgrade chain is in the FAILED phase and all other CSVs are in the REPLACING phase.
Allows CSVs to move from the FAILED phase to the REPLACING phase.

These changes allow OLM to recover from any number of failed CSVs in an upgrade path, replacing the latest FAILED CSV with the next upgrade.

Docs: Catalog Update Formulary

Mon, 01 Jan 0001 00:00:00 +0000

Background

File-based-catalogs (FBC) and catalog templates (templates) provide operator authors with standardized schemas to express operator upgrade graphs. However, without explicit tooling users require clear examples of how to achieve routine goals. This document is an attempt to establish a formulary of common operations, specifically with the intention of making these pieces automatable. This is in no way an exhaustive list.

Conventions

Formulae will be identified as FBC, basic catalog template (basic template), or semver catalog template (semver template). Schema manipulations will be modeled using YAML and yq. Wherever possible, example input will be limited to the relevant object hierarchies. Truncation is indicated by elipses (…) before and after the text.

Examples

For brevity, all formulae will refer to the same example, for semver template and FBC. For convenience, all semver bundle image pullspecs will express versions which match the bundle version (instead of SHAs).

FBC example

Formulae presume the following content is saved to the file fbc.yaml

---
defaultChannel: stable-v1.0
name: testoperator
schema: olm.package
---
entries:
 - name: testoperator.v1.0.0
 - name: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
name: candidate-v1.0
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.1.0
 replaces: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
name: candidate-v1.1
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.0.1
name: fast-v1.0
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.1.0
 replaces: testoperator.v1.0.1
name: fast-v1.1
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.0.1
name: stable-v1.0
package: testoperator
schema: olm.channel

basic template example

Formulae presume the following context is saved to the file basic.yaml

schema: olm.template.basic
entries:
 - schema: olm.package
 name: testoperator
 defaultChannel: stable
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator.v0.1.0
 - name: testoperator.v0.2.0
 replaces: testoperator.v0.1.0
 - schema: olm.bundle
 image: quay.io/organization/testoperator:v0.1.0
 - schema: olm.bundle
 image: quay.io/organization/testoperator:v0.2.0

semver template example

Formulae presume the following content is saved to the file semver.yaml

schema: olm.semver
generateMajorChannels: false
generateMinorChannels: true
candidate:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.0
 - image: quay.io/organization/testoperator:v1.0.1
 - image: quay.io/organization/testoperator:v1.1.0
fast:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.1
 - image: quay.io/organization/testoperator:v1.1.0
stable:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.1

Formulae

Adding a new bundle to an existing channel

FBC

Add a new testoperator.v1.1.1 edge to an existing candidate-v1.1 channel

yq eval 'select(.schema=="olm.channel" and .name == "candidate-v1.1").entries += [{"name" : "testoperator.v1.1.1"}]' fbc.yaml

produces updated candidate-v1.1 channel:

...
entries:
 - name: testoperator.v1.1.0
 replaces: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
 - name: testoperator.v1.1.1
name: candidate-v1.1
package: testoperator
schema: olm.channel
...

Warning

This example illustrates how it is possible to inadvertently create multiple channel heads opm validate result for this output would be:

FATA[0000] invalid index:
└── invalid package "testoperator":
 └── invalid channel "candidate-v1.1":
 ├── multiple channel heads found in graph: testoperator.v1.1.0, testoperator.v1.1.1

basic

Add a new testoperator.v0.2.1 bundle pullspec to the existing stable channel

yq eval 'select(.schema == "olm.template.basic").entries[] |= select(.schema == "olm.channel" and .name == "stable").entries += [{"name" : "testoperator.v0.2.1", "replaces": "testoperator.v0.2.0"}]' basic.yaml

produces updated stable channel:

...
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator.v0.1.0
 - name: testoperator.v0.2.0
 replaces: testoperator.v0.1.0
 - name: testoperator.v0.2.1
 replaces: testoperator.v0.2.0
...

semver

Add a new testoperator.v1.1.1 bundle pullspec to the Candidate channel archetype

yq eval '.candidate.bundles += [{"image" : "quay.io/organization/testoperator:v1.1.1"}]' semver.yaml

produces updated Candidate archetype contents:

...
candidate:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.0
 - image: quay.io/organization/testoperator:v1.0.1
 - image: quay.io/organization/testoperator:v1.1.0
 - image: quay.io/organization/testoperator:v1.1.1
...

Adding a new ‘replaces’ link between two existing bundles

Note

This operation is not possible for the semver template, which exclusively leverages semver ordering to generate an opinonated upgrade graph.

FBC

Adding a new testoperator.v1.1.1 bundle version edge with a replaces link to its predecessor testoperator.v1.1.0 version which already exists in the channel.

yq eval 'select(.schema == "olm.channel" and .name == "candidate-v1.1").entries += [{"name" : "testoperator:v1.1.1", "replaces": "testoperator:v1.1.0"}]' fbc.yaml

produces updated candidate-v1.1 channel:

...
entries:
 - name: testoperator.v1.1.0
 replaces: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
 - name: testoperator:v1.1.1
 replaces: testoperator:v1.1.0
name: candidate-v1.1
package: testoperator
schema: olm.channel
...

basic

Adding a new testoperator.v0.1.1 bundle version edge with a replaces link to its predecessor testoperator.v0.1.0 version which already exists in the channel.

yq eval 'select(.schema == "olm.template.basic").entries[] |= select(.schema == "olm.channel" and .name == "stable").entries += [{"name" : "testoperator:v0.1.1", "replaces": "testoperator:v0.1.0"}]' basic.yaml

produces updated stable channel:

...
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator.v0.1.0
 - name: testoperator.v0.2.0
 replaces: testoperator.v0.1.0
 - name: testoperator:v0.1.1
 replaces: testoperator:v0.1.0
...

Removing a specific bundle version

Note

This operation is not possible for the semver template, which exclusively leverages semver ordering to generate an opinonated upgrade graph.

FBC

Remove the upgrade edge from the example candidate-v1.1 channel which refers to bundle version testoperator.v1.1.0.

yq eval 'del(select(.schema == "olm.channel" and .name == "candidate-v1.1" ).entries[]| select(.name == "testoperator.v1.1.0"))' fbc.yaml

produces updated candidate-v1.1 channel:

...
entries: []
name: candidate-v1.1
package: testoperator
schema: olm.channel
...

Warning

Please note that removing the only edge for a channel as in this example will yield an explicitly empty array. This will produce an error in opm validate.

basic

Remove the upgrade edge from the example stable channel which refers to bundle version testoperator.v0.2.0.

yq eval 'select(.schema == "olm.template.basic").entries[] |= del(select(.schema == "olm.channel" and .name == "stable").entries[]| select(.name == "testoperator.v0.2.0"))' basic.yaml

produces updated stable channel:

...
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator.v0.1.0
...

Substituting a bundle version in the upgrade graph

FBC

For all graph edges, replaces instances of testoperator.v1.1.0 with a different bundle version testoperator.v1.1.0-CVE

yq '(.. | select(has("entries") and .entries[].name == "testoperator.v1.1.0" ).entries[]).name = "testoperator.v1.1.0-cve"' fbc.yaml

produces updated channels:

---
defaultChannel: stable-v1.0
name: testoperator
schema: olm.package
---
entries:
 - name: testoperator.v1.0.0
 - name: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
name: candidate-v1.0
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.1.0-cve
 replaces: testoperator.v1.0.1
 skips:
 - testoperator.v1.0.0
name: candidate-v1.1
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.0.1
name: fast-v1.0
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.1.0-cve
 replaces: testoperator.v1.0.1
name: fast-v1.1
package: testoperator
schema: olm.channel
---
entries:
 - name: testoperator.v1.0.1
name: stable-v1.0
package: testoperator
schema: olm.channel

basic

For all channels, replace instances of testoperator:v0.1.0 with testoperator:v0.1.0-CVE

yq '(..| select(. == "testoperator.v0.1.0")) |="testoperator.v0.1.0-CVE"' basic.yaml

produces updated channels:

...
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator.v0.1.0-CVE
 - name: testoperator.v0.2.0
 replaces: testoperator.v0.1.0-CVE
...

semver

For all channels, replace instances of quay.io/organization/testoperator:v1.1.0 with quay.io/organization/testoperator:v1.1.0-CVE

yq '(..| select(has("image") and .image == "quay.io/organization/testoperator:v1.1.0")).image = "quay.io/organization/testoperator:v1.1.0-cve"' semver.yaml

produces updated template:

schema: olm.semver
generatemajorchannels: false
generateminorchannels: true
candidate:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.0
 - image: quay.io/organization/testoperator:v1.0.1
 - image: quay.io/organization/testoperator:v1.1.0-cve
fast:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.1
 - image: quay.io/organization/testoperator:v1.1.0-cve
stable:
 bundles:
 - image: quay.io/organization/testoperator:v1.0.1

Introducing a new replacement relationship in the upgrade graph

Note

This operation is not possible for the semver template, which exclusively leverages semver ordering to generate an opinonated upgrade graph.

FBC

Substitute an existing ‘replaces’ link target for testoperator.v1.1.0 with a different bundle version testoperator.v1.0.0.

yq eval 'select(.schema == "olm.channel" and .name == "candidate-v1.1").entries |= [{"name" : "testoperator:v1.1.0", "replaces": "testoperator:v1.0.0"}]' fbc.yaml

produces updated candidate-v1.1 channel:

...
entries:
 - name: testoperator:v1.1.0
 replaces: testoperator:v1.0.0
name: candidate-v1.1
package: testoperator
schema: olm.channel
...

basic

Substitute an existing ‘replaces’ link target for testoperator.v0.2.0 with a different bundle version testoperator.v0.1.5.

yq eval '.entries[] |= select(.schema == "olm.channel" and .name == "stable").entries |= [{"name" : "testoperator:v0.2.0", "replaces": "testoperator:v0.1.5"}]' basic.yaml

produces updated stable channel:

...
 - schema: olm.channel
 package: testoperator
 name: stable
 entries:
 - name: testoperator:v0.2.0
 replaces: testoperator:v0.1.5
...

Docs: OLM Performance Profiling Instrumentation

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Background

OLM utilizes the pprof package from the standard go library to expose performance profiles for the OLM and Catalog Operator.

Due to the sensitive nature of profiling data, the profiling endpoints will reject any clients that do not present a verifiable certificate. Both operators must be configured with a serving certificate and a client CA bundle in order to access the profiling endpoints.

This document will dive into the steps to enable olm performance profiling and retrieving pprof data from each component.

Enabling Performance Profiling

Creating a Certificate

A valid server certificate must be created for each component before the Performance Profiling functionality can be enabled. If you are unfamiliar with certificate generation, we recommend using the OpenSSL tool-kit and refer to the request certificate documentation.

Once you have generated a private and public key, this data should be stored in a TLS Secret:

$ export PRIVATE_KEY_FILENAME=private.key # Replace with the name of the file that contains the private key you generated.
$ export PUBLIC_KEY_FILENAME=certificate.crt # Replace with the name of the file that contains the public key you generated.

$ kubectl -n my-namespace create secret tls my-name --cert=$PUBLIC_KEY_FILENAME --key=$PRIVATE_KEY_FILENAME

Updating OLM to Use the TLS Secret

Patch the OLM or Catalog Deployment’s pod template to use the generated TLS secret:

Defining a volume and volumeMount
Adding the client-ca, tls-key and tls-cert arguments
Replacing all mentions of port 8080 with 8443
Updating the livenessProbe and readinessProbe to use HTTPS as the scheme

The steps to patch an existing OLM deployment can be seen below:

$ export TLS_SECRET=my-tls-secret
$ export CERT_PATH=/var/run/secrets # Define where to mount the certs.
# Set Deployment name to olm-operator or catalog-operator
$ export DEPLOYMENT_NAME=olm-operator

$ kubectl patch deployment $DEPLOYMENT_NAME -n olm --type json -p='[
 # Mount the secret to the pod
 {"op": "add", "path": "/spec/template/spec/volumes", "value":[{"name": '$TLS_SECRET', "secret": {"secretName": '$TLS_SECRET'}}]},
 {"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value":[{"name": '$TLS_SECRET', "mountPath": '$CERT_PATH'}]},

 # Add startup arguments
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--client-ca"},
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-key"},
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.key"},
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"--tls-cert"},
 {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value":"'$CERT_PATH'/tls.crt"},

 # Replace port 8080 with 8443
 {"op": "replace", "path": "/spec/template/spec/containers/0/ports/0", "value":{"containerPort": 8443}},
 {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/port", "value":8443},
 {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/port", "value":8443},

 # Update livenessProbe and readinessProbe to use HTTPS
 {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/scheme", "value":"HTTPS"},
 {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/httpGet/scheme", "value":"HTTPS"},
]'
deployment.apps/olm-operator patched

Accessing PPROF Data

You will need to be able to access OLM port, for dev purposes the following commands may prove useful:

# Set Deployment name to olm-operator or catalog-operator
$ export DEPLOYMENT_NAME=olm-operator
$ kubectl port-forward deployment/$DEPLOYMENT_NAME 8443:8443 -n olm

You can then curl the OLM /debug/pprof endpoint to retrieve default pprof profiles like so:

$ curl https://localhost:8443/debug/pprof/heap --cert certificate.crt --key private.key --insecure -o olm-heap

$ go tool pprof --top olm-heap

Please review the official pprof documentation to learn more about pprof.