What are Kubernetes Custom Resource Definitions (CRDs)?
Custom resource definitions (CRDs) are Kubernetes API extensions that can define new object types. Pods, ReplicaSets, ConfigMaps, and Ingresses are examples of common built-in resources. CRDs allow you to add entirely new types to this list and then manage them using familiar Kubernetes tools like Kubectl.
The CRD mechanism is intentionally abstract and can be used in multiple ways to store data and create new functionality. You'll find custom resources in many popular community tools: Cert-Manager defines objects that represent SSL certificates and issuers, while Helm represents graphs as their own CRD.
In this section:
What makes a resource?
Kubernetes resources define the types of data you can store in your cluster. They are accessible through the Kubernetes API which provides endpoints to create, list, and modify items in each resource type.
You can add custom resources to store your own arbitrary data in your cluster. Items you create will be stored by the etcd control plane component, alongside built-in resource instances. Custom resources are automatically displayed by the API, so you don't need to configure your own tools to create item instances.
CRDs act as simple data structures by default. Although CRDs in nature often have their own behaviors, this is not provided by the CRD mechanism. Custom Kubernetes controllers and operators can be used to implement functionality around custom resources. Without a controller, the items in a CRD will still exist as static data in the cluster that you can interact with through the CRUD endpoints provided by the Kubernetes API.
CRDs are dynamic components that can be created and deleted at any time. Some object types included in Kubernetes are also implemented as CRDs, which provides greater modularity at the core of the cluster.
Creation of a CRD
CRDs are themselves a type of Kubernetes object. You create them the same way as any other resource, by writing a YAML file and applying it to your cluster:
apiVersion: apiextensions.k8s.io/v1 child: CustomResourceDefifinish metadata: name: custom-apps.crds.example.com spec: group: crds.example.com versions: -name: v1 served: true storage: true schedule: openAPIV3Schema: type: object properties: spec: type: object properties: app-name: type: string App-version: type: string release-count: type: integer scope: Namespaced names: plural: custom apps singular: custom app child: Custom App
Use Kubectl to add the CustomApp CRD to your cluster:
$ kubectl apply -f custom-apps-crd.yaml customresourcedefinition.apiextensions.k8s.io/custom-apps.crds.example.com created
The YAML file defines a CRD that can be used to store application data. CRDs need a metadata.name
et spec.group
field in a specific format: the group takes the form of a sub-domain to which the CRD belongs. The same subdomain must be included in the CRDs metadata.name
. The value of names.plural
is added as a new subdomain component to create the final version metadata.name
.
La spec.names
The field defines how you will refer to the CRD when using the Kubernetes API and Kubectl commands. In this example, you can run kubectl get custom-apps
et kubectl get custom-app/example-app
to interact with objects that use the CRD. When you create a new object as a YAML file, you must set kind: CustomApp
to make it an instance of the CRD.
The CRD is configured as an object type at the namespace level by the scope
field. You can use Cluster
as an alternate value for this field to create objects that exist at the cluster level, outside of any namespace.
The data associated with your custom objects is defined in the spec.versions.schema.openAPIV3Schema
field. Each "version" listed creates a new version of the CRD API that you can reference with the apiVersion
in your resource YAML files. CRD data is configured using OpenAPI properties; here, each "custom application" in your cluster should have app-name
, app-version
et release-count
properties defined in its YAML spec
.
Using your CRD
Provisioning API endpoints for a new CRD can take a few minutes. Check the progress by retrieving the CRD details with Kubectl:
$ kubectl describe crd custom-apps.crds.example.com ... Status: Accepted Names: Kind: CustomApp List Kind: CustomAppList Plural: custom-apps Singular: custom-app Conditions: Last Transition Time: 2022-04-04T13: 29:24Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: 2022-04-04T13:29:24Z Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established ...
The CRD is ready to use when you see Type: Established
towards the end of the command output. Kubernetes will accept requests to the CRD API endpoint. In this case, the base API URL will be:
/apis/custom-apps.crds.example.com/v1/namespaces/*/custom-apps/…
You can now use Kubectl to view objects that were created with the CRD:
$ kubectl get custom-apps No resources found in default namespace.
Although no object yet exists, Kubernetes now knows that it has a resource type called custom-apps
.
To create a "custom application" object, write a new YAML file with kind: CustomApp
. There apiVersion
should be set to the group name and API version provided by the CRD. Within the spec
include the properties you defined in the CRD schema.
apiVersion: crds.example.com/v1 child: Custom App metadata: name: demo-app-1 spec: app-name: demo app App-version: 1.1.0 release-count: 5
Use Kubectl to add the object to your cluster:
$ kubectl apply -f custom-app.yaml customapp.crds.example.com/demo-app created
You can now retrieve the object details using the usual Kubectl commands:
$ kubectl describe custom-app/demo-app-1 Name: demo-app Namespace: default Labels: Notes: API Version: crds.example.com/v1 Kind: CustomApp ... Spec: App - Name: demo-app App - Version: 1.1.0 Release - Count: 5 ... Events:
You have a working custom resource that now stores some data in your Kubernetes cluster. You can remove the CRD by deleting it in Kubectl; this will automatically clean up all objects that use it.
$ kubectl delete crd custom-apps.crds.example.com customresourcedefinition.apiextensions.k8s.io "custom-apps.crds.example.com" deleted
Building Declarative APIs with CRDs
This CRD does not add any functionality to the cluster. It stores data, provides an API to interact with it, and can be referenced by other objects. CRDs become more powerful when paired with a custom controller that can take responsibility for managing them.
Controllers track resources and take action in response to their events. Writing a controller for your CRDs allows you to turn them into declarative APIs that drive real change within your cluster. Your objects can represent the desired state, instead of the precise current state.
Cert-Manager uses this pattern to automatically acquire SSL certificates when new CertificateRequest objects appear in your cluster. Within the Kubernetes core, nodes pull and run container images in response to spawning pods. Controllers allow you to attach behavior to your own CRDs, so adding a "custom application" could result in its configuration being retrieved from an external service. You can start building controllers using the Go SDK to integrate your own code with the Kubernetes runtime.
When to use CRDs?
CRDs are best used to manage data internal to your organization, team, or project. They are designed to represent clearly defined schemas, either as static values or as declarative APIs backed by a custom controller implementation.
Advanced features allow you to implement validation routines for fields in your schema. You can also use finalizers to manage object deletions and adopt the versioning system to manage changes to your CRD definitions.
CRDs sometimes overlap with Kubernetes ConfigMaps. They are built-in objects for storing generic configuration data associated with your applications. A ConfigMap is appropriate when consuming the values at a specific place in your cluster, such as a pod that accesses database settings as environment variables injected from a ConfigMap.
CRDs are intended for use when data needs to be a first-class citizen in your cluster. You can create multiple instances of the CRD's resource type, interact directly with them using Kubectl, and create your own structured schemas that guide users to entering correct values. They may be a better choice when the data exists independently of anything else in your cluster.
In this section:
Kubernetes Custom Resource Definitions (CRDs) define new object types that you can use with the Kubernetes API and Kubectl. Each CRD gets its own versioned API, has a structured schema, and can be used to implement new features in the cluster when supported by a companion controller.
CRDs can sometimes seem complicated and reserved for advanced situations. This must not be the case. Simple CRDs for storing static values in your cluster are easy to create, as illustrated by the "custom application" example above. They can be used to contain autonomous cluster data so that it receives first-class treatment in Kubernetes.
It is also important to recognize where CRDs do not fit. Built-in objects such as ConfigMaps and Secrets are designed to accommodate most forms of configuration that will be used directly by your app's pods. Writing a CRD that defines your application's configuration file schema is usually unnecessary and more difficult to maintain over time because you won't benefit from ConfigMap features such as automatic continuous updates and injection of environment variables.