Provisioning PVCs on the Block Volume Service

You can dynamically provision a block volume using the CSI plugin specified by the oci-bv storage class's definition (provisioner: blockvolume.csi.oraclecloud.com). For example, if the cluster administrator has not created any suitable PVs that match the PVC request.

You define a PVC in a file called csi-bvs-pvc.yaml. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mynginxclaim
spec:
  storageClassName: "oci-bv"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Enter the following command to create the PVC from the csi-bvs-pvc.yaml file:

kubectl create -f  csi-bvs-pvc.yaml

The output from the above command confirms the creation of the PVC:

persistentvolumeclaim "mynginxclaim" created

Verify that the PVC has been created by running kubectl get pvc:

kubectl get pvc

The output from the above command shows the current status of the PVC:

		
NAME               STATUS   VOLUME   CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mynginxclaim       Pending                                    oci-bv         4m

The PVC has a status of Pending because the oci-bv storage class's definition includes volumeBindingMode: WaitForFirstConsumer.

You can use this PVC when creating other objects, such as pods. For example, you could create a new pod from the following pod definition, which instructs the system to use the mynginxclaim PVC as the nginx volume, which is mounted by the pod at /data.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: mynginxclaim

Having created the new pod, you can verify that the PVC has been bound to a new persistent volume by entering:

kubectl get pvc

The output from the above command confirms that the PVC has been bound:

			
NAME               STATUS    VOLUME                               CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mynginxclaim       Bound     ocid1.volume.oc1.iad.<unique_ID>     50Gi       RWO           oci-bv         4m

You can verify that the pod is using the new persistent volume claim by entering:

kubectl describe pod nginx

You can view capacity statistics for the new persistent volume using a metrics aggregation tool such as Prometheus.

A Kubernetes volume snapshot is a snapshot of a persistent volume on a storage system. You can use a volume snapshot to provision a new persistent volume. For more information, about Kubernetes volume snapshots, see Volume Snapshots in the Kubernetes documentation.

When using the CSI volume plugin to connect clusters to block volumes in the Block Volume service, you can use block volume backups to provision Kubernetes volume snapshots. You can use a volume snapshot to create a new block volume and pre-populate it with data from the block volume backup. For more information about block volume backups in the Block Volume service, see Overview of Block Volume Backups.

You can use the CSI volume plugin to provision a volume snapshot in one of two ways:

• Dynamically: You can request the creation of a backup of the block volume provisioning a persistent volume. You specify the persistent volume claim using the VolumeSnapshot object, and you specify the parameters to use to create the block volume backup using the VolumeSnapshotClass object. Dynamically provisioned volume snapshots are also referred to as dynamic volume snapshots. See Creating Dynamically Provisioned Volume Snapshots.
• Statically: You can provide details of an existing block volume backup using the VolumeSnapshotContent object. Statically provisioned volume snapshots are also referred to as static volume snapshots, and as pre-provisioned volume snapshots. Creating Statically Provisioned Volume Snapshots

When you create a dynamically provisioned volume snapshot, the same freeform tags and defined tags that were applied to the source block volume are applied to the block volume backup. However, you can use parameters to apply additional tags to the block volume backup (see Tagging Block Volume Backups).

There are a number of prerequisites to meet before creating volume snapshots for use with clusters created by Kubernetes Engine. See Prerequisites for Creating Volume Snapshots.

Note the following when creating and using volume snapshots:

• You can only create volume snapshots when using the CSI volume plugin (that is, you cannot create volume snapshots when using the FlexVolume volume plugin).
• In the case of dynamic volume backups, the CSI volume plugin creates a new block volume backup to provision a dynamic volume snapshot in the same compartment as the cluster. In the case of static volume snapshots, the block volume backup provisioning a static volume snapshot can be in a different compartment to the cluster, provided appropriate policy statements exist to enable the cluster to access that other compartment (see Prerequisites for Creating Volume Snapshots).
• You cannot use the CSI volume plugin to re-populate an existing volume with data. In other words, you cannot restore (revert) data in an existing persistent volume to an earlier state by changing the volume snapshot specified in the persistent volume claim's manifest. You can only use the CSI volume plugin to populate a new volume with data.
• When you create a block volume backup (for example, when creating a dynamic volume snapshot), the encryption key that was used when creating the block volume is also used to create the block volume backup. You cannot specify a new encryption key when creating a block volume backup. So the block volume backup uses the same encryption as the block volume it backs up.
• The size specified for a persistent volume provisioned by a volume snapshot must not be smaller than the size of the original volume from which the snapshot was created.
• When the CSI volume plugin creates a new block volume to back a persistent volume provisioned by a volume snapshot, the placement of the block volume depends on the topology requirements of the volume creation request. For example, if the CSI volume plugin creates a block volume for a pod that uses a persistent volume claim, the block volume is created in the same availability domain as the worker node on which the pod is running.
• Cross-namespace snapshots are not supported.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

ALLOW any-user to manage volume-backups in compartment <compartment-name> where request.principal.type = 'cluster'
ALLOW any-user to use volumes in compartment <compartment-name> where request.principal.type = 'cluster'

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sample-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "oci-bv"
  resources:
    requests:
      storage: 50Gi

kubectl create -f csi-mypvctobackup.yaml

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
spec:
  containers:
    - name: sample-nginx
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: sample-volume
  volumes:
  - name: sample-volume
    persistentVolumeClaim:
      claimName: sample-pvc

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: my-snapclass
driver: blockvolume.csi.oraclecloud.com
parameters:
  backupType: full
deletionPolicy: Delete

kubectl create -f csi-mysnapshotclass.yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: my-snapclass
  source:
    persistentVolumeClaimName: sample-pvc

kubectl create -f csi-mysnapshot.yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: my-static-snapshot-content
spec:
  deletionPolicy: Retain
  driver: blockvolume.csi.oraclecloud.com
  source:
    snapshotHandle: ocid1.volumebackup.oc1.iad.aaaaaa______xbd
  volumeSnapshotRef:
    name: my-static-snapshot
    namespace: default

kubectl create -f csi-mystaticsnapshotcontent.yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: my-static-snapshot
spec:
  source:
    volumeSnapshotContentName: my-static-snapshot-content

kubectl create -f csi-mystaticsnapshot.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-fromsnapshot
  namespace: default
spec:
  storageClassName: oci-bv
  dataSource:
    name: test-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

kubectl create -f csi-mypvcfromsnapshot.yaml

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod-restore
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pvc-fromsnapshot

oci.oraclecloud.com/freeform-tags: '{"<first-tag-key>":","<first-tag-value>","<second-tag-key>":"<second-tag-value>"...}'

oci.oraclecloud.com/defined-tags: '{"<tag-namespace>": {"<first-tag-key>":","<first-tag-value>","<second-tag-key>":"<second-tag-value>"...}}'

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: my-snapclass
driver: blockvolume.csi.oraclecloud.com
parameters:
  backupType: full
  oci.oraclecloud.com/freeform-tags: '{"<first-tag-key>":","<first-tag-value>","<second-tag-key>":"<second-tag-value>"...}'
deletionPolicy: Delete

A Kubernetes clone is an exact duplicate of an existing persistent volume on a storage system. You can clone an existing persistent volume to provision a new persistent volume claim. The new persistent volume contains a copy of the data from the source persistent volume, but is independent of the source persistent volume. You can use volume clones to test configuration changes quickly, without impacting a production environment. For more information about Kubernetes clones, see CSI Volume Cloning in the Kubernetes documentation..

In the Block Volume service, you can clone a block volume to create a new block volume pre-populated with data from the source block volume. For more information about cloning block volumes in the Block Volume service, see Cloning a Block Volume.

When using the CSI volume plugin to connect clusters to block volumes in the Block Volume service, you can provision a new PVC with a new block volume that has been cloned from an existing block volume provisioning another existing PVC. To indicate that you want the CSI volume plugin to clone the existing block volume for the new PVC, you specify the existing PVC as the datasource for the new PVC.

The new PVC can be used in the same way as any other PVC, and is entirely separate to the existing PVC specified as the datasource. Similarly, the new block volume and the existing block volume from which it is cloned are entirely separate resources, and can be updated, cloned, snapshot, and deleted independently of each other.

As soon as the source block volume has been cloned and a new block volume created (usually within a few seconds), the new block volume has a state of Available. All data present in the source block volume at that moment is copied to the new block volume (no subsequent changes are copied). However, the data is copied in the background, which can take up to thirty minutes depending on the size of the block volume (for example, it can take up to fifteen minutes to copy a 1 TB block volume). Therefore, to avoid the possibility of errors or data corruption, the new PVC has a state of Pending until all the data has been copied. When all the data has been copied, the new PVC is bound to the PV provisioned by the new block volume, and the new PVC has a state of Available.

There are a number of prerequisites to meet before provisioning a new persistent volume claim by cloning the block volume that is already provisioning an existing persistent volume claim. See Prerequisites for Cloning an Existing Block Volume to Provision a New PVC.

Note the following when cloning a block volume to provision a new PVC:

• The CSI volume plugin creates the new block volume in the same availability domain, region, and tenancy as the source block volume.
• The new block volume created by the CSI volume plugin can itself be cloned as soon at it has a state of Available.
• Any topology requirements in a volume creation request are ignored. For example, if the CSI volume plugin clones a block volume for a pod that uses a persistent volume claim, the new block volume is created in the same availability domain as the worker node on which the pod is running.
• You cannot use the CSI volume plugin to clone block volumes created by the FlexVolume volume plugin.
• You cannot delete the source PVC while cloning is in progress. Similarly, you cannot delete a source block volume while data is being copied to a block volume that has been cloned from it.
• Cross-namespace cloning is not supported. The new PVC and the source PVC must both be in the same namespace.
• You do not have to explicitly specify a storage class for the new PVC. If you do explicitly specify a storage class for the new PVC, the storage class you specify can be different to the storage class specified for the source PVC. If you do not specify a storage class for the new PVC, the default storage class is used.

For example, you define a persistent volume claim named my-clone-pvc in a file called csi-myclonepvc.yaml. The my-clone-pvc persistent volume claim is provisioned by a block volume created by cloning the block volume that provisions the my-source-pvc persistent volume claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-clone-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: custom-clone-storage-class
  resources:
    requests:
      storage: 50Gi
  dataSource:
    kind: PersistentVolumeClaim
    name: my-source-pvc

Create the persistent volume claim:

kubectl create -f csi-myclonepvc.yaml

You can use the persistent volume claim when defining other objects, such as pods. For example, the following pod definition instructs the system to use the my-clone-pvc persistent volume claim as the nginx volume, which is mounted by the pod at /sample-volume.

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
spec:
  containers:
    - name: sample-nginx
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: sample-volume
  volumes:
  - name: sample-volume
    persistentVolumeClaim:
      claimName: my-clone-pvc

Having created the new pod, the my-clone-pvc persistent volume claim is bound to a new persistent volume provisioned by a block volume that has been cloned from the block volume provisioning the my-source-pvc persistent volume claim.

You can create a PVC from an existing block volume or a block volume backup for use by the FlexVolume volume plugin. For example, if the cluster administrator has created a block volume backup for you to use when provisioning a new persistent volume claim. Such a block volume backup might come with data ready for use by other objects such as pods.

You define a PVC in a file called flex-pvcfrombackup.yaml. You use the volume.beta.kubernetes.io/oci-volume-source annotation element to specify the source of the block volume to use when provisioning a new persistent volume claim using the FlexVolume volume plugin. You can specify the OCID of either a block volume or a block volume backup as the value of the annotation. In this example, you specify the OCID of the block volume backup created by the cluster administrator. For example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myvolume
  annotations:
    volume.beta.kubernetes.io/oci-volume-source: ocid1.volumebackup.oc1.iad.abuw...
spec:
  selector:
    matchLabels:
      topology.kubernetes.io/zone: US-ASHBURN-AD-1
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Note that the flex-pvcfrombackup.yaml file includes the matchLabels element, which is only applicable in the case of the FlexVolume volume plugin.

Enter the following command to create the PVC from the flex-pvcfrombackup.yaml file:

kubectl create -f flex-pvcfrombackup.yaml

The output from the above command confirms the creation of the PVC:

persistentvolumeclaim "myvolume" created

Verify that the PVC has been created and bound to a new persistent volume created from the volume backup by entering:

kubectl get pvc

The output from the above command shows the current status of the PVC:

			
NAME           STATUS    VOLUME                               CAPACITY   ACCESSMODES   STORAGECLASS   AGE
myvolume       Bound     ocid1.volume.oc1.iad.<unique_ID>     50Gi       RWO           oci            4m

You can use the new persistent volume created from the volume backup when defining other objects, such as pods. For example, the following pod definition instructs the system to use the myvolume PVC as the nginx volume, which is mounted by the pod at /data.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: myvolume

Having created the new pod, you can verify that it is running and using the new persistent volume claim by entering:

kubectl describe pod nginx

The Oracle Cloud Infrastructure Block Volume service always encrypts all block volumes and volume backups at rest by using the Advanced Encryption Standard (AES) algorithm with 256-bit encryption. By default all volumes and their backups are encrypted using the Oracle-provided encryption keys. Each time a volume is cloned or restored from a backup the volume is assigned a new unique encryption key.

You have the option to encrypt all of your volumes and their backups using the keys that you own and manage using the Vault service, for more information see Key Management. If you do not configure a volume to use the Vault service or you later unassign a key from the volume, the Block Volume service uses the Oracle-provided encryption key instead. This applies to both encryption at-rest and paravirtualized in-transit encryption.

All the data moving between the instance and the block volume is transferred over an internal and highly secure network. If you have specific compliance requirements related to the encryption of the data while it is moving between the instance and the block volume, the Block Volume service provides the option to enable in-transit encryption for paravirtualized volume attachments on virtual machine (VM) instances. Some bare metal shapes support in-transit encryption for the instance's iSCSI-attached block volumes.

For more information about block volume encryption, and in-transit encryption support, see Block Volume Encryption.

When Kubernetes PVCs are backed by the Block Volume service, you choose how block volumes are encrypted by specifying:

• The master encryption key to use, by setting the kms-key-id property in the Kubernetes storage class's definition. You can specify the OCID of a master encryption key in the Vault service.
• How the block volume is attached to the compute instance, by setting the attachment-type property in the Kubernetes storage class's definition to either iscsi or paravirtualized.
• Whether in transit encryption is enabled for each node pool in a cluster, by setting the node pool's isPvEncryptionInTransitEnabled property (using the CLI, the API, or the node pool's Use in transit encryption: option in the Console).

The interaction of the settings you specify determines how block volumes are encrypted, as shown in the table:

Before you can create a cluster for which you want to manage the master encryption key yourself, you have to:

• Create a suitable master encryption key in Vault (or obtain the OCID of such a key). See Managing Keys.
• Create a policy granting access to the master encryption key. See Create Policy to Access User-Managed Encryption Keys for Encrypting Boot Volumes, Block Volumes, and/or File Systems.

For more information about key rotation in the Vault service, see Rotating a Vault Key.

When a PVC is created using the CSI volume plugin (provisioner: blockvolume.csi.oraclecloud.com), you can expand the volume size online. By doing so, you make it possible to initially deploy applications with a certain amount of storage, and then subsequently increase the available storage without any downtime.

If you want to support storage request increases, set allowVolumeExpansion: true in the definition of the storage class that you specify for the PVC. For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-bv
provisioner: blockvolume.csi.oraclecloud.com
parameters:
  attachment-type: "paravirtualized"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The default oci-bv storage class for the CSI volume plugin has allowVolumeExpansion: true by default.

To expand the size of a volume, edit the PVC manifest and update the volume size, and then apply the manifest. When the disk is next rescanned to enable the operating system to identify the expanded volume size (which can take a few minutes), the increased storage automatically becomes available to pods using the PVC. The pods do not have to be restarted.

Enter the following command to confirm the PVC has been bound to a newly-enlarged block volume:

kubectl get pvc <pvc_name> -o yaml

Note the following:

• Volume expansion is supported in clusters running Kubernetes 1.19 or later.
• The default oci-bv storage class for the CSI volume plugin is configured with allowVolumeExpansion: true in clusters running Kubernetes 1.19 or later. Definitions of oci-bv storage classes in existing clusters running Kubernetes 1.19 or later are automatically edited to set allowVolumeExpansion: true.
• You cannot reduce the size of a block volume. You can only specify a larger value than the block volume’s current size. If you update a PVC manifest to request less storage than previously requested, the storage request fails.
• For more information about increasing block volume sizes in the Block Volume service, see Resizing a Volume. In particular, note the recommendation to create a backup before resizing a block volume.

Block volumes in the Block Volume service can be configured for different levels of performance, according to expected workload I/O requirements. Block volume performance is expressed in volume performance units (VPUs). A number of performance levels are available, including:

• Lower Cost (0 VPUs)
• Balanced (10 VPUs)
• Higher Performance (20 VPUs)
• Ultra High Performance (between 30 VPUs and 120 VPUs)

By default, block volumes are configured for the Balanced performance level (10 VPUs). For more information about the different block volume performance levels, see Block Volume Performance Levels.

When you define a PVC using the CSI volume plugin (provisioner: blockvolume.csi.oraclecloud.com), you can specify a different block volume performance level in the storage class definition that is appropriate for the expected workload.

Note that you cannot subsequently change the performance level of a block volume backing a PVC. Instead, you have to define a new storage class, set the performance level as required, and create a new PVC provisioned by that new storage class.

Block volumes in the Block Volume service can be configured for different types of file system. The most appropriate file system to use depends on (among other things) the expected file size, and the number of files to be processed. A number of file system types are available, including:

• ext3: The ext3 file system type includes journaling capabilities to improve reliability and availability. Consistency checks after a power failure or an uncontrolled system shutdown are unnecessary.
• ext4: In addition to ext3 features, the ext4 file system type supports extents (contiguous physical blocks), pre-allocation, delayed allocation, faster file system checking, more robust journaling, and other enhancements.
• XFS: The XFS file system is a high-performance journaling file system type, which provides high scalability for I/O threads, file system bandwidth, file and file system size, even when the file system spans many storage devices.

The ext3 and ext4 file systems are generally considered better-suited for applications that use a single read/write thread and small files. Whereas, the XFS file system is generally considered better-suited for applications that have multiple read/write threads and larger files.

When a PVC is created using the CSI volume plugin (provisioner: blockvolume.csi.oraclecloud.com), you can specify a file system type for the block volume that is appropriate for the expected workload.

By default, block volumes are configured with an ext4 file system. If the ext4 file system is not the most appropriate file system for the expected workload, you can specify an alternative file system type in the storage class definition.

To create a PVC backed by a block volume with an ext3 or XFS file system, set the fstype parameter in the custom storage class definition as follows:

• for ext3, set csi.storage.k8s.io/fstype: ext3
• for XFS, set csi.storage.k8s.io/fstype: xfs

For example, to create a PVC backed by a block volume with an ext3 file system:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: oci-bv-ext3
provisioner: blockvolume.csi.oraclecloud.com
parameters:
  csi.storage.k8s.io/fstype: ext3
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForConsumer

Create a manifest for a PVC provisioned by the oci-bv-ext3 storage class and include a request for storage. For example, in a file called csi-bvs-pvc-fstype.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: oci-bv-claim-ext3
spec:
  storageClassName: oci-bv-ext3
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Enter the following command to create the PVC from the csi-bvs-pvc-fstype.yaml file:

kubectl apply -f csi-bvs-pvc-fstype.yaml

Having created the PVC, you can use it when creating other objects, such as pods. For example, you could create a new pod from the following pod definition, which instructs the system to use the oci-bv-claim-ext3 PVC as the nginx volume, which is mounted by the pod at /data.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
      volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: oci-bv-claim-ext3

Having created the new pod, you can verify that the PVC has been bound to a new persistent volume by entering:

kubectl get pvc

The output from the above command confirms that the PVC has been bound:

			
NAME                    STATUS    VOLUME                               CAPACITY   ACCESSMODES   STORAGECLASS        AGE
oci-bv-claim-ext3       Bound     ocid1.volume.oc1.iad.<unique_ID>     50Gi       RWO           oci-bv-ext3         4m

You can verify that the pod is using the new persistent volume claim by entering:

kubectl describe pod nginx

Note that you cannot subsequently change the file system of a block volume backing a PVC. Instead, you have to define a new storage class, set the file system as required, and create a new PVC provisioned by that new storage class.