Troubleshooting Ultra High Performance Volume Attachments
This topic covers troubleshooting steps you can take as well as prerequisites to verify for volumes configured for the Ultra High Performance level where either the volume fails to attach or the volume attachment is not multipath-enabled.
Troubleshooting Volume Attachment Failures
The Block Volume Management plugin is required for
volumes configured for Ultra High Performance , attached using the iSCSI
attachment type. If the volume fails to attach to the instance, the issue is likely
caused by incorrect configuration for the Block Volume Management plugin. See the
troubleshooting suggestions in this section for these issues.
If you have not configured permissions correctly for the Block Volume Management plugin, the volume will
fail to attach to the instance.
Details
The volume will not show up as attached in the Console and you will see a
NotAuthorizedOrNotFound error message in the Block Volume Management plugin log.
The Block Volume Management plugin log is
located in:
The following is an sample error log entry for this issue:
2021/08/13 09:14:25.864932 compute_client_command.go:255: Updating volume attachment to the state LOGIN_SUCCEEDED ...
2021/08/13 09:14:26.155473 compute_client_command.go:260: Service error:NotAuthorizedOrNotFound.
volume attachment ocid1.volumeattachment.oc1.iad.<volume-attachment_ID> not found.
http status code: 404. Opc request id: <request_ID>
Cause
The Block Volume Management plugin does not
have sufficent permissions to send the iSCSI login status notification to the
service.
Resolution
To configure permissions for the Block Volume Management plugin:
Create Dynamic Group: Create a dynamic group with the matching rules
in the following code sample, to include all instances in the specified
compartments:
ANY {instance.compartment.id = 'ocid1.tenancy.oc1..<tenancy_ID>', instance.compartment.id = 'ocid1.compartment.oc1..<compartment_OCID>'
Configure Policy for Dynamic Group: Configure a policy granting
permissions to the dynamic group created in the previous step to enable the
instance agent access to call the Block Volume service to retrieve the
attachment configuration.:
Allow dynamic-group InstantAgent to use instances in tenancy
Allow dynamic-group InstantAgent to use volume-attachments in tenancy
The compute instance must have either a public IP address or a service gateway to be able
to connect to Oracle services or the volume will fail to attach.
Details
The volume will not show up as attached in the Console and you will see a
user agent can not be blank error message in the Block Volume Management plugin log.
The Block Volume Management plugin log is
located in:
The following is an sample error log entry for this issue:
2021/10/15 22:16:07.881953 compute_client_command.go:255: Updating volume attachment to the state LOGIN_SUCCEEDED ...
2021/10/15 22:16:07.882185 compute_client_command.go:260: user agent can not be blank
2021/10/15 22:16:07.882204 iscsi_commands_helper.go:302: user agent can not be blank
2021/10/15 22:16:07.882212 iscsi_commands_helper.go:310: user agent can not be blank
Cause
The Block Volume Management plugin cannot send
the iSCSI login status notification to the service due to the network
configuration.
Resolution
If the instance does not have a public IP address, set up a service gateway on the
virtual cloud network (VCN). The service gateway lets your instance privately access
Oracle services without exposing the data to the public internet. Here are special
notes for setting up the service gateway for the Block Volume Management plugin:
When creating the service gateway, enable the service label called All
<region> Services in Oracle Services
Network.
When setting up routing for the subnet that contains the instance, set up a
route rule with Target Type set to Service Gateway, and the
Destination Service set to All <region>
Services in Oracle Services Network.
When you attach a volume configured for the Ultra High Performance level, to
achieve the optimal performance, the volume attachment must be multipath-enabled. The
Block Volume service attempts to enable the
attachment for multipath when the volume is being attached. If not all of the
prerequisites have been addressed, the volume attachment will not be
multipath-enabled.
To check whether a volume attachment is multipath-enabled in the Console from the
Volume Details page:
Open the navigation menu and click Storage. Under Block Storage, click Block Volumes.
Click the block volume that you want to check the volume attachment for.
Click Attached Instances in the Resources section.
Check the value displayed in the Multipath column.
Yes: The volume is configured for the Ultra High
Performance level and the volume attachment is
multipath-enabled. No further action is required.
No: The volume is not configured for the Ultra High
Performance level, the volume does not need to be
multipath-enabled. No further action is required.
No with a warning icon: The volume is configured for the Ultra
High Performance level, but the volume attachment is not
multipath-enabled. To achieve optimal performance, you need to ensure
the volume is attached to a supported instance shape, and that the
required prerequisites are configured.
If the volume is configured for the Ultra High Performance level, but not
configured for multipath as required, the Multipath column will contain
No with a warning triangle, as shown for the first row in the following
screenshot:
If your volume is not configured for multipath, to troubleshoot the issue, review the
information covered in this section and address any issues raised.
You must attach a volume configured for the Ultra High Performance level to an
instance based on a supported shape, configured for at least 16 cores.
Supported Shapes
All current bare metal shapes support multipath-enabled iSCSI attachments. See for
more information Bare Metal Shapes for performance
characteristics of block volumes attached to bare metal instances.
If the volume isn't attached to an instance with a supported shape configuration, you
need to detach the volume and attach it to an instance with a supported shape
configuration.
You can also edit the existing instance so that it has the correct shape
configuration, but you need to ensure that you detach and reattach the volume after
you edit the instance.
Warning
If the instance has less than 8 OCPUs, you might see an issue,
where after you edit the instance to support multipath-enabled attachments, the
volume attachment is still not multipath-enabled, even after you detach and reattach
the volume. In this scenario, you need to re-create the instance from a supported
shape, and then attach the volume to the new instance. For more
information, see Paravirtualized volume attachment not multipath-enabled after instance is resized.
You must attach a volume configured for the Ultra High Performance level to an
instance running an image that supports multipath attachments. This includes custom
images.
Supported Images for iSCI Attachments
Only platform images running Oracle Linux or custom images based on an Oracle Linux
image support multipath attachments.
Use one of the latest platform Oracle Linux images, with an Unbreakable Enterprise
Kernel (UEK) version of UEK6U1 or higher.
For custom images, the Unbreakable Enterprise Kernel (UEK) version must also be UEK6U1 or higher. The UEK6U1 UEK is associated with the kernel major release version 5.4.17-2036, released in November, 2020. You also need to update the Storage.Iscsi.MultipathDeviceSupported property for the custom image to true and launch the instance again. For more information, see Configuring Image Capabilities for Custom Images
Supported Images for Paravirtualized Attachments 🔗
For multipath-enabled attachments, the attached instance must be running one of the following images or a custom image based on one of these images:
Oracle Linux
Ubuntu
CentOS
Windows
Note
Multipath-enabled attachments are not supported for Oracle Autonomous
Linux instances.
Use one of the latest platform Oracle Linux images, with an Unbreakable Enterprise
Kernel (UEK) version of UEK6U1 or higher.
If you have updated the shape configuration or the image for an instance to one that
supports multipath attachments, you need to detach the volume from the instance and then
attach it again to the instance.
For iSCSI attachments, if you have taken all the steps described in this topic and you are still encountering an issue with the volume attachment, use the steps described in Step 4: Generate a Diagnostic File for Oracle Cloud Agent to generate a diagnostic file and contact Oracle support. This step does not apply to paravirtualized attachments.
In certain scenarios, a volume attachment shows as multipath-enabled in the Console, however the attachment isn't actually multipath-enabled, and the volume doesn't achieve the performance expected for the Ultra High Performance level. This issue can occur when you use both the oci-utils and oci-iscsi-config tools at the same time to configure a volume.
Use one of the following methods to check to see if you're encountering this issue.
Use the multipath command to confirm whether a volume attachment is actually multipath-enabled on a Linux instance. Log in to the instance and run the multipath command with the ll tag, as follows:
# multipath -ll
If the command output returns nothing, this confirms that the instance doesn't have any multipath-enabled attachments.
If any of them have node.startup=automatic, the volume attachment isn't multipath-enabled. They must all show node.startup=manual.
Resolution
If you determine that the attachment isn't multipath-enabled, you can work around this issue using the /etc/fstab file. Update the /etc/fstab file to tell the systemd service to wait until the multipathd service is running before mounting the file system. To do this, add x-systemd.requires=multipathd.service for the volume. For example: