Importing an External OpenSearch Cluster

You can bring existing clusters from your external OpenSearch installations to Oracle Cloud Infrastructure. This enables you to take advantage of the managed service capabilities provided by Search with OpenSearch.

This topic describes the process to bring your existing OpenSearch clusters to Oracle Cloud Infrastructure, and includes the following tasks:

  1. Take a snapshot of the cluster.
  2. Upload snapshot to Object Storage.
  3. Create the destination cluster in Search with OpenSearch.
  4. Configure a dynamic policy with the required permissions.
  5. Register the repository.
  6. Restore the snapshot.

Prerequisites

Before proceeding with the steps described in this topic, you need a tenancy that you can work in. The user account you use to log into the tenancy requires sufficient access to configure service and user permissions for Oracle Cloud Infrastructure services and resources. If you need more information about how to get started with Oracle Cloud Infrastructure, see the following resources:

To complete these procedures, you need access to create an Object Storage bucket and upload a file to the bucket. If you need to configure these permissions for a specific user, see example policies in Let Object Storage admins manage buckets and objects and Let users write objects to Object Storage buckets.

You also need to configure the service and user permissions required to create OpenSearch clusters.

The following policy example includes the required permissions:

Allow service opensearch to manage vnics in compartment <YOUR_COMPARTMENT>
Allow service opensearch to manage vcns in compartment <YOUR_COMPARTMENT>
Allow service opensearch to use subnets in compartment <YOUR_COMPARTMENT>
Allow service opensearch to use network-security-groups in compartment <YOUR_COMPARTMENT>
Allow <YOUR_USER> to manage opensearch-family in compartment <YOUR_COMPARTMENT>

The service permissions included in this example are required as specified. You can configure the user permissions, specified in the last line in this example, with more granularity, find policy statement examples demonstrating this in Sample Policies.

Take a Snapshot of the Cluster

Generate a snapshot of your cluster from your external OpenSearch installation. See OpenSearch - Take and restore snapshots for an overview of cluster snapshots and how to generate them.

Upload Snapshot to Object Storage Bucket

After the snapshot is generated, upload the snapshot file to an Object Storage bucket in your tenancy. For a tutorial that walks you through how to create a bucket and upload files to the bucket, see Putting Data into Object Storage.

Make note of the following information for the bucket you uploaded the snapshot to. You can find this information on the Bucket Details page in the Console.

  • Namespace
  • Compartment name
  • Bucket name
  • Repository name
  • Snapshot file name

Also make note of the compartment OCID.

To get the compartment OCID:

  1. On the Bucket Details page click the link for Compartment in Cluster Information.
  2. Click Copy for OCID in Compartment Information.

Create Destination Cluster

Before the snapshot from your external OpenSearch installation can be imported into Oracle Cloud Infrastructure, you need to create an OpenSearch cluster in Search with OpenSearch. This cluster will be the destination cluster that the snapshot is imported into.

See Creating an OpenSearch Cluster for a step-by-step procedure describing how to create the cluster, including the required prerequisites.

Important

When you create the cluster, ensure that you configure the cluster with the same node configuration as the source cluster that you are importing the snapshot for.

Make note of the following information for the cluster:

  • OCID
  • Compartment name

You can find this information in the Console on the Cluster Details page, see Getting an OpenSearch Cluster's Details. Also make note of the compartment OCID.

To get the compartment OCID:

  1. On the Cluster Details page click the link for Compartment in Cluster Information>.
  2. Click Copy for OCID in Compartment Information.

IAM Policy

You need to configure permissions to provide the OpenSearch cluster access to the Object Storage bucket in your tenancy.

The following policy example includes the required permissions:

DEFINE tenancy opensearch-tenancy as <OpenSearch_Tenancy_ID>
ADMIT resource opensearch opensearchsnapshots of tenancy opensearch-tenancy to manage object-family in compartment <snapshot_bucket_compartment> where ALL {request.principal.clusterid='<cluster_OCID>', request.principal.ownertenant='<customer_tenancy_OCID>', request.principal.ownercompartment='<customer_compartment_OCID>', target.bucket.name='<snapshot_bucket_name>'}

<OpenSearch_Tenancy_ID> is the OCID for the OCI Search with OpenSearch tenancy. You can find this value on the Cluster details page for your cluster in the Opensearch Tenancy OCID field under the Additional Info section.

<customer_tenancy_OCID> is the OCID for your tenancy where you created the OpenSearch cluster.

<customer_compartment_OCID> is the OCID for the compartment where the OpenSearch cluster is located.

Register the Repository

Register the repository, as shown in the following example:

PUT _snapshot/<repository_name>
{
   "type": "oci",
      "settings": {
      "client": "default",
      "endpoint": "<objectstorage_endpoint>",
      "bucket": "<bucket_name>",
      "namespace": "<namespace>",
      "authType": "RESOURCE_PRINCIPAL",
      "bucket_compartment_id": "<bucket_compartment_OCID>",
      "forceBucketCreation": true
   }
}

The following table describes the parameters to specify when you register the repository.

Parameter Description

repository name

The name you assign to the repository for the snapshot.

endpoint

The endpoint for the Object Storage bucket.

For example, the US East (Ashburn) region Object Storage endpoint is https://objectstorage.us-ashburn-1.oraclecloud.com.

namepace

The Object Storage namespace for the bucket.

authType

Specify to use the resource principal which allows Search with OpenSearch to work on your behalf.

bucket_compartment_id

The OCID of the compartment where the bucket is located.

forceBucketCreation

Specifies to create the Object Storage bucket if it does not exist already. Default is false.

Restore Snapshot

Before you restore a snapshot, you can retrieve all the existing snapshots for a cluster. First, retrieve all the repositories registered with the cluster, as shown in the following example:

GET _snapshot/_all?pretty

Then, retrieve all the snapshots in repository, as shown in the following example:

GET _snapshot/<repository_name>?pretty

Now, restore the snapshot, as shown in the following example:

POST _snapshot/<repository_name>/<snapshot_name>/_restore
{
    "indices": "<index_name_to_include>,-<index_name_to_exclude>",
    "ignore_unavailable": true,
    "include_global_state": false,
    "rename_pattern": "(.+)",
    "rename_replacement": "restored_$1",
    "include_aliases": true
}

You can specify one or more indices to include, <index_name_to_include>, and one or more indices to exclude, -<index_name_to_exclude>.

For more information about this API operation, see Restore snapshots.