Cross Cluster Connections in OCI Search with OpenSearch
Learn about cross cluster connections in OCI Search with OpenSearch.
You can use cross cluster connections to search and analyze data across multiple OpenSearch clusters, enabling you to gain insights from distributed data sources. You to perform operations such as cross-cluster search, as if the data was from a single source.
Cross cluster connections are available by default with the Security plugin, but you need to configure each cluster to allow remote connections from other clusters. This involves setting up cluster connections and configuring access permissions.
When configuration a cross cluster connection in OCI Search with OpenSearch, the outbound cluster is the cluster you want to connect to other clusters. The inbound cluster is the cluster being connected to from the outbound cluster. The following table maps the OCI Search with OpenSearch terminology to the terminology used by OpenSearch for cross cluster search and cross cluster replication.
|
OCI Search with OpenSearch Term |
OpenSearch Cross Cluster Search Term |
OpenSearch Cross Cluster Replication Term |
|---|---|---|
| outbound cluster | coordinating cluster | leader cluster/index |
| inbound cluster | remote cluster | follower cluster/index |
Prerequisites
The user performing the cross cluster operation requires permissions for both the inbound and the outbound clusters. See Performing Cross Cluster Search and Performing Cross Cluster Replication for the specific cluster permissions required.
For more information about required IAM policies and permission details for Search with OpenSearch, see Search with OpenSearch IAM Policies.
Cross cluster connections are only supported in OCI Search with OpenSearch for clusters using OpenSearch 2.15 or later. This applies to both the outbound and inbound clusters.
Limits and Considerations
- You can configure a maximum of five clusters per tenancy for cross cluster connections.
- Clusters can connect to other clusters with the same OpenSearch version or one version less.
- Cross cluster connections are supported for clusters within the same OC1 region, or across all regions within the OC1 realm.
Configuring Cross Cluster Connections
Performing Cross Cluster Search
The user performing a cross cluster search requires OpenSearch permissions to query both the outbound cluster and the inbound cluster.
The following is an example of a cross cluster search query:
curl -X GET "https://<outbound_cluster_opensearch_private_IP>:9200/<inbound_cluster_alias>:<index_name>/_search?q=title:Kubernetes&pretty"For more information, see Cross-cluster search.
Performing Cross Cluster Replication
With cross cluster replication, you replicate indexes, mappings, and metadata from one OpenSearch cluster (outbound cluster) to another OpenSearch cluster (inbound cluster). All subsequent operations on the outbound cluster are replicated on the inbound cluster, such as creating, updating, or deleting documents. Use cross cluster replication for the following scenarios:
- To maintain a backup cluster that you can switch to if the primary cluster fails.
- Provide faster query performance by keeping data closer to users in geographically distributed regions.
- Distribute query and indexing loads across more than one cluster.
The following can impact cross cluster replication performance:
- Network latency between the outbound and inbound clusters.
- The size and frequency of updates to the outbound index.
- Resource availability on both the outbound cluster and the inbound clusters.
Permissions
To enable cross cluster replication, you need cluster-level permissions for replication and index-level permissions for both the outbound cluster and inbound cluster. For more information, see Map the leader and follower cluster roles.
Replication Operations
The following table lists the replication operations you can perform. See Cross-cluster replication API for more information.
|
Operation |
Description |
Request |
|---|---|---|
| Start replication | Begins the replication of an index from the outbound cluster to the inbound cluster. |
|
| Stop replication | Ends the replication and converts the inbound index to a standard index |
|
| Pause replication | Pauses replication of the outbound index. |
|
| Resume replication | Resumes replication of the outbound index. |
|
| Get replication status |
Gets the status of index replication. Possible statuses are
Use the syncing details to measure replication lag. |
|
| Get leader cluster stat | Gets information about replicated outbound indexes on a specified cluster. |
|
| Get follower cluster stats | Gets information about inbound indexes on a specified cluster. |
|
| Get auto-follow stats | Gets information about autofollow activity and any replication rules configured on the specified cluster. |
|
| Update settings | Updates settings on the inbound index. |
|
| Create replication rule | Automatically starts replication on indexes matching a specified pattern. If a new index on the outbound cluster matches the pattern, OpenSearch automatically creates a inbound index and begins replication. You can also use this API to update existing replication rules. |
|
| Delete replication rule | Deletes the specified replication rule. This operation prevents any new indexes from being replicated but doesn't stop existing replication already started based on the rule. Replicated indexes remain read-only until you stop replication. |
|