Network Requirements for Oracle Exadata Database Service on Cloud@Customer
Review the network requirements for provisioning Oracle Exadata Database Service on Cloud@Customer at your site.
- Network Requirements for Oracle Exadata Database Service on Cloud@Customer
To provide secure and reliable network connectivity for different application and management functions, Exadata Database Service on Cloud@Customer uses different networks. - Data Center Network Services for Oracle Exadata Database Service on Cloud@Customer
Before you deploy Oracle Exadata Database Service on Cloud@Customer, ensure that your data center network meets requirements. - IP Addresses and Subnets for Oracle Exadata Database Service on Cloud@Customer
You must allocate a range of IP addresses to the administration network, and another range of IP addresses to the RoCE private network. - Uplinks for Oracle Exadata Database Service on Cloud@Customer
Ensure that your Oracle Exadata Database Service on Cloud@Customer system meets control plane server and database server uplink requirements. - Network Cabling for Oracle Exadata Database Service on Cloud@Customer
You can choose to use the supplied network equipment, or you can build your own SFP network. - Establish a Secure Connection Between Your CPS and OCI Using OCI’s FastConnect Service
Consider the solution outlined below, which leverages OCI’s FastConnect service, if you want additional isolation for the connection between your CPS and OCI in addition to the default TLS tunnel approach.
Network Requirements for Oracle Exadata Database Service on Cloud@Customer
To provide secure and reliable network connectivity for different application and management functions, Exadata Database Service on Cloud@Customer uses different networks.
The following list outlines the minimum network requirements to install an Exadata Database Service on Cloud@Customer system:
- Exadata Cloud@Customer Service Network: These network will be set
up to Oracle specifications and should not be modified by customer
without Oracle agreement.
- Control Plane Network
This virtual private network (VPN) connects the two control plane servers that are located in the Exadata Database Service on Cloud@Customer rack to Oracle Cloud Infrastructure. The VPN facilitates secure customer-initiated operations using the Oracle Cloud Infrastructure Console and APIs. It also facilitates secure monitoring and administration of the Oracle-managed infrastructure components in Exadata Database Service on Cloud@Customer.
- Control Plane Connectivity Considerations
In order for the control plane to function, the control plane server must be able to connect to certain Oracle Cloud Infrastructure (OCI) addresses. You must enable TCP port 443 outbound access to the endpoints in a specific OCI region as follows:
Table 3-2 Ports to Open for Control Plane Connectivity
Description / Purpose Open Port Location Outgoing Tunnel Service for Cloud Automation delivery
443 outbound
Use this URL format, replacing oci_region with your region:https://wss.exacc.oci_region.oci.oraclecloud.comSecure Tunnel Service for remote Oracle operator access
443 outbound
Use these URL formats, replacing oci_region with your region:https://mgmthe1.exacc.oci_region.oci.oraclecloud.comhttps://mgmthe2.exacc.oci_region.oci.oraclecloud.comObject Storage Service to retrieve system updates, infrastructure monitoring, and log collection
443 outbound
Use this URL format, replacing oci_region with your region:https://objectstorage.oci_region.oraclecloud.comhttps://swiftobjectstorage.oci_region.oraclecloud.comhttps://*.objectstorage.oci_region.oci.customer-oci.comMonitoring Service to record and process Infrastructure Monitoring Metrics (IMM)
443 outbound
Use this URL format, replacing oci_region with your region:https://telemetry-ingestion.oci_region.oraclecloud.comIdentity Service for Authorization and Authentication
443 outbound
Use this URL format, replacing oci_region with your region:https://identity.oci_region.oraclecloud.comhttps://auth.oci_region.oraclecloud.comOutgoing Tunnel Service for Cloud Automation delivery
443 outbound
Use this URL format, replacing oci_region with your region:https://wsshe.adbd-exacc.oci_region.oci.oraclecloud.comLogging Service
443 outbound
Use this URL format replacing oci_region with your region:https://frontend.logging.ad1.oci_region.oracleiaas.comhttps://frontend.logging.ad2.oci_region.oracleiaas.comhttps://frontend.logging.ad3.oci_region.oracleiaas.comhttps://controlplane.logging.ad1.oci_region.oracleiaas.comhttps://controlplane.logging.ad2.oci_region.oracleiaas.comhttps://controlplane.logging.ad3.oci_region.oracleiaas.comResource Principal based authentication and Autonomous Database service delivery
443 outbound
Use this URL format, replacing oci_region with your region:https://database.oci_region.oraclecloud.comVM console
443 outbound
Use this URL format, replacing oci_region with your region:https://console1.exacc.oci_region.oci.oraclecloud.comhttps://console2.exacc.oci_region.oci.oraclecloud.comTemporary Secure Tunnel Service for remote Oracle operator access for ADB-D resources
443 outbound Use this URL format, replacing region with your region:https://mgmthe.adbd-exacc.region.oci.oraclecloud.comMonitoring Service to record and process Infrastructure Monitoring Metrics (IMM) resources
443 outbound Use this URL format, replacing region with your region:https://ingestion.logging.region.oci.oraclecloud.comMetering and Monitoring
443 outbound Use this URL format, replacingoci_regionwith your region:https://*.oci_region.functions.oci.oraclecloud.comhttps://*.functions.oci_region.oci.oraclecloud.comNote
ExaDB-C@C infrastructure access to the stated service endpoints in Table 3-2 Ports to Open for Control Plane Connectivity is mandatory for full service functionality. Failing to permit all the mandatory URIs may result in a service reduction or features not working as designed.Note that the Control Plane Server must be able to establish TCP Port 443 outbound access only. While outbound connections on Port 443 must be allowed, TCP Port 443 inbound access is not required, and it may be desirable from a security standpoint to block inbound connections. (Functionally, bi-directional traffic is still possible over the connection once the secure outbound connection is established.)
The Control Plane Server requires customer DNS and NTP services to be functional. Minimum bandwidth requirements for the Control Plane Server internet connection to OCI are 50/10 mbs download/upload.
Some customers have security policies requiring the use of proxies for all internet connections to IT infrastructure. Customer
If you are using IP address filtering based firewall rules, due to the dynamic nature of cloud interfaces, you must allow traffic with all the relevant IP CIDR ranges associated with your OCI region as identified by https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json.HTTPproxy, for example, passive/corporate proxy supported for the Control Plane Server connection to OCI. CustomerHTTPS, challenge proxy, and traffic inspection are not supported.
- Control Plane Connectivity Considerations
- Administration Network
This network connects Exadata Database Service on Cloud@Customer servers and switches to the two control plane servers that are located in the Exadata Database Service on Cloud@Customer rack. It facilitates customer-initiated operations using the Oracle Cloud Infrastructure Console and APIs. It also facilitates monitoring and administration of the Oracle-managed infrastructure components in Exadata Database Service on Cloud@Customer.
This network is fully contained within the Exadata Database Service on Cloud@Customer rack, and does not connect to your corporate network. However, the Exadata infrastructure is indirectly connected to your corporate network through the control plane servers. This connection is required to provide Domain Name System (DNS) and Network Time Protocol (NTP) services to the Exadata infrastructure. Therefore, the IP addresses that are allocated to the administration network must not exist elsewhere in your corporate network.
Each Oracle Database server and Exadata Storage Server has two network interfaces connected to the administration network. One provides management access to the server through one of the embedded Ethernet ports (
NET0). The other provides access to the Integrated Lights-Out Management (ILOM) subsystem through a dedicated ILOM Ethernet port. Exadata Database Service on Cloud@Customer is delivered with the ILOM andNET0ports connected to the Ethernet switch in the rack. Cabling or configuration changes to these interfaces are not permitted. - InfiniBand or RDMA Over Converged
Ethernet (ROCE) Network
This network connects the database servers, Exadata Storage Servers, and control plane servers using the InfiniBand or ROCE switches on the rack. Each server contains two InfiniBand network interfaces (
IB0andIB1) or ROCE interface (re0andre1) that are connected to separate InfiniBand or ROCE switches in the rack. Primarily, Oracle Database uses this network for Oracle RAC cluster interconnect traffic, and for accessing data on Exadata Storage Servers.This non-routable network is fully contained within the Exadata Cloud@Customer rack, and does not connect to your corporate network. However, because the control plane servers are connected to the InfiniBand or ROCE network and to your corporate network, the IP addresses that are allocated to the InfiniBand or ROCE network must not exist elsewhere in your corporate network.
- Control Plane Network
- Customer Network: Customer-owned and managed networks required
for the Exadata Cloud@Customer data plane to access related
systems.
- Client
Network
This network connects the Exadata Cloud@Customer database servers to your existing client network and is used for client access to the virtual machines. Applications access databases on Exadata Database Service on Cloud@Customer through this network using Single Client Access Name (SCAN) and Oracle Real Application Clusters (Oracle RAC) Virtual IP (VIP) interfaces.
The client access network uses a pair of network interfaces on each database server, which are connected to the customer network.
Note
When you enable Data Guard, replication of data happens over the client network by default. - Backup Network
This network is similar to the client access network, as it connects the Exadata Database Service on Cloud@Customer Oracle Database servers to your existing network. It can be used for access to the virtual machines for various purposes, including backups and bulk data transfers.
Like the client network, the backup network uses a pair of network interfaces on each database server, which are connected to the customer network. Physically connecting the backup network to a customer network is required.
If the customer's on-premises storage (NFS or ZDLRA) is to be used exclusively as a backup destination for databases, then no external connectivity to OCI is required for the backup network.
Exadata Cloud@Customer also offers an Oracle-managed cloud object storage backup destination. If Oracle's Object Storage Service is to be leveraged as a backup destination for database backups, then ensure that the backup network can reach the Object Storage Service through external connection. You must enable TCP port 443 outbound access for the backup network as follows:
Table 3-3 Ports to Open for Backup Network
Description / Purpose Open Port Location Object Storage Service for cloud-based database backups (Optional)
443 outbound
Use this URL format, replacing oci_region with your region:https://objectstorage.oci_region.oraclecloud.comhttps://swiftobjectstorage.oci_region.oraclecloud.com - Disaster Recovery Network (authorized customers only)Note
This additional network is available only to certain authorized customers under special circumstances. Your system may not be equipped with it.The Disaster Recovery network is configured similarly to the client access network but is dedicated solely to transmitting Data Guard traffic. If your system includes this network, Data Guard traffic will be routed through it instead of the client network, utilizing Single Client Access Name (SCAN) and Oracle Real Application Clusters (Oracle RAC) Virtual IP (VIP) interfaces.
Only the Data Guard prechecks require the client network to be open. Once the precheck stage is complete, Data Guard provisioning automatically registers the database with the Disaster Recovery network and uses it for both datafile restoration and redo transport. If opening the client network is not an option, you must manually register the primary database with the Disaster Recovery network listener before initiating Data Guard provisioning.
The Disaster Recovery network consists of a pair of network interfaces on each database server, which are directly connected to the customer network.
- Client
Network
Data Center Network Services for Oracle Exadata Database Service on Cloud@Customer
Before you deploy Oracle Exadata Database Service on Cloud@Customer, ensure that your data center network meets requirements.
Domain Name System (DNS) Requirements
As part of the deployment process, you must decide on the host names and IP addresses to be used for various Oracle Exadata Database Service on Cloud@Customer network interfaces. Oracle requires that you register the host names and IP addresses for the Oracle Exadata Database Service on Cloud@Customer client and backup network interfaces in your corporate DNS. At least one reliable DNS server is required, which must be accessible to the control plane servers and to all of the servers on the client network. Up to three DNS servers can be registered in Oracle Exadata Database Service on Cloud@Customer to ensure coverage in case a server is unavailable.
Network Time Protocol (NTP) Services Requirements
Oracle Exadata Database Service on Cloud@Customer uses NTP to ensure that all system components are synchronized to the same time. At least one reliable NTP server is required, which must be accessible to the control plane servers and to all of the servers on the client network. Up to three NTP servers can be registered in Oracle Exadata Database Service on Cloud@Customer to ensure coverage in case a server is unavailable.
IP Addresses and Subnets for Oracle Exadata Database Service on Cloud@Customer
You must allocate a range of IP addresses to the administration network, and another range of IP addresses to the RoCE private network.
Administration and Private Network Requirements for Oracle Exadata Database Service on Cloud@Customer
No overlap is permitted between the address ranges for the administration network and the RoCE private network, and all IP addresses should be unique within your corporate network. You must also allocate IP addresses from your corporate network to the control plane servers. These network configuration details are specified when you create the Exadata infrastructure.
When you create the Exadata infrastructure, the console pre-populates default values for the administration network CIDR block and the InifinBand network CIDR block. You can use the suggested CIDR blocks if there is no overlap with existing IP addresses in your corporate network.
Review IP address requirements for each of these networks. The table specifies the maximum and minimum CIDR block prefix length that are allowed for each network. The maximum CIDR block prefix length defines the smallest block of IP addresses that are required for the network. To allow for possible future expansion within Oracle Exadata Database Service on Cloud@Customer, work with your network team to reserve enough IP addresses to accommodate any future growth.
| Network Type | IP Address Requirements |
|---|---|
| Administration network |
Maximum CIDR block prefix length:
Minimum CIDR block prefix length:
|
| Private network |
Maximum CIDR block prefix length:
Minimum CIDR block prefix length:
|
| Control plane network | 2 IP addresses, 1 for each control plane server |
For more information about Administration and Private Network CIDR requirements, see Table 5-5 X9M CIDR Requirements and Table 5-6 X8M CIDR Requirements in Using the Console to Create Oracle Exadata Database Service on Cloud@Customer Infrastructure.
Host Name and IP Address Requirements for Oracle Exadata Database Service on Cloud@Customer
To connect to your corporate network, Oracle Exadata Database Service on Cloud@Customer requires several host names and IP addresses for network interfaces on the client network and the backup network. The precise number of IP addresses depends on the Exadata system shape. These network configuration details, including host names and IP addresses, are specified when you create a VM cluster network. All IP addresses must be statically assigned IP addresses, not dynamically assigned (DHCP) addresses. The client network and the backup network require separate subnets.
The following table outlines the IP address requirements for the client and backup networks. The table specifies the maximum and recommended CIDR block prefix length for each network. The maximum CIDR block prefix length defines the smallest block of IP addresses that are required for the network. To allow for possible future expansion within Oracle Exadata Database Service on Cloud@Customer, work with your network team to reserve enough IP addresses to accommodate any future growth.
| Network Type | IP Address Requirements for Base System, Quarter Rack, or Half Rack | IP Address Requirements for Full Rack |
|---|---|---|
| Client network |
Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
| Backup network | Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
| (For authorized customers only) Disaster Recovery network |
Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
Maximum CIDR block prefix length:
Recommended CIDR block prefix length:
|
Uplinks for Oracle Exadata Database Service on Cloud@Customer
Ensure that your Oracle Exadata Database Service on Cloud@Customer system meets control plane server and database server uplink requirements.
Control Plane Servers
Four uplinks (2 per server) are required to connect the Control Plane Servers to your corporate network to support the outbound secure network connections to the OCI services required by Oracle Exadata Database Service on Cloud@Customer.
Database Server Connections
Typically, four uplinks are required for each database server to connect to your corporate network. Using this configuration, two uplinks support the client network and the other two uplinks support the backup network.
On Quarter Rack, Half Rack, or Full Rack systems, you can choose to use 10 Gbps RJ45 copper, 10 Gbps SFP+ fiber, or 25Gbps SFP28 fiber network connections to your corporate network. However, you cannot mix copper and fiber networks on the same physical server. For example, you cannot use fiber for the client network and copper for the backup network.
On Base System configurations, the options are more limited because of the physical network interfaces that are available on each database server. On Base Systems, you can choose to use copper or fiber network connections only for the client network, while the backup network uses a fiber connection.
You can also use shared network interfaces on the Base System for the client network and the backup network, which reduces the uplink requirement to two uplinks for each database server. Using shared network interfaces also enables you to use copper network connections to support both the client and backup networks on Base System configurations. However, in general, Oracle recommends that you do not use shared network interfaces, because sharing networks compromises the bandwidth and availability of both networks. Shared network interfaces are not supported for Quarter, Half, and Full Rack configurations.
Network Cabling for Oracle Exadata Database Service on Cloud@Customer
You can choose to use the supplied network equipment, or you can build your own SFP network.
Supplied Network Equipment Option
Every Oracle Exadata Database Service on Cloud@Customer rack is shipped with all of the network equipment and cables that are required to interconnect all hardware in the Oracle Exadata Database Service on Cloud@Customer rack.
Small Form-Factory Pluggable Network Option
Oracle supplies small form-factor pluggable (SFP) network interfaces to enable connectivity to your corporate network. However, if you choose to configure an SFP network, then you are responsible to provide the required cabling to connect the Exadata Database Servers and Control Plane Servers to your corporate network.
Establish a Secure Connection Between Your CPS and OCI Using OCI’s FastConnect Service
Consider the solution outlined below, which leverages OCI’s FastConnect service, if you want additional isolation for the connection between your CPS and OCI in addition to the default TLS tunnel approach.
For more information, see Oracle Cloud Infrastructure FastConnect.
Oracle Exadata Database Service on Cloud@Customer service supports the public or private peering connectivity model of FastConnect.
Figure 3-1 Oracle Exadata Database Service on Cloud@Customer FastConnect connectivity to OCI through public peering
As shown in the figure, the Oracle Exadata Database Service on Cloud@Customer Control Plane network egresses to FastConnect provider and to Oracle edge. Customers who may already have existing FastConnect connectivity can use it to connect Oracle Exadata Database Service on Cloud@Customer to the OCI region using public peering.
Figure 3-2 Oracle Exadata Database Service on Cloud@Customer FastConnect connectivity to OCI through private peering
Configuring FastConnect for Oracle Exadata Database Service on Cloud@Customer
You can set up and configure Oracle Cloud Infrastructure FastConnect either before or after deploying Oracle Exadata Database Service on Cloud@Customer.
- As shown in the figures, it is important to set the Oracle Exadata Database Service on Cloud@Customer Control Plane Server network egress rules to forward traffic over FastConnect, and also have a route to an internet-facing customer DNS. This "Customer DNS" is used by Oracle Exadata Database Service on Cloud@Customer infrastructure to resolve OCI public endpoints.
- Corporate HTTP proxy between Oracle Exadata Database Service on Cloud@Customer Control Plane Servers and OCI region is not recommended with FastConnect as it is already a dedicated network. If a corporate proxy is highly desired, then the proxy needs to have additional routing to ensure that Oracle Exadata Database Service on Cloud@Customer network traffic is sent over FastConnect.
- If you are using private peering, then ensure that you configure transit routing at your VCN side. For more information, see Transit Routing to the Oracle Services Network.