Network Requirements for Oracle Exadata Database Service on Cloud@Customer

Review the network requirements for provisioning Oracle Exadata Database Service on Cloud@Customer at your site.

Network Requirements for Oracle Exadata Database Service on Cloud@Customer

To provide secure and reliable network connectivity for different application and management functions, Exadata Database Service on Cloud@Customer uses different networks.

The following list outlines the minimum network requirements to install an Exadata Database Service on Cloud@Customer system:

  • Exadata Cloud@Customer Service Network: These network will be set up to Oracle specifications and should not be modified by customer without Oracle agreement.
    • Control Plane Network

      This virtual private network (VPN) connects the two control plane servers that are located in the Exadata Database Service on Cloud@Customer rack to Oracle Cloud Infrastructure. The VPN facilitates secure customer-initiated operations using the Oracle Cloud Infrastructure Console and APIs. It also facilitates secure monitoring and administration of the Oracle-managed infrastructure components in Exadata Database Service on Cloud@Customer.

      • Control Plane Connectivity Considerations

        In order for the control plane to function, the control plane server must be able to connect to certain Oracle Cloud Infrastructure (OCI) addresses. You must enable TCP port 443 outbound access to 5 endpoints in a specific OCI region as follows:

        Table 3-2 Ports to Open for Control Plane Connectivity

        Description / Purpose Open Port Location

        Outgoing Tunnel Service for Cloud Automation delivery

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://wss.exacc.oci_region.oci.oraclecloud.com

        Secure Tunnel Service for remote Oracle operator access

        443 outbound

        Use these URL formats, replacing oci_region with your region:
        https://mgmthe1.exacc.oci_region.oci.oraclecloud.com
        https://mgmthe2.exacc.oci_region.oci.oraclecloud.com

        Object Storage Service to retrieve system updates

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://objectstorage.oci_region.oraclecloud.com
        https://swiftobjectstorage.oci_region.oraclecloud.com

        Monitoring Service to record and process Infrastructure Monitoring Metrics (IMM)

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://telemetry-ingestion.oci_region.oraclecloud.com

        Identity Service for name resolution of Oracle operators

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://identity.oci_region.oraclecloud.com

        Outgoing Tunnel Service for Cloud Automation delivery

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://wsshe.adbd-exacc.oci_region.oci.oraclecloud.com

        Logging Service

        443 outbound

        Use this URL format replacing oci_region with your region:
        • https://frontend.logging.ad1.oci_region.oracleiaas.com/
        • https://frontend.logging.ad2.oci_region.oracleiaas.com/
        • https://frontend.logging.ad3.oci_region.oracleiaas.com/
        • https://controlplane.logging.ad1.oci_region.oracleiaas.com/
        • https://controlplane.logging.ad2.oci_region.oracleiaas.com/
        • https://controlplane.logging.ad3.oci_region.oracleiaas.com/

        Resource Principal based authentication and Autonomous Database service delivery

        443 outbound

        Use this URL format, replacing oci_region with your region:
        https://database.oci_region.oraclecloud.com

        Note that the Control Plane Server must be able to establish TCP Port 443 outbound access only. While outbound connections on Port 443 must be allowed, TCP Port 443 inbound access is not required, and it may be desirable from a security standpoint to block inbound connections. (Functionally, bi-directional traffic is still possible over the connection once the secure outbound connection is established.)

        The Control Plane Server requires customer DNS and NTP services to be functional. Minimum bandwidth requirements for the Control Plane Server internet connection to OCI are 50/10 mbs download/upload.

        Some customers have security policies requiring the use of proxies for all internet connections to IT infrastructure. Customer HTTP proxy, for example, passive/corporate proxy supported for the Control Plane Server connection to OCI. Customer HTTPS, challenge proxy, and traffic inspection are not supported.

        If you are using IP address filtering based firewall rules, due to the dynamic nature of cloud interfaces, you must allow traffic with all the relevant IP CIDR ranges associated with your OCI region as identified by https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json.
    • Administration Network

      This network connects Exadata Database Service on Cloud@Customer servers and switches to the two control plane servers that are located in the Exadata Database Service on Cloud@Customer rack. It facilitates customer-initiated operations using the Oracle Cloud Infrastructure Console and APIs. It also facilitates monitoring and administration of the Oracle-managed infrastructure components in Exadata Database Service on Cloud@Customer.

      This network is fully contained within the Exadata Database Service on Cloud@Customer rack, and does not connect to your corporate network. However, the Exadata infrastructure is indirectly connected to your corporate network through the control plane servers. This connection is required to provide Domain Name System (DNS) and Network Time Protocol (NTP) services to the Exadata infrastructure. Therefore, the IP addresses that are allocated to the administration network must not exist elsewhere in your corporate network.

      Each Oracle Database server and Exadata Storage Server has two network interfaces connected to the administration network. One provides management access to the server through one of the embedded Ethernet ports (NET0). The other provides access to the Integrated Lights-Out Management (ILOM) subsystem through a dedicated ILOM Ethernet port. Exadata Database Service on Cloud@Customer is delivered with the ILOM and NET0 ports connected to the Ethernet switch in the rack. Cabling or configuration changes to these interfaces are not permitted.

    • InfiniBand or RDMA Over Converged Ethernet (ROCE) Network

      This network connects the database servers, Exadata Storage Servers, and control plane servers using the InfiniBand or ROCE switches on the rack. Each server contains two InfiniBand network interfaces (IB0 and IB1) or ROCE interface (re0 and re1) that are connected to separate InfiniBand or ROCE switches in the rack. Primarily, Oracle Database uses this network for Oracle RAC cluster interconnect traffic, and for accessing data on Exadata Storage Servers.

      This non-routable network is fully contained within the Exadata Cloud@Customer rack, and does not connect to your corporate network. However, because the control plane servers are connected to the InfiniBand or ROCE network and to your corporate network, the IP addresses that are allocated to the InfiniBand or ROCE network must not exist elsewhere in your corporate network.

  • Customer Network: Customer-owned and managed networks required for the Exadata Cloud@Customer data plane to access related systems.
    • Client Network

      This network connects the Exadata Cloud@Customer database servers to your existing client network and is used for client access to the virtual machines. Applications access databases on Exadata Database Service on Cloud@Customer through this network using Single Client Access Name (SCAN) and Oracle Real Application Clusters (Oracle RAC) Virtual IP (VIP) interfaces.

      The client access network uses a pair of network interfaces on each database server, which are connected to the customer network.

      Note

      When you enable Data Guard, replication of data happens only over the client network.
    • Backup Network

      This network is similar to the client access network, as it connects the Exadata Database Service on Cloud@Customer Oracle Database servers to your existing network. It can be used for access to the virtual machines for various purposes, including backups and bulk data transfers.

      Like the client network, the backup network uses a pair of network interfaces on each database server, which are connected to the customer network. Physically connecting the backup network to a customer network is required.

      If the customer's on-premises storage (NFS or ZDLRA) is to be used exclusively as a backup destination for databases, then no external connectivity to OCI is required for the backup network.

      Exadata Cloud@Customer also offers an Oracle-managed cloud object storage backup destination. If Oracle's Object Storage Service is to be leveraged as a backup destination for database backups, then ensure that the backup network can reach the Object Storage Service through external connection. You must enable TCP port 443 outbound access for the backup network as follows:

      Table 3-3 Ports to Open for Backup Network

      Description / Purpose Open Port Location

      Object Storage Service for cloud-based database backups (Optional)

      443 outbound

      Use this URL format, replacing oci_region with your region:
      https://objectstorage.oci_region.oraclecloud.com
      https://swiftobjectstorage.oci_region.oraclecloud.com

Data Center Network Services for Exadata Database Service on Cloud@Customer

Before you deploy Exadata Database Service on Cloud@Customer, ensure that your data center network meets requirements.

Domain Name System (DNS) Requirements

As part of the deployment process, you must decide on the host names and IP addresses to be used for various Exadata Database Service on Cloud@Customer network interfaces. Oracle requires that you register the host names and IP addresses for the Exadata Database Service on Cloud@Customer client and backup network interfaces in your corporate DNS. At least one reliable DNS server is required, which must be accessible to the control plane servers and to all of the servers on the client network. Up to three DNS servers can be registered in Exadata Database Service on Cloud@Customer to ensure coverage in case a server is unavailable.

Network Time Protocol (NTP) Services Requirements

Exadata Database Service on Cloud@Customer uses NTP to ensure that all system components are synchronized to the same time. At least one reliable NTP server is required, which must be accessible to the control plane servers and to all of the servers on the client network. Up to three NTP servers can be registered in Exadata Database Service on Cloud@Customer to ensure coverage in case a server is unavailable.

IP Addresses and Subnets for Exadata Database Service on Cloud@Customer

You must allocate a range of IP addresses to the administration network, and another range of IP addresses to the InfiniBand network.

InfiniBand Requirements for Oracle Cloud@Customer

No overlap is permitted between the address ranges for the administration network and the InfiniBand network, and all IP addresses should be unique within your corporate network. You must also allocate IP addresses from your corporate network to the control plane servers. These network configuration details are specified when you create the Exadata infrastructure.

When you create the Exadata infrastructure, the console pre-populates default values for the administration network CIDR block and the InifinBand network CIDR block. You can use the suggested CIDR blocks if there is no overlap with existing IP addresses in your corporate network.

Review IP address requirements for each of these networks. The table specifies the maximum and minimum CIDR block prefix length that are allowed for each network. The maximum CIDR block prefix length defines the smallest block of IP addresses that are required for the network. To allow for possible future expansion within Exadata Database Service on Cloud@Customer, you can choose a smaller CIDR block prefix length, within the allowable range, which reserves more IP addresses for the network.

Network Type IP Address Requirements
Administration network
Maximum CIDR block prefix length:
/23
Minimum CIDR block prefix length:
/16
InifinBand network
Maximum CIDR block prefix length:
/22
Minimum CIDR block prefix length:
/19
Control plane network 2 IP addresses, 1 for each control plane server

Host Name and IP Address Requirements for Oracle Cloud@Customer

To connect to your corporate network, Exadata Database Service on Cloud@Customer requires several host names and IP addresses for network interfaces on the client network and the backup network. The precise number of IP addresses depends on the Exadata system shape. These network configuration details, including host names and IP addresses, are specified when you create a VM cluster network. All IP addresses must be statically assigned IP addresses, not dynamically assigned (DHCP) addresses. The client network and the backup network require separate subnets.

The following table outlines the IP address requirements for the client and backup networks. The table specifies the maximum and recommended CIDR block prefix length for each network. The maximum CIDR block prefix length defines the smallest block of IP addresses that are required for the network. To allow for possible future expansion within Exadata Database Service on Cloud@Customer, a smaller CIDR block prefix length is recommended, which reserves more IP addresses for the network.

Network Type IP Address Requirements for Base System, Quarter Rack, or Half Rack IP Address Requirements for Full Rack
Client network
Maximum CIDR block prefix length:
/28
Recommended CIDR block prefix length:
/27
Maximum CIDR block prefix length:
/27
Recommended CIDR block prefix length:
/26
Backup network Maximum CIDR block prefix length:
/29
Recommended CIDR block prefix length:
/28
Maximum CIDR block prefix length:
/28
Recommended CIDR block prefix length:
/27

Uplinks for Exadata Database Service on Cloud@Customer

Ensure that your Exadata Database Service on Cloud@Customer system meets control plane server and database server uplink requirements.

Control Plane Servers

Four uplinks (2 per server) are required to connect the Control Plane Servers to your corporate network to support the outbound secure network connections to the OCI services required by the Exadata Cloud@Customer service.

Database Server Connections

Typically, four uplinks are required for each database server to connect to your corporate network. Using this configuration, two uplinks support the client network and the other two uplinks support the backup network.

On Quarter Rack, Half Rack, or Full Rack systems, you can choose to use 10 Gbps RJ45 copper, 10 Gbps SFP+ fiber, or 25Gbps SFP28 fiber network connections to your corporate network. However, you cannot mix copper and fiber networks on the same physical server. For example, you cannot use fiber for the client network and copper for the backup network.

On Base System configurations, the options are more limited because of the physical network interfaces that are available on each database server. On Base Systems, you can choose to use copper or fiber network connections only for the client network, while the backup network uses a fiber connection.

You can also use shared network interfaces on the Base System for the client network and the backup network, which reduces the uplink requirement to two uplinks for each database server. Using shared network interfaces also enables you to use copper network connections to support both the client and backup networks on Base System configurations. However, in general, Oracle recommends that you do not use shared network interfaces, because sharing networks compromises the bandwidth and availability of both networks. Shared network interfaces are not supported for Quarter, Half, and Full Rack configurations.

Network Cabling for Exadata Database Service on Cloud@Customer

You can choose to use the supplied network equipment, or you can build your own SFP network.

Supplied Network Equipment Option

Every Exadata Database Service on Cloud@Customer rack is shipped with all of the network equipment and cables that are required to interconnect all hardware in the Exadata Database Service on Cloud@Customer rack.

Small Form-Factory Pluggable Network Option

Oracle supplies small form-factor pluggable (SFP) network interfaces to enable connectivity to your corporate network. However, if you choose to configure an SFP network, then you are responsible to provide the required cabling to connect the Exadata Database Servers and Control Plane Servers to your corporate network.

Oracle Exadata Database Service on Cloud@Customer Gen2 Oracle Cloud Infrastructure FastConnect

Oracle Cloud Infrastructure FastConnect enables a private high bandwidth connection between your on-premises data center and Oracle Cloud Infrastructure Region.

For more information, see Oracle Cloud Infrastructure FastConnect.

Oracle Exadata Database Service on Cloud@Customer service supports the public peering connectivity model of FastConnect.

Figure 3-1 Oracle Exadata Database Service on Cloud@Customer FastConnect connectivity to OCI through public peering

This figure illustrates Oracle Exadata Database Service on Cloud@Customer FastConnect connectivity to OCI through public peering.

As shown in the figure, the Oracle Exadata Database Service on Cloud@Customer Control Plane network egresses to FastConnect provider and to Oracle edge. Customers who may already have existing FastConnect connectivity can use it to connect Oracle Exadata Database Service on Cloud@Customer to the OCI region using public peering.

Configuring FastConnect for Oracle Exadata Database Service on Cloud@Customer

You can set up and configure Oracle Cloud Infrastructure FastConnect either before or after deploying Oracle Exadata Database Service on Cloud@Customer.

Oracle Exadata Database Service on Cloud@Customer rack egress network configuration for FastConnect
  • As shown in the figure, it is important to set the Oracle Exadata Database Service on Cloud@Customer Control Plane Server network egress rules to forward traffic over FastConnect, and also have a route to an internet-facing customer DNS. This "Customer DNS" is used by Oracle Exadata Database Service on Cloud@Customer infrastructure to resolve OCI public endpoints.
  • Corporate HTTP proxy between Oracle Exadata Database Service on Cloud@Customer Control Plane Servers and OCI region is not recommended with FastConnect as it is already a dedicated network. If a corporate proxy is highly desired, then the proxy needs to have additional routing to ensure that Oracle Exadata Database Service on Cloud@Customer network traffic is sent over FastConnect.