Oracle Cloud Infrastructure Documentation

Planning and Understanding ODH Clusters

Before creating Big Data Service clusters, you must plan and understand clusters, instance types and shapes, and cluster profiles.

Complete the following:

Planning the Cluster Layout, Shape, and Storage

Before you start the process to create a cluster, you must plan the layout of the cluster, the node shapes, and storage.

Cluster Layout

Nodes and services are organized differently on clusters, based on whether the cluster is highly available (HA) and secure, or not.

About using HA clusters

Use HA clusters for production environments. They're required for resiliency and to minimize downtime.

In this release, a cluster must be both HA and secure, or neither.

Types of nodes

The types of nodes are as follows:

  • Master or utility nodes include the services required for the operation and management of the cluster. These nodes don't store or process data.
  • Worker nodes store and process data. The loss of a worker node doesn't affect the operation of the cluster, although it can affect performance.
  • Compute only worker nodes process data. The loss of a compute only worker node doesn't affect the operation of the cluster, although it can affect performance.
    Note

    Compute only worker nodes aren't supported for CDH clusters.
  • Edge nodes are extended nodes to the cluster that have only clients installed. You can install additional packages and run additional applications in this node instead of worker/compute/master nodes to avoid classpath conflicts and resources issues with cluster services.

High availability (HA) cluster layout

A high availability cluster has two master nodes, two utility nodes, three or more worker nodes, and zero or more compute only worker nodes.

Type of node Services on ODH Services on CDH
First master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS JournalNode
  • HDFS NameNode
  • HDFS ZKFailoverController
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Spark3 Client
  • Spark3 History Server
  • YARN Client
  • YARN ResourceManager
  • ZooKeeper Server
  • HDFS Failover Controller
  • HDFS JournalNode
  • HDFS NameNode
  • Hive Client
  • Key Trustee KMS Key Management Server Proxy
  • Key Trustee Server Active Database
  • Key Trustee Server Active Key Trustee Server
  • Spark Client
  • Spark History Server
  • YARN (MR2 Included) JobHistory Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
Second master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS JournalNode
  • HDFS NameNode
  • HDFS ZKFailoverController
  • Kerberos Client
  • MapReduce2 Client
  • MapReduce2 History Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN Registry DNS
  • YARN ResourceManager
  • YARN Timeline Service V1.5
  • ZooKeeper Server
  • HDFS Balancer
  • HDFS Failover Controller
  • HDFS HttpFS
  • HDFS JournalNode
  • HDFS NameNode
  • Hive Client
  • Hue Load Balancer
  • Hue Server
  • Hue Kerberos Ticket Renewer
  • Key Trustee KMS Key Management Server Proxy
  • Key Trustee Server Passive Database
  • Key Trustee Server Passive Key Trustee Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
First utility node
  • Ambari Metrics Monitor
  • Ambari Server
  • HDFS Client
  • HDFS JournalNode
  • Hive Metastore
  • HiveServer2
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS Client
  • HDFS JournalNode
  • Hive Client
  • Cloudera Management Service Alert Publisher
  • Cloudera Management Service Event Server
  • Cloudera Management Service Host Monitor
  • Cloudera Management Service Navigator Audit Server
  • Cloudera Management Service Navigator Metadata Server
  • Cloudera Management Service Reports Manager
  • Cloudera Management Service Monitor
  • Sentry Server
  • Spark Client
  • YARN (MR2 Included) Client
  • ZooKeeper Server
Second utility node
  • Ambari Metrics Collector
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Spark3 Client
  • YARN Client
  • HDFS Client
  • Hive Client
  • Hive Metastore Server
  • HiveServer2
  • Hive WebHCat Server
  • Hue Load Balancer
  • Hue Server
  • Hue Kerberos Ticket Renewer
  • Oozie Server
  • Sentry Server
  • Spark Client
  • YARN (MR2 Included) Client
Worker nodes (3 minimum)
  • Ambari Metrics Monitor
  • HDFS DataNode
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Spark3 Thrift Server
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
  • HDFS DataNode
  • Hive Client
  • Spark Client
  • YARN (MR2 Included) NodeManager
Compute only worker nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
 NA
Edge nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
 NA

Minimal (nonHA) Cluster Layout

A nonhigh availability cluster has one master node, one utility node, three or more worker nodes, and zero or more compute only worker nodes.

Type of node Services on ODH Services on CDH
Master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS NameNode
  • Hive Client
  • MapReduce2 Client
  • Spark3 Client
  • Spark3 History Server
  • YARN Client
  • YARN Registry DNS
  • YARN ResourceManager
  • ZooKeeper Server
  • HDFS Balancer
  • HDFS NameNode
  • Hive Client
  • Spark Client
  • Spark History Server
  • YARN (MR2 Included) JobHistory Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
Utility node
  • Ambari Metrics Collector
  • Ambari Metrics Monitor
  • Ambari Server
  • HDFS Client
  • HDFS Secondary NameNode
  • Hive Metastore
  • HiveServer2
  • MapReduce2 Client
  • MapReduce2 History Server
  • Oozie Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN Timeline Service V1.5
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS HttpFS
  • HDFS SecondaryNameNode
  • Hive Client
  • Hive Metastore Server
  • HiveServer2
  • Hive WebHCat Server
  • Hue Load Balancer
  • Hue Server
  • Cloudera Management Service Alert Publisher
  • Cloudera Management Service Event Server
  • Cloudera Management Service Host Monitor
  • Cloudera Management Service Navigator Audit Server
  • Cloudera Management Service Navigator Metadata Server
  • Cloudera Management Service Reports Manager
  • Cloudera Management Service Monitor
  • Oozie Server
  • Spark Client
  • YARN (MR2 Included) Client
  • ZooKeeper Server
Worker nodes
  • Ambari Metrics Monitor
  • HDFS DataNode
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Spark3 Thrift Server
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS DataNode
  • Hive Client
  • Spark Client
  • YARN (MR2 Included) NodeManager
  • ZooKeeper Server
Compute only worker nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
 NA
Edge nodes
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
 NA
Supported Node Shapes

The node shape describes the compute resources allocated to the node.

The shapes used for master/utility nodes and worker nodes can be different. But all master/utility nodes must be of the same shape and all worker nodes must be of the same shape.

The following table shows what shapes can be used for the different node types. See Compute Shapes for more detailed information.

For a list of the resources provided by each shape, see:

Node Type Available Shapes Required Number of Virtual Network Interface Cards (VNICs)
Master or utility

VM.Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E5.Flex

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseIO2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

 3 minimum

Used for the cluster subnet, the DP access subnet, and the customer's subnet

*You must specify a minimum of 3 OCPU and 32 GB memory.
Worker

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E5.Flex

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

 2 minimum

Used for the cluster subnet and the your subnet
Compute only worker

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E5.Flex

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

 2 minimum

Used for the cluster subnet and the your subnet

Compute only worker nodes aren't supported for CDH clusters.
Edge

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E5.Flex

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

2 minimum

Used for the cluster subnet and the customer's subnet

Note: Because the Edge node is specific to client application usecases, choose shape as required by the application.

Edge nodes aren't supported for CDH clusters.

Cloud SQL query server

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E5.Flex

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.HPC2.36

BM.Standard2.52

BM.Optimized3.36*

 --

* Be aware that VM.Standard2.1 and VM.Standard2.2 are small shapes and won't support running large workloads. For VM.Standard.E4.Flex, VM.Standard3.Flex, VM.Standard.E5.Flex, and VM.Optimized3.Flex you must specify minimum of 1 OCPU and 16GB memory.

Note

Note:The following shapes aren't supported for CDH clusters. They're supported for ODH clusters only.

VM.Standard.E4.Flex

VM.Standard.E5.Flex

VM.Standard3.Flex

VM.Optimized3.Flex

VM.DenseIO.E4.Flex

BM.Standard3.64

BM.Optimized3.36

BM.DenseIO.E4.128

Not all shapes are available by default. To see what shapes are available by default through the Cloud Console, see Finding Tenancy Limits. To submit a request to increase the service limits, see Requesting a Service Limit Increase.

Block Storage Node Shapes

Nodes based on standard VM shapes use network-attached block storage.

Note

Block storage isn't supported for nodes based on DenseIO and HPC shapes.

All nodes have a boot volume of 150 GB.

Option Limits/Guidelines
Minimum initial block storage 150 GB
Default initial block storage * 150 GB
Minimum additional block storage 150 GB
Default additional block storage * 1 TB
Incremental step for (initial and additional) block storage 50 GB
Maximum block storage for a single node

48 TB

The 48 TB total results from 12 volumes of 4 TB each.

If you add block storage multiple times, the maximum remains 48 TB, but it might be spread across more than 12 volumes.
Maximum block volume size

4 TB

If you specify the maximum 48 TB, 12 drives of 4 TB each are created.

If you specify a lower number, enough 4 TB devices for that amount are created, and more devices are created as you add more storage.

You can't add more block storage to master or utility nodes. Therefore, the following figures show initial sizes only.

Option Limits/Guidelines
Minimum initial block storage 150 GB
Default initial block storage 1 TB
Minimum additional block storage 150 GB
Default additional block storage 1 TB
Incremental step for (initial and additional) block storage 50 GB
Maximum block storage for a single node 32 TB
Maximum block volume size 32 TB
MySQL placement For utility nodes move /var/lib/mysql to /u01 and create a symbolic link. This prevents filling up the boot volume.
Option Guidelines
Default initial block storage 2 TB
Minimum initial block storage 150 GB

Query server storage is used for temporary table space to perform heavy JOIN and GROUP BY operations. 2 TB is recommended for typical processing. For small environments, for example development, this number can be adjusted down.

For best performance, consider these factors:

  • I/O throughput
  • Networking between the compute device and block storage device.

See Block Volume Performance in the Oracle Cloud Infrastructure documentation.

The following table describes how Big Data Service allocates block volume storage for nodes of different sizes.

What Amount
Initial volume allocation for master nodes and utility nodes 1 large volume
Volume allocation for additional block storage for master nodes and utility nodes 1 large volume
Initial volume allocation for worker nodes.

  • Storage: Less than 6 TB.

    Volume size: 500 GB The last volume might be smaller than 500 GB.

  • Storage: 6 TB to 48 TB.

    Volume size: Divide evenly into 12 volumes, each of which is at least 500 GB.

  • Storage: Greater than 48 TB.

    Volume size: Not allowed.
Volume allocation for additional block storage for worker nodes

The minimum number of volumes that can accommodate the storage size, with a maximum volume size of 4 TB per volume. (The last volume might be smaller than 4 TB.)

We recommend that you use edge nodes for staging.

Understanding Instance Types and Shapes

Big Data Service cluster nodes run in Oracle Cloud Infrastructure compute instances (servers).

When you create a cluster, you choose an instance type, which determines whether the instance runs directly on the bare metal instance of the hardware or in a virtualized environment. You also choose a shape, which configures the resources assigned to the instance.

About Instance Types

  • Bare metal: A bare metal compute instance uses a dedicated physical server for the node, for highest performance and strongest isolation.

  • Virtual Machine (VM): Through virtualization, a VM compute instance can host multiple, isolated nodes that run on a single physical bare metal machine. VM instances are less expensive than bare metal instances and are useful for building less demanding clusters that don't require the performance and resources (CPU, memory, network bandwidth, storage) of an entire physical machine for each node.

VM instances run on the same hardware as bare metal instances, with the same firmware, software stack, and networking infrastructure.

For more information about compute instances, see Overview of Compute Service.

About Shapes

The shape determines the number of CPUs, amount of memory, and other resources allocated to the compute instance hosting the cluster node. See Planning the Cluster Layout, Shape, and Storage in the Oracle Cloud Infrastructure documentation for the available shapes.

The shapes of the Big Data Service master nodes and the worker nodes don't have to match. But the shapes of all the master nodes must match each other and the shapes of all the worker nodes must match each other.

Understanding Cluster Profiles

Cluster profiles enable you to create optimal clusters for a specific workload or technology. After creating a cluster with a specific cluster profile, more Hadoop services can be added to the cluster.

Cluster Profile Types

Oracle Big Data Service enables you to create clusters for numerous cluster profile types.

Cluster profile Components (Secure and Highly Available) Components
HADOOP_EXTENDED1 Hive, Spark, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue, Oozie, Tez Hive, Spark, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue, Oozie, Tez
HADOOP HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue
HIVE Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue, Tez Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue, Tez
SPARK Spark, Hive2, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue Spark, Hive2, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue 2
HBASE HBase, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue HBase, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue
TRINO Trino, Hive3, HDFS, ZooKeeper, Ambari Metrics, Ranger, Hue Trino, Hive3, HDFS, ZooKeeper, Ambari Metrics, Hue
KAFKA Kafka Broker, HDFS, ZooKeeper, Ambari Metrics, Ranger, Hue Kafka Broker, HDFS, ZooKeeper, Ambari Metrics, Hue

1 HADOOP_EXTENDED consists of components that you created clusters before cluster profiles were available.

2Hive metastore component from Hive service is used for managing the metadata in Spark.

3Hive metastore component from Hive service is used for managing the Hive metadata entities in Trino.

Apache Hadoop Versions in Cluster Profiles

The following table lists the Hadoop component versions included in cluster profiles corresponding to ODH version.

ODH 1.x

Cluster profile Version
HADOOP_EXTENDED HDFS 3.1, Hive 3.1, Spark 3.0.2
HADOOP HDFS 3.1
HIVE Hive 3.1
SPARK Spark 3.0.2
HBASE HBase 2.2
TRINO Trino 360
KAFKA Kafka 2.1.0

ODH 2.x

Cluster profile Version
HADOOP_EXTENDED HDFS 3.3, Hive 3.1, Spark 3.2
HADOOP HDFS 3.3
HIVE Hive 3.1
SPARK Spark 3.2
HBASE HBase 2.2
TRINO Trino 389