Oracle Cloud Infrastructure Documentation

Using bda-oss-admin to Manage Storage and Other Configuration Settings

Use the bda-oss-admin command line utility to add and manage storage providers, user credentials for storage access, and other resources for use with a cluster. Configuration details are stored in the core.site.xml file on the cluster.

Note

bda-oss-admin CLI can be used only in clusters that use Cloudera Distribution including Hadoop.

To enter commands at the command line, use SSH to connect as the opc user to any node on the cluster and enter the bda-oss-admin command. See Connecting to a Cluster Node Using SSH.

Understanding bda-oss-admin Environment Variables

Some bda-oss-admin options can be set as environment variables so that you aren't required to specify the values every time you run the commands.

The following tables show the environment variables that correspond to (and can be substituted for) the listed bda-oss-admin options.

Options and Environment Variables for All bda-oss-admin Commands

The following values must be set for all bda-oss-admin commands, either on the command line as options to the command or as shell environment variables.

Command Option Environment Variable Description
--cm-admin CM_ADMIN Cloudera Manager administrator user name.
--cm-passwd CM_PASSWORD Cloudera Manager administrator password. The command will prompt for the password if it is not provided.
--cm-url CM_URL Cloudera Manager URL (e.g. https://servername.bigdata.examplecloud.com:7183)

Options and Environment Variables for bda-oss-admin Storage Credentials Commands

Setting Environment Variables

To set these as environment variables, you can create and run a shell script.

In the following example, a Linux bash shell script named bdcsvars.sh sets the Cloudera Manager credentials required by all bda-oss-admin commands:

#!/bin/bash
export CM_ADMIN="my_cm_admin_username"
export CM_PASSWORD="my_cm_admin_password"
export CM_URL="https://my_CM_hostname_:7183"

In the following example, a Linux bash shell script named storvars.sh sets the storage credentials :

#!/bin/bash
export ST_USER="MyServiceName-MyIdentityDomain:MyUserName"
export ST_KEY="Wel_123"
export ST_AUTH="http://storage.a123456.examplecloud.com/auth/v1.0"
export ST_PROVIDER="MyProviderName"

If you are working with multiple storage providers, it may be convenient to create shell scripts for all of them. Then you only have to run a script to set storage credentials for whichever provider you are using.

To run the above scripts from the directory in which they reside:

# source ./bdacsvars.sh
# source ./storvars.sh
Reviewing the Configuration

Configurations that you set with the bda-oss-admin commands, parameters, and environment variables are stored in the Hadoop /etc/hadoop/conf/core-site.xml configuration file.

You can look in the core-site.xml file to see the current configuration, but you shouldn't edit it directly. Use bda-oss-admin instead.

bda-oss-admin Command

Use the Big Data Service command line utility bda-oss-admin to manage users and resources of your cluster.

To issue bda-oss-admin commands, you must connect to a node as the opc user and then use the sudo command to switch to the root user. See Connecting to a Cluster Node Using SSH.

Syntax

bda-oss-admin [options] subcommand [arguments]...

Options

Option Description
--version Show the bda-oss-admin version
--cm-admin user_name Cloudera Manager administrator user name
--cm-passwd password Cloudera Manager administrator (cluster administrator) password. The command will prompt for the password if it is not provided.
--b64–cm-passwd password The Cloudera Manager password is Base64 encoded. It will be decoded before uploading.
--cm-url url Cloudera Manager URL; for example, https://servername.bigdata.oraclecloud.com:7183

-b

--b64-encoded-pwds

 The password is Base64 encoded. It will be decoded before uploading.

-h

--help

 Show help for this command.

When you specify any of the above options on the command line, the options must be placed immediately after the bda-oss-admin command and before any of its subcommands. For example, this command is legal:

# bda-oss-admin --region region_name add_oci_cred

However, the following command is not legal, because the --region option is placed after the list_oci_cred subcommand:

# bda-oss-admin list_oci_cred --region region_name

bda-oss-admin Subcommands

Command Task
bda-oss-admin add_bdcs_cp_extensions_mr Add Big Data Service classpath extensions to the MapReduce configuration
bda-oss-admin add_hadoop_prop Add properties to the Hadoop configuration
bda-oss-admin add_krb_delegate Add credentials for an Oracle Cloud Infrastructure Object Storage Classic user with administrative privileges
bda-oss-admin add_oci_cred Add Oracle Cloud Infrastructure Object Storage credentials to the Hadoop configuration
bda-oss-admin add_proxy_user Add a proxy user to delegate tokens to other users
bda-oss-admin add_s3_cred Add Amazon Web Services S3 credentials to the Hadoop configuration
bda-oss-admin delete_oci_cred Remove he Oracle Cloud Infrastructure Object Storage Classic credentials from the Hadoop configuration
bda-oss-admin delete_s3_cred Remove Amazon Web Services S3 credentials from the Hadoop configuration
bda-oss-admin list_oci_cred List the Oracle Cloud Infrastructure Object Storage Classic Object Storage credentials from the Hadoop configuration
bda-oss-admin list_s3_cred List the Amazon Web Services S3 credentials from the Hadoop configuration
bda-oss-admin print_yarn_hadoop_env List Oracle Cloud Infrastructure Object Storage Classic credentials from the Hadoop configuration
bda-oss-admin print_yarn_mapred_cp Display the YARN MapReduce default classpath
bda-oss-admin remove_bdcs_cp_extensions_mr Remove Oracle Big Data Service classpath extensions from the MapReduce configuration.
bda-oss-admin restart_cluster Restart the cluster (only stale services)
bda-oss-admin set_bdcs_proxy_user Add the Big Data Service user as a proxy user to a Hadoop configuration (internal command)
bda-oss-admin update_oci_cred Update the Oracle Cloud Infrastructure Object Storage credentials in the Hadoop configuration.
bda-oss-admin update_s3_cred Update the Amazon Web Services S3 credentials in the Hadoop configuration
bda-oss-admin add_bdcs_cp_extensions_mr

Use bda-oss-admin add_bdcs_cp_extensions_mr to add the Oracle Big Data Service classpath extensions to the MapReduce configuration (in the mapred-site.xml and hadoop-env.sh files).

Syntax

bda-oss-admin add_bdcs_cp_extensions_mr [options]

Options

Option Description

-h

--help

 Show help for this command

Example

# bda-oss-admin add_bdcs_cp_extensions_mr
Changes will not affect the cluster until services get restarted. See the restart_cluster command
bda-oss-admin add_hadoop_prop

Use bda-oss-admin add_hadoop_prop to add a property to the Hadoop core-site.xml configuration file via the Cloudera Manager API.

Syntax

In the following syntax, line breaks are added for clarity. Don't include them when entering the command.

bda-oss-admin add_hadoop_prop 
      [<options>] 
      <property_name> 
      <property_value>

where

<property_name> is the name of the property, for example swift.service.Foo.secrets.file.

<property_value> is the value for property, for example jceks://hdfs/user/bigdatamgr/secrets.jceks.

Options

Option Explanation

-h

--help

 Show help for this command.
bda-oss-admin add_krb_delegate

Use bda-oss-admin add_krb_delegate to add a Hadoop delegation token.

Syntax

bda-oss-admin add_krb_delegate <options>

Options

Option Explanation

--proxy-user <proxy_user>

Proxy user which can delegate tokens to other users.

--proxy-groups-value <proxy_groups_value>

 Proxy groups value. Default is all.

--proxy-hosts-value <proxy_hosts_value>

 Proxy host value. Default is all.

--proxy-users-value <proxy_users_value>

 Proxy users value. Default is all.

-h

--help

 Show help for this command.
bda-oss-admin add_oci_cred

Use bda-oss-admin add_oci_cred to register a storage provider with a cluster or to modify the details about the provider.

Use this command to register an Oracle Cloud Infrastructure Object Storage provider for the first time. You can also re-run the command with different or fewer parameters to modify the registration. Only one Oracle Cloud Infrastructure Object Storage provider can be registered at a time.

Syntax

bda-oss-admin add_oci_cred [<options>]

Options

Option Explanation

-r [<region>]

--region [<region>]

The region where the object storage is located, used to define the URL endpoint. For example, if your region is us-ashburn-1, the hostname https://objectstorage.us-ashburn-1.oraclecloud.com is used. If a hostname is defined, the region won't be used.

-H [<uri>]

--hostname [<uri>]

The URL of the host endpoint. For example, https://objectstorage.us-ashburn-1.oraclecloud.com.

-t [<tenancy_ocid>]

--tenancy [<tenancy_ocid>]

The Oracle Cloud ID (OCID) that identifies the tenancy.

-u [<user_ocid>]

--user [<user_ocid>]

The OCID that identifies the user.

-f [<fingerprint>]

--fingerprint [<fingerprint>]

The fingerprint of the SSH private key for the storage.

-l [<pem_local_path>]

--local-key-file [<pem_local_path>]

A local file that contains the SSH private key for the storage. (This file is automatically copied to all cluster nodes using dcli if run on a cluster, to the file defined by the --key-file option).

-k [<pem_cluster_path>]

--key-file [<pem_cluster_path>]

The path (on the cluster) to the SSH private key. See also --local-key-file).

The default is /opt/oracle/bigdatamanager/jetty-distribution/bigdatamanager/etc/storageCreds/default_oci_api_key.pem.

-p [<passphrase>]

--passphrase [<passphrase>]

 Passphrase used for the key, if it's encrypted.

-c [<compartment>]

--compartment [<compartment>]

The OCID of the compartment containing the storage, used to list the buckets in the storage.

--proxy-uri [<proxy_uri>]

The proxy uri; e.g. http://proxy.domain:80.

--proxy-user [<proxy_user>]

The proxy user name.

--proxy-password [<proxy_password>]

The proxy user password.

-N

--no-verify

Do not verify accounts against actual storage service before adding.

--dcli-path [<dcli-path>]

 The dcli command path. The default path is:

/opt/oracle/bda/bin/dcli

-h

--help

 Show help for this command.
bda-oss-admin add_proxy_user

Use bda-oss-admin add_proxy_user to add a proxy user to delegate tokens to other users.

Syntax

bda-oss-admin add_proxy_user <options>

Options

Option Explanation

--proxy-user <proxy_user>

 Identifies a proxy user which can delegate tokens to other users.

--proxy-groups-value <proxy_user>

 Proxy groups value. Default is all.

--proxy-hosts-value <proxy_user>

 Proxy hosts value. Default is all.

--proxy-users-value <proxy_user>

 Proxy users value. Default is all.

-h

--help

 Show help for this command.
bda-oss-admin add_s3_cred

Use bda-oss-admin add_s3_cred to register an Amazon Web Services S3 storage provider with the Big Data Service cluster.

Syntax

In the following syntax, line breaks are added for clarity. Don't include them when entering the command.

bda-oss-admin add_s3_cred  
      [<options>] 
      <aws_access_key> 
      <aws_secret_key>

Parameters

Parameter Explanation

<aws_access_key>

A username, provided as an alphanumeric text string that uniquely identifies the user who owns the account. No two accounts can have the same AWS Access Key.

<aws_secret_key>

 A string that's used as a password . It's called secret because it is assumed to be known by the owner only. An AWS secret key combined with an AWS access key forms a secure information set that confirms the user's identity.

Options

Option Explanation

--proxy-host <proxy_host>

 The hostname of the proxy host.

--proxy-port <proxy_port>

 The port used for the proxy server.

--ssl-enabled | --no-ssl-enabled

 Enable or disable a Secure Sockets Layer (SSL) connection to the S3 provider.

-h

--help

 Show help for this command.
bda-oss-admin delete_oci_cred

Use bda-oss-admin delete_oci_cred to remove Oracle Cloud Infrastructure Object Storage credentials from the Hadoop configuration.

Syntax

bda-oss-admin delete_oci_cred  [<options>]

Options

Option Explanation

-h

--help

 Show help for this command.
bda-oss-admin delete_s3_cred

Use bda-oss-admin delete_s3_cred to remove Amazon Web Services S3 credentials from the Hadoop configuration.

Syntax

bda-oss-admin delete_s3_cred [<options>]

Options

Option Explanation

-h

--help

 Show help for this command.
bda-oss-admin list_oci_cred

Use bda-oss-admin list_oci_cred to list the Oracle Cloud Infrastructure Object Storage credentials stored in the Hadoop configuration.

Syntax

bda-oss-admin list_oci_cred

Options

Option Explanation

-h

--help

 Show help for this command.
bda-oss-admin list_s3_cred

Use bda-oss-admin list_s3_cred to list the Amazon Web Service S3 credentials stored in the Hadoop configuration.

Syntax

bda-oss-admin list_s3_cred [<options>]

Options

Option Explanation

-h

--help

 Show help for this command.
bda-oss-admin remove_bdcs_cp_extensions_mr

Use bda-oss-admin remove_bdcs_cp_extensions_mr to remove the Big Data Service classpath extensions from the MapReduce configuration file, mapred-site.xml.

Syntax

bda-oss-admin remove_bdcs_cp_extensions_mr [options]

Options

Option Description

-h

--help

 Show help for this command

Example

# bda-oss-admin remove_bdcs_cp_extensions_mr
Changes will not affect the cluster until services get restarted. See the restart_cluster command
bda-oss-admin restart_cluster

Use bda-oss-admin restart_cluster to restart the cluster.

Syntax

 bda-oss-admin restart_cluster [options]

Options

Option

Description

-h

--help

Show help for this command.

Example

# bda-oss-admin restart_cluster
Restarting the cluster...
*********************
Cluster restarted successfully
bda-oss-admin set_bdcs_proxy_user

Use bda-oss-admin set_bdcs_proxy_user to add proxy users.

Usage

This is an internally used command to bootstrap Big Data Service. Restart Hadoop services to enable this change.

Syntax

bda-oss-admin set_bdcs_proxy_user [<options>]

Options

Option Explanation

--bdcs-host <host>

 Host name from Big Data Service REST service

--bdcs-allowed-group <allowed_group>

 Operating system group allowed to run Big Data Service commands (for example, huser, *)

--bdcs-user <user>

 User running the Big Data Service service

-h

--help

 Show help for this command.
bda-oss-admin update_oci_cred

Use bda-oss-admin update_oci_cred to change the details about an Oracle Infrastructure Object Storage provider that's already registered with a cluster.

Usage

The options for bda-oss-admin update_oci_cred are identical to those for bda-oss-admin add_oci_cred. Run this command by using only the options whose values you want to change.

bda-oss-admin update_s3_cred

Use bda-oss-admin update_s3_cred to change the details about an Amazon Web Services S3 provider that's already registered with a cluster.

Usage

The options for bda-oss-admin update_s3_cred are identical to those for bda-oss-admin add_s3_cred. Run this command by using only the options whose values you want to change.