Using bda-oss-admin to Manage Storage and Other Configuration Settings
Use the bda-oss-admin
command line utility to add and manage storage
providers, user credentials for storage access, and other resources for use with a cluster.
Configuration details are stored in the core.site.xml
file on the
cluster.
bda-oss-admin
CLI can be used only in clusters that use Cloudera
Distribution including Hadoop.To enter commands at the command line, use SSH to connect as the opc
user to any node on the cluster and enter the bda-oss-admin
command. See Connecting to a Cluster Node Using SSH.
Understanding bda-oss-admin Environment Variables
Some bda-oss-admin
options can be set as environment variables so that you aren't required to specify the values every time you run the commands.
The following tables show the environment variables that correspond to (and can be
substituted for) the listed bda-oss-admin
options.
Options and Environment Variables for All bda-oss-admin Commands
The following values must be set for all bda-oss-admin
commands,
either on the command line as options to the command or as shell environment
variables.
Command Option | Environment Variable | Description |
---|---|---|
--cm-admin |
CM_ADMIN |
Cloudera Manager administrator user name. |
--cm-passwd |
CM_PASSWORD |
Cloudera Manager administrator password. The command will prompt for the password if it is not provided. |
--cm-url |
CM_URL |
Cloudera Manager URL (e.g.
https://servername.bigdata.examplecloud.com:7183 ) |
Options and Environment Variables for bda-oss-admin Storage Credentials Commands
To set these as environment variables, you can create and run a shell script.
In the following example, a Linux bash shell script named
bdcsvars.sh
sets the Cloudera Manager credentials required by
all bda-oss-admin
commands:
#!/bin/bash
export CM_ADMIN="my_cm_admin_username"
export CM_PASSWORD="my_cm_admin_password"
export CM_URL="https://my_CM_hostname_:7183"
In the following example, a Linux bash shell script named
storvars.sh
sets the storage credentials :
#!/bin/bash
export ST_USER="MyServiceName-MyIdentityDomain:MyUserName"
export ST_KEY="Wel_123"
export ST_AUTH="http://storage.a123456.examplecloud.com/auth/v1.0"
export ST_PROVIDER="MyProviderName"
If you are working with multiple storage providers, it may be convenient to create shell scripts for all of them. Then you only have to run a script to set storage credentials for whichever provider you are using.
To run the above scripts from the directory in which they reside:
# source ./bdacsvars.sh
# source ./storvars.sh
Configurations that you set with the bda-oss-admin
commands,
parameters, and environment variables are stored in the Hadoop
/etc/hadoop/conf/core-site.xml
configuration file.
You can look in the core-site.xml
file to see the current configuration, but you shouldn't edit it directly. Use bda-oss-admin
instead.
bda-oss-admin Command
Use the Big Data Service command line utility bda-oss-admin
to
manage users and resources of your cluster.
To issue bda-oss-admin
commands, you must connect to a node as the opc
user and then use the sudo
command to switch to the root
user. See Connecting to a Cluster Node Using SSH.
Syntax
bda-oss-admin [options] subcommand [arguments]...
Options
Option | Description |
---|---|
--version |
Show the bda-oss-admin
version |
--cm-admin
user_name |
Cloudera Manager administrator user name |
--cm-passwd
password |
Cloudera Manager administrator (cluster administrator) password. The command will prompt for the password if it is not provided. |
--b64–cm-passwd
password |
The Cloudera Manager password is Base64 encoded. It will be decoded before uploading. |
--cm-url url |
Cloudera Manager URL; for example,
https://servername.bigdata.oraclecloud.com:7183 |
|
The password is Base64 encoded. It will be decoded before uploading. |
|
Show help for this command. |
When you specify any of the above options on the command line, the options must be
placed immediately after the bda-oss-admin
command and before any
of its subcommands. For example, this command is legal:
# bda-oss-admin --region region_name add_oci_cred
However, the following command is not legal, because the --region
option is placed after the list_oci_cred
subcommand:
# bda-oss-admin list_oci_cred --region region_name
bda-oss-admin Subcommands
Command | Task |
---|---|
bda-oss-admin add_bdcs_cp_extensions_mr | Add Big Data Service classpath extensions to the MapReduce configuration |
bda-oss-admin add_hadoop_prop | Add properties to the Hadoop configuration |
bda-oss-admin add_krb_delegate | Add credentials for an Oracle Cloud Infrastructure Object Storage Classic user with administrative privileges |
bda-oss-admin add_oci_cred | Add Oracle Cloud Infrastructure Object Storage credentials to the Hadoop configuration |
bda-oss-admin add_proxy_user | Add a proxy user to delegate tokens to other users |
bda-oss-admin add_s3_cred | Add Amazon Web Services S3 credentials to the Hadoop configuration |
bda-oss-admin delete_oci_cred | Remove he Oracle Cloud Infrastructure Object Storage Classic credentials from the Hadoop configuration |
bda-oss-admin delete_s3_cred | Remove Amazon Web Services S3 credentials from the Hadoop configuration |
bda-oss-admin list_oci_cred | List the Oracle Cloud Infrastructure Object Storage Classic Object Storage credentials from the Hadoop configuration |
bda-oss-admin list_s3_cred | List the Amazon Web Services S3 credentials from the Hadoop configuration |
bda-oss-admin print_yarn_hadoop_env | List Oracle Cloud Infrastructure Object Storage Classic credentials from the Hadoop configuration |
bda-oss-admin print_yarn_mapred_cp | Display the YARN MapReduce default classpath |
bda-oss-admin remove_bdcs_cp_extensions_mr | Remove Oracle Big Data Service
classpath extensions from the MapReduce
configuration. |
bda-oss-admin restart_cluster | Restart the cluster (only stale services) |
bda-oss-admin set_bdcs_proxy_user | Add the Big Data Service user as a proxy user to a Hadoop configuration (internal command) |
bda-oss-admin update_oci_cred | Update the Oracle Cloud Infrastructure Object Storage credentials in the Hadoop configuration. |
bda-oss-admin update_s3_cred | Update the Amazon Web Services S3 credentials in the Hadoop configuration |
Use bda-oss-admin add_bdcs_cp_extensions_mr
to add the Oracle Big Data Service
classpath
extensions to the MapReduce configuration (in the mapred-site.xml
and hadoop-env.sh
files).
Syntax
bda-oss-admin add_bdcs_cp_extensions_mr [options]
Options
Option | Description |
---|---|
|
Show help for this command |
Example
# bda-oss-admin add_bdcs_cp_extensions_mr
Changes will not affect the cluster until services get restarted. See the restart_cluster command
Use bda-oss-admin add_hadoop_prop
to add a property to the Hadoop
core-site.xml
configuration file via the Cloudera Manager
API.
Syntax
In the following syntax, line breaks are added for clarity. Don't include them when entering the command.
bda-oss-admin add_hadoop_prop
[<options>]
<property_name>
<property_value>
where
<property_name> is the name of the property, for example
swift.service.Foo.secrets.file
.
<property_value> is the value for property, for example
jceks://hdfs/user/bigdatamgr/secrets.jceks
.
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Use bda-oss-admin add_krb_delegate
to add a Hadoop delegation
token.
Syntax
bda-oss-admin add_krb_delegate <options>
Options
Option | Explanation |
---|---|
|
Proxy user which can delegate tokens to other users. |
|
Proxy groups value. Default is
all . |
|
Proxy host value. Default is
all . |
|
Proxy users value. Default is
all . |
|
Show help for this command. |
Use bda-oss-admin add_oci_cred
to register a storage provider with a
cluster or to modify the details about the provider.
Use this command to register an Oracle Cloud Infrastructure Object Storage provider for the first time. You can also re-run the command with different or fewer parameters to modify the registration. Only one Oracle Cloud Infrastructure Object Storage provider can be registered at a time.
Syntax
bda-oss-admin add_oci_cred [<options>]
Options
Option | Explanation |
---|---|
|
The region where the object storage is located, used to define
the URL endpoint. For example, if your region is
|
|
The URL of the host endpoint. For example, |
|
The Oracle Cloud ID (OCID) that identifies the tenancy. |
|
The OCID that identifies the user. |
|
The fingerprint of the SSH private key for the storage. |
|
A local file that contains the SSH private key for the storage.
(This file is automatically copied to all cluster nodes using
|
|
The path (on the cluster) to the SSH private key. See also
The default is
|
|
Passphrase used for the key, if it's encrypted. |
|
The OCID of the compartment containing the storage, used to list the buckets in the storage. |
|
The proxy uri; e.g. |
|
The proxy user name. |
|
The proxy user password. |
|
Do not verify accounts against actual storage service before adding. |
|
The dcli command path. The
default path
is:
|
|
Show help for this command. |
Use bda-oss-admin add_proxy_user
to add a proxy user to delegate
tokens to other users.
Syntax
bda-oss-admin add_proxy_user <options>
Options
Option | Explanation |
---|---|
|
Identifies a proxy user which can delegate tokens to other users. |
|
Proxy groups value. Default is
all . |
|
Proxy hosts value. Default is
all . |
|
Proxy users value. Default is
all . |
|
Show help for this command. |
Use bda-oss-admin add_s3_cred
to register an Amazon Web Services S3
storage provider with the Big Data Service cluster.
Syntax
In the following syntax, line breaks are added for clarity. Don't include them when entering the command.
bda-oss-admin add_s3_cred
[<options>]
<aws_access_key>
<aws_secret_key>
Parameters
Parameter | Explanation |
---|---|
|
A username, provided as an alphanumeric text string that uniquely identifies the user who owns the account. No two accounts can have the same AWS Access Key. |
|
A string that's used as a password . It's called secret because it is assumed to be known by the owner only. An AWS secret key combined with an AWS access key forms a secure information set that confirms the user's identity. |
Options
Option | Explanation |
---|---|
|
The hostname of the proxy host. |
|
The port used for the proxy server. |
|
Enable or disable a Secure Sockets Layer (SSL) connection to the S3 provider. |
|
Show help for this command. |
Use bda-oss-admin delete_oci_cred
to remove Oracle Cloud
Infrastructure Object Storage credentials from the Hadoop configuration.
Syntax
bda-oss-admin delete_oci_cred [<options>]
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Use bda-oss-admin delete_s3_cred
to remove Amazon Web Services S3
credentials from the Hadoop configuration.
Syntax
bda-oss-admin delete_s3_cred [<options>]
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Use bda-oss-admin list_oci_cred
to list the Oracle Cloud
Infrastructure Object Storage credentials stored in the Hadoop configuration.
Syntax
bda-oss-admin list_oci_cred
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Use bda-oss-admin list_s3_cred
to list the Amazon Web Service S3
credentials stored in the Hadoop configuration.
Syntax
bda-oss-admin list_s3_cred [<options>]
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Displays YARN MapReduce snippet for hadoop-env.sh
.
Syntax
bda-oss-admin print_yarn_hadoop_env [<options>]
Options
Option | Explanation |
---|---|
|
Show help for this command. |
Use bda-oss-admin print_yarn_mapred_cp
to display the Apache
YARN/MapReduce default classpath.
Syntax
bda-oss-admin print_yarn_mapred_cp [options]
Options
Option | Description |
---|---|
|
Show help for this command. |
Example
# bda-oss-admin print_yarn_mapred_cp
YARN Mapreduce Conf mapreduce_application_classpath:
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH,/opt/oracle/orabalancer-2.4.0-h2/jlib/orabalancer-2.4.0.jar,/opt/oracle/orabalancer-2.4.0-h2/jlib/commons-math-2.2.jar,/opt/oracle/bda/bdcs/bdcs-rest-api-app/current/lib-hadoop/*
Use bda-oss-admin remove_bdcs_cp_extensions_mr
to remove the Big
Data Service classpath
extensions from the MapReduce configuration file,
mapred-site.xml
.
Syntax
bda-oss-admin remove_bdcs_cp_extensions_mr [options]
Options
Option | Description |
---|---|
|
Show help for this command |
Example
# bda-oss-admin remove_bdcs_cp_extensions_mr
Changes will not affect the cluster until services get restarted. See the restart_cluster command
Use bda-oss-admin restart_cluster
to restart the
cluster.
Syntax
bda-oss-admin restart_cluster [options]
Options
Option |
Description |
---|---|
|
Show help for this command. |
Example
# bda-oss-admin restart_cluster
Restarting the cluster...
*********************
Cluster restarted successfully
Use bda-oss-admin set_bdcs_proxy_user
to add proxy
users.
Usage
This is an internally used command to bootstrap Big Data Service. Restart Hadoop services to enable this change.
Syntax
bda-oss-admin set_bdcs_proxy_user [<options>]
Options
Option | Explanation |
---|---|
|
Host name from Big Data Service REST service |
|
Operating system group allowed to run Big Data
Service commands (for example, huser, * ) |
|
User running the Big Data Service service |
|
Show help for this command. |
Use bda-oss-admin update_oci_cred
to change the details about an
Oracle Infrastructure Object Storage provider that's already registered with a cluster.
Usage
The options for bda-oss-admin update_oci_cred
are identical to those
for bda-oss-admin add_oci_cred
. Run this command by using only the
options whose values you want to change.
Use bda-oss-admin update_s3_cred
to change the details about an
Amazon Web Services S3 provider that's already registered with a cluster.
Usage
The options for bda-oss-admin update_s3_cred
are identical to those
for bda-oss-admin add_s3_cred
. Run this command by using only the
options whose values you want to change.