Enabling Oracle Cloud Infrastructure Logging Spark Diagnostic Logs
Follow these steps to enable Oracle Cloud Infrastructure Logging Spark diagnostic logs.
Follow these steps in the Console to use Oracle Cloud Infrastructure Logging for Spark diagnostic logs in Data Flow.
Note
Spark Diagnostic Logs (Oracle Cloud Infrastructure Logging Service Logs) can be enabled only at the Data Flow Application level and can't be overridden.- After you have created an Application that uses Oracle Cloud Infrastructure Logging, open the details page for that Application.
- Under Resources, click Logs.
- Click Diagnostic logs.
- Toggle Enable log. The Enable Log panel is displayed.
- Select a Compartment.
- Select a Log group.
- Enter the Log name.
- Select the amount of time for Log retention.
-
Click Enable log.
Wait for the Status of the Diagnostic logs to be
Active
. After it's active, any runs using this Application have the diagnostic logs feature turned on for the Spark Driver and Executor logs. The Driver and Executor logs are technically 'Service logs' as they come from an Oracle Cloud Infrastructure native service such as Data Flow.
Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.
-
Create a Data Flow Application.
- Request
-
oci data-flow application create \ --compartment-id $COMPARTMENT_ID \ --display-name "Sample CSV Processing App with OCI Logging Service Logs" \ --driver-shape "VM.Standard2.1" \ --executor-shape "VM.Standard2.1" \ --file-uri "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar" \ --language "JAVA" \ --num-executors 1 \ --spark-version "3.0.2" \ --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"
- Response
-
{ "data": { "archive-uri": "", "arguments": null, "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": null, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "description": null, "display-name": "Sample CSV Processing App with OCI Logging Service Logs", "driver-shape": "VM.Standard2.1", "execute": null, "executor-shape": "VM.Standard2.1", "file-uri": "oci://dataflow_sample_apps@<tenancy-name>/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "<data-flow-application-ocid>", "language": "JAVA", "lifecycle-state": "ACTIVE", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": null, "num-executors": 1, "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": null, "private-endpoint-id": "", "spark-version": "3.0.2", "time-created": "2022-04-08T19:50:57.795000+00:00", "time-updated": "2022-04-08T19:50:57.795000+00:00", "warehouse-bucket-uri": null } }
-
Create a log group.
- Request
-
oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid>" }
-
Retrieve the log group ID.
- Request
-
export LOG_GROUP_WR_ID=<log-workrequest-ocid> oci logging work-request get --work-request-id $LOG_GROUP_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid>", "operation-type": "CREATE_LOG_GROUP", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "loggroup", "entity-uri": "/logGroups/<log-group-ocid>", "identifier": "<log-group-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-04-08T19:53:57.065000+00:00", "time-finished": "2022-04-08T19:53:57.065000+00:00", "time-started": "2022-04-08T19:53:57.065000+00:00" } }
-
Enable diagnostic (service) logs on the Data Flow Application.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json
- config.json
-
{ "compartment-id": "<compartment-ocid>", "source": { "resource": "<data-flow-application-ocid>", "service": "dataflow", "source-type": "OCISERVICE", "category": "diagnostic" } }
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid>" }
-
Retrieve the log ID for the diagnostic (service) logs.
- Request
-
export LOG_WR_ID=<log-workrequest-ocid> oci logging work-request get --work-request-id $LOG_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid>", "operation-type": "CREATE_LOG", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "log", "entity-uri": "/logs/<log-ocid>", "identifier": "<log-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-04-08T20:12:29.660000+00:00", "time-finished": "2022-04-08T20:12:33.356000+00:00", "time-started": "2022-04-08T20:12:29.660000+00:00" } }
-
Create a Data Flow Application.
Follow these steps in the API to use Oracle Cloud Infrastructure Logging Spark diagnostic logs in Data Flow.
-
Create a Data Flow Application.
- Request
-
oci data-flow application create \ --compartment-id $COMPARTMENT_ID \ --display-name "Sample CSV Processing App with OCI Logging Service Logs" \ --driver-shape "VM.Standard2.1" \ --executor-shape "VM.Standard2.1" \ --file-uri "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar" \ --language "JAVA" \ --num-executors 1 \ --spark-version "3.0.2" \ --class-name "com.oracle.oci.dataflow.samples.DataFlowJavaSample"
- Response
-
{ "data": { "archive-uri": "", "arguments": null, "class-name": "com.oracle.oci.dataflow.samples.DataFlowJavaSample", "compartment-id": "<compartment-ocid>", "configuration": null, "defined-tags": { "Dataflow_Resource_Category": { "category": "DEV_QA" } }, "description": null, "display-name": "Sample CSV Processing App with OCI Logging Service Logs", "driver-shape": "VM.Standard2.1", "execute": null, "executor-shape": "VM.Standard2.1", "file-uri": "oci://dataflow_sample_apps@bigdatadatasciencelarge/dataflow-java-sample-1.0-SNAPSHOT.jar", "freeform-tags": {}, "id": "data-flow-application-ocid>", "language": "JAVA", "lifecycle-state": "ACTIVE", "logs-bucket-uri": "oci://dataflow-logs@<tenancy-name>/", "metastore-id": null, "num-executors": 1, "owner-principal-id": "<user-ocid>", "owner-user-name": "<user-name>", "parameters": null, "private-endpoint-id": "", "spark-version": "3.0.2", "time-created": "2022-04-08T19:50:57.795000+00:00", "time-updated": "2022-04-08T19:50:57.795000+00:00", "warehouse-bucket-uri": null } }
-
Create a log group.
- Request
-
oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dataflow-logs"
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid" }
-
Retrieve the log group ID.
- Request
-
export LOG_GROUP_WR_ID=<log-workrequest-ocid oci logging work-request get --work-request-id $LOG_GROUP_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid", "operation-type": "CREATE_LOG_GROUP", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "loggroup", "entity-uri": "/logGroups/<log-group-ocid>", "identifier": "<log-group-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-04-08T19:53:57.065000+00:00", "time-finished": "2022-04-08T19:53:57.065000+00:00", "time-started": "2022-04-08T19:53:57.065000+00:00" } }
-
Enable diagnostic (service) logs on the Data Flow Application.
- Request
-
export LOG_GROUP_ID=<log-group-ocid> oci logging log create --display-name "sample_csv_processing_diagnostic" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json
- config.json
-
{ "compartment-id": "<compartment-ocid>", "source": { "resource": "data-flow-application-ocid>", "service": "dataflow", "source-type": "OCISERVICE", "category": "diagnostic" } }
- Response
-
{ "opc-work-request-id": "<log-workrequest-ocid" }
-
Retrieve the log ID for the diagnostic (service) logs.
- Request
-
export LOG_WR_ID=<log-workrequest-ocid oci logging work-request get --work-request-id $LOG_WR_ID
- Response
-
{ "data": { "compartment-id": "<compartment-ocid>", "id": "<log-workrequest-ocid", "operation-type": "CREATE_LOG", "percent-complete": 100.0, "resources": [ { "action-type": "CREATED", "entity-type": "log", "entity-uri": "/logs/<log-ocid>", "identifier": "<log-ocid>" } ], "status": "SUCCEEDED", "time-accepted": "2022-04-08T20:12:29.660000+00:00", "time-finished": "2022-04-08T20:12:33.356000+00:00", "time-started": "2022-04-08T20:12:29.660000+00:00" } }
-
Create a Data Flow Application.