Scenario: Sending Streaming Data to Logging Analytics

Default policies provide you the access you need. The workflow for creating a connector with a Streaming source and Logging Analytics target includes a default policy for reading streams and a default policy for writing to Logging Analytics. If you're new to policies, see Getting Started with Policies and Common Policies.

The goal of this scenario is to analyze stream data.

Setting up this scenario involves creating a connector to move streams to Logging Analytics for analysis.

You can complete these tasks in the Oracle Cloud Infrastructure Console, CLI, or API.

1. Open the navigation menu and click Analytics & AI. Under Messaging, click Connector Hub.

2. Choose the Compartment where you want to create the connector.

3. Click Create connector.

4. On the Create connector page, move streams to Logging Analytics:

   1. Type a Connector name. Avoid entering confidential information. Example: "Metrics to Bucket"
   2. Select the Resource compartment where you want to store the new connector.
   3. Under Configure connector, select your source and target services to move log data to a metric:
      • Source: Streaming
      • Target: Logging Analytics
   4. Under Configure source, select the stream:

      • Compartment: Select the compartment containing the stream.

      • Stream pool: Select the name of the stream pool containing the stream.

      • Stream: Select the name of the stream that you want to receive data from.

      • Read position: Select Latest to start reading at messages published after saving the connector.

   5. Skip the optional task (Configure task).
   6. Under Configure target, select the log group and source that you want to move the stream to:

      • Compartment: Select the compartment containing the log group.

      • Log group: Select the log group you want.

      • Log source identifier: Select the log source you want.

5. If you are prompted to create a policy (required for the connector to access source, task, and target services), click Create.

6. Click Create.

   The new connector immediately begins moving the stream to the selected log group and source.

1. Define the Streaming source for the connector: Save a file named source.json.

   Example source.json file

   {
     "kind": "streaming",
     "streamId": "<stream_OCID>",
     "cursor" : {
         "kind": "LATEST"
     }
   }

2. Define the Object Storage target for the connector: Save a file named target.json.

   Example target.json file

   {
    "kind": "loggingAnalytics",
    "logGroupId": "<log_group_OCID>",
    "logSourceIdentifier": "<log_source_name>"
   }

3. Create the connector using the oci sch service-connector create command:

   Example command for creating a connector

   oci sch service-connector create
    --compartment-id <compartment_OCID>
    --display-name "Stream to Logging Analytics"
    --source file://source.json
    --target file://target.json

   For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

   For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

Use the CreateServiceConnector operation to create the connector.

POST /20200909/serviceConnectors
Host: service-connector-hub.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "compartmentId": "<compartment_OCID>",
  "displayName": "Stream to Logging Analytics",
  "source": {
    "kind": "streaming",
    "streamId": "<stream_OCID>",
    "cursor" : {
        "kind": "LATEST"
    }
  },
  "target": {
   "kind": "loggingAnalytics",
   "logGroupId": "<log_group_OCID>",
   "logSourceIdentifier": "<log_source_name>"
  }
}

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.