Scenario: Sending Streaming Data to Logging Analytics

Learn how to send streaming data from the OCI Streaming service to Logging Analytics using Connector Hub. Streaming ingests and consumes high-volume data streams in real time.

Required IAM Policy

Default policies provide you the access you need. The workflow for creating a connector with a Streaming source and Logging Analytics target includes a default policy for reading streams and a default policy for writing to Logging Analytics. If you're new to policies, see Getting Started with Policies and Common Policies.

Goal

The goal of this scenario is to analyze stream data.

Setting Up This Scenario

Setting up this scenario involves creating a connector to move streams to Logging Analytics for analysis.

You can complete these tasks in the Oracle Cloud Infrastructure Console, CLI, or API.

Using the Console

Open the navigation menu and click Analytics & AI. Under Messaging, click Connector Hub.
Choose the Compartment where you want to create the connector.
Select Create connector.
On the Create connector page, move streams to Logging Analytics:
1. Type a Connector name. Avoid entering confidential information. Example: "Metrics to Bucket"
2. Select the Resource compartment where you want to store the new connector.
3. Under Configure connector, select your source and target services to move log data to a metric:
  - Source: Streaming
  - Target: Logging Analytics
4. Under Configure source, select the stream:
  - Compartment: Select the compartment containing the stream.
  - Stream pool: Select the name of the stream pool containing the stream.
  - Stream: Select the name of the stream that you want to receive data from.
  - Read position: Select Latest to start reading at messages published after saving the connector.
5. Skip the optional task (Configure task).
6. Under Configure target, select the log group and source that you want to move the stream to:
  - Compartment: Select the compartment containing the log group.
  - Log group: Select the log group you want.
  - Log source identifier: Select the log source you want.
If you are prompted to create a policy (required for the connector to access source, task, and target services), select Create.
Select Create.

The new connector immediately begins moving the stream to the selected log group and source.

Using the CLI

Define the Streaming source for the connector: Save a file named source.json.
Example source.json file
```
{
  "kind": "streaming",
  "streamId": "<stream_OCID>",
  "cursor" : {
      "kind": "LATEST"
  }
}
```

Define the Object Storage target for the connector: Save a file named target.json.

Example target.json file

{
 "kind": "loggingAnalytics",
 "logGroupId": "<log_group_OCID>",
 "logSourceIdentifier": "<log_source_name>"
}

Create the connector using the oci sch service-connector create command:
Example command for creating a connector
oci sch service-connector create --compartment-id <compartment_OCID> --display-name "Stream to Logging Analytics" --source file://source.json --target file://target.json
For a complete list of parameters and values for CLI commands, see the CLI Command Reference.
For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

For help with troubleshooting, see Troubleshooting Connectors and Troubleshooting Notifications.

Using the API

Use the CreateServiceConnector operation to create the connector.

Example CreateServiceConnector request

POST /20200909/serviceConnectors
Host: service-connector-hub.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "compartmentId": "<compartment_OCID>",
  "displayName": "Stream to Logging Analytics",
  "source": {
    "kind": "streaming",
    "streamId": "<stream_OCID>",
    "cursor" : {
        "kind": "LATEST"
    }
  },
  "target": {
   "kind": "loggingAnalytics",
   "logGroupId": "<log_group_OCID>",
   "logSourceIdentifier": "<log_source_name>"
  }
}