Creating a Connector with a Monitoring Source

[
  {
    "namespace":"oci_computeagent",
    "compartmentId":"ocid1.tenancy.oc1..exampleuniqueID",
    "name":"DiskBytesRead",
    "dimensions":{
      "resourceId":"ocid1.instance.region1.phx.exampleuniqueID"
    },
    "metadata":{
      "unit":"bytes"
    }
  },
  "datapoints":[
    {
      "timestamp":"2022-03-10T22:19:00Z",
      "value":10.4
    },
    {
      "timestamp":"2022-03-10T22:20:00Z",
      "value":11.3
    }
  ]
]

• Console
• CLI
• API

• 1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. On the Service connectors page, select a compartment.
  3. Click Create service connector.
  4. On the Create service connector page, enter a user-friendly name for the new connector and an optional description. Avoid entering confidential information.
  5. Select the compartment where you want to store the new connector.
  6. Under Configure service connector, for Source, select Monitoring.
  7. For Target, select the service that you want to transfer the metric data points to:
     • Functions: Send metric data points to a function.
     • Object Storage: Send metric data points to a bucket.
     • Streaming: Send metric data points to a stream.
  8. (Optional) To enable service logs for the new connector, click the Resource logging switch and provide the following values:
     • Log Category: The value Connector Tracking is automatically selected.
     • Compartment: Select the compartment that you want for storing the service logs for the connector.
     • Log Group: Select the log group that you want for storing the service logs. To create a new log group, click Create New Group and then enter a name.
     • Log Name: Optionally enter a name for the log.
     • Show Advanced Options:
       • Enable auto archiving to object storage (legacy): Creates a copy of the logs in an Archive Storage bucket.
       • Log Retention: Optionally specify how long to keep the service logs (default: 30 days).
  9. Under Configure source connection, select the source metric:
     • Metric compartment: Select the compartment that contains the metrics that you want.
     • Namespaces: Select one or more metric namespaces that include the metrics that you want. All metrics in the selected namespaces are retrieved.

       The namespaces must begin with "oci_". Example: oci_computeagent.

     • To add metric namespaces from another compartment, click + Another compartment.

       The maximum number of metric compartments per Monitoring source is 5. The maximum number of namespaces per Monitoring source (across all metric compartments) is 50. Following are example sets of compartments in a single source that remain within this maximum:

       • 5 metric compartments with 10 namespaces each
       • 3 metric compartments with varying numbers of namespaces (20, 20, 10)
       • 1 metric compartment with 50 namespaces
  10. (Optional) Under Configure function task, configure a function task to process metric data points using the Functions service):
      • Select task: Select Function.
      • Compartment: Select the compartment that contains the function that you want.
      • Function application: Select the name of the function application that includes the function you want.
      • Function: Select the name of the function that you want to use to process the data received from the source.

        For use by the connector as a task, the function must be configured to return one of the following responses:

        • List of JSON entries (must set the response header Content-Type=application/json)
        • Single JSON entry (must set the response header Content-Type=application/json)
        • Single binary object (must set the response header Content-Type=application/octet-stream)
      • Show additional options: Click this link and specify limits for each batch of data sent to the function. If you choose to use manual settings, provide values for batch size limit (KBs) and batch time limit (seconds).

      Considerations for function tasks:

      • Connector Hub doesn't parse the output of the function task. The output of the function task is written as-is to the target. For example, when using a Notifications target with a function task, all messages are sent as raw JSON blobs.
      • Functions are invoked synchronously with 6 MB of data per invocation. If data exceeds 6 MB, then the connector invokes the function again to move the data that's over the limit. Such invocations are handled sequentially.
      • Functions can execute for up to five minutes.
      • Function tasks are limited to scalar functions.
  11. If you selected Functions as the target, under Configure target, configure the function to send the metric data points to. Then, skip to step 13.
      • Compartment: Select the compartment that contains the function that you want.
      • Function application: Select the name of the function application that contains the function that you want.
      • Function: Select the name of the function that you want to send the data to.

      Considerations for Functions targets:

      • The connector flushes source data as a JSON list in batches. Maximum batch, or payload, size is 6 MB.
      • Functions are invoked synchronously with 6 MB of data per invocation. If data exceeds 6 MB, then the connector invokes the function again to move the data that's over the limit. Such invocations are handled sequentially.
      • Functions can execute for up to five minutes.
      • Don't return data from Functions targets to connectors. Connector Hub doesn't read data returned from Functions targets.
  12. If you selected Object Storage as the target, under Configure target, configure the bucket to send the metric data points to. Then, skip to step 13.
      • Compartment: Select the compartment that contains the bucket that you want.
      • Bucket: Select the name of the bucket that you want to send the data to.
      • Object Name Prefix: Optionally enter a prefix value.
      • Show additional options: Click this link and optionally enter values for batch size (in MBs) and batch time (in milliseconds).

      Considerations for Object Storage targets:

      • Batch rollover details:

        • Batch rollover size: 100 MB
        • Batch rollover time: 7 minutes

      • Files saved to Object Storage are compressed using gzip.

      • Format of data moved from a Monitoring source: Objects. The connector partitions source data from Monitoring by metric namespace and writes the data for each group (namespace) to an object. Each object name includes the following elements.

        <object_name_prefix>/<service_connector_ocid>/<metric_compartment_ocid>/<metric_namespace>/<data_start_timestamp>_<data_end_timestamp>.<sequence_number>.<file_type>.gz

        Within an object, each set of data points is appended to a new line.

  13. If you selected Streaming as the target, under Configure target, configure the stream to send the metric data points to.
      • Compartment: Select the compartment that contains the stream that you want.
      • Stream: Select the name of the stream that you want to send the data to.

      Considerations for Streaming targets:

      • Private endpoint configuration isn't supported. For stream pool configuration details, see Creating Stream Pools.
      • Format of data moved from a Monitoring source: Each object is written as a separate message.
  14. To accept default policies, click the Create link provided for each default policy.

      Default policies are offered for any authorization required for this connector to access source, task, and target services.

      You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a connector. The only exception is when the exact policy already exists in IAM, in which case the default policy isn't offered. For more information about this authorization requirement, see Authentication and Authorization.

      • If you don't have permissions to accept default policies, contact your administrator.
      • Automatically created policies remain when connectors are deleted. As a best practice, delete associated policies when deleting the connector.

      To review a newly created policy, click the associated view link.

  15. (Optional) Assign tags to the connector. Click Show advanced options and then provide values for the tagging fields.
      • Tag namespace: To add a defined tag, select an existing namespace. If adding a free-from tag, leave blank.
      • Tag key: To add a defined tag, select an existing tag key. To add a free-form tag, type the key name that you want.
      • Tag value: Type the tag value that you want.
      • Add tag: Click to add another tag.
  16. Click Create.

• Use the oci sch service-connector create command and required parameters to create a connector with a Monitoring source:

  oci sch service-connector create [...] --source <monitoring_json_input>

  For a complete list of flags and variable options for CLI commands, see the Command Line Reference.

• Run the CreateServiceConnector operation to create a connector.

  To create a connector with a Monitoring source, populate source in the request (CreateServiceConnectorDetails) with monitoring details. For an example, see MonitoringSourceDetails.