Oracle Cloud Infrastructure Documentation

Ingest Custom Logs from OCI Streaming Service Using Service Connector

If you want to analyze and derive insights from your high-volume custom streaming data in OCI Streaming service by using Oracle Logging Analytics, then complete the following steps to ingest the streaming logs.

Steps to collect custom logs from OCI Streaming using service connector:

  1. Provide permission to access and collect logs from OCI Streaming service. See Allow Collection of Logs from OCI Streaming Service.

  2. Identify or create the compartment and Logging Analytics log group where the logs must be collected.

  3. Create a parser to parse your custom logs. Use the example log content from your streaming data to identify the fields. See Create a Parser.

    Note

    Logging Analytics supports parsers of the formats Regex, XML, and JSON. Ensure the following:

    • The log data in a stream must be of a single format, Regex, XML, or JSON and not have multiple formats.

    • The parser identified for a stream must be of the same format as the log data in that stream.

  4. Create a log source that best suits your custom logs and include the parser the you created in the previous step. See Create a Source.

    Alternatively, you can select from the available Oracle-defined log sources. See Oracle-defined Sources.

  5. Create a new service connector to establish a connection between OCI Streaming and Oracle Logging Analytics to route your custom logs. Specify the target as Logging Analytics and add new source corresponding to your Streaming Custom Logs.

    For detailed steps to create the service connector using the console, API, or CLI, see OCI Service Connector Hub Documentation.

After the connection is established with OCI Streaming, the custom logs will start flowing into Oracle Logging Analytics.

Allow Collection of Logs from OCI Streaming Service

Create policies to enable the users to collect the logs from OCI Streaming and ingest them into Oracle Logging Analytics.

The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements: 

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <userGroup> to MANAGE serviceconnectors in tenancy

In the above policy statements,

  • Log_Group_Compartment_OCID: The compartment OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Log_Group_OCID: The OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector hub.

The following permissions are for collecting logs from OCI Streaming service. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements:

allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment id <Source_Stream_Compartment_OCID>
    where all
    {request.principal.type='serviceconnector',
     target.stream.id='<Stream_OCID>',
     request.principal.compartment.id='<Service_Connector_Compartment_OCID>'}

In the above policy statements,

  • Source_Stream_Compartment_OCID: The compartment OCID of the stream data in OCI Streaming.

  • Stream_OCID: The OCID of the stream in OCI Streaming.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector.