Ingest Logs from Other OCI Services Using Service Connector

You can analyze the logs to troubleshoot issues, monitor health and performance and observe the operational tasks in Oracle Cloud Infrastructure services by ingesting the logs into Oracle Logging Analytics.

Use the Service Connector to identify your Oracle Cloud Infrastructure service as the source of the logs and Oracle Logging Analytics as the destination. For information on how the Service Connector Hub works, see Service Connector Hub Overview in Oracle Cloud Infrastructure Documentation.

Note

After the service connector is created, an entity is automatically created for processing the logs. To ensure proper log collection, the entity must not be deleted.

In case of Oracle Operator Access Control Logs, the entity is not automatically created. To create an entity, see Create an Entity to Represent Your Log-Emitting Resource.

Topics

Important: Oracle recommends that you use the data ingestion work flow available in Logging Analytics console to quickly ingest logs from other OCI services. Go to Logging Analytics Home or Log Explorer, click Compass, and click Add Data.

  • For all types of logs from OCI services except OCI Audit Logs and IDCS Audit Logs, expand the section Monitor OCI resources and click Configure log collection for OCI resources.
  • In case of OCI Audit Logs or IDCS Audit Logs, expand the section Security and Compliance and click the logs of your choice. In this work flow, all the required resources like policies, log group, and service connector are automatically created.

Follow the intuitive steps in the work flow to start ingesting logs. As a prerequisite, ensure that you have the required permissions to complete the steps. For a quick walk through of the steps, watch Video: How to Quickly Ingest Logs into Logging Analytics from Other OCI Services in Oracle Cloud Observability and Management Platform.

Alternatively, you can manually set up the log collection by performing the following steps:

Additional Information

  • List of Oracle-defined sources for collecting logs: For the list of Oracle-defined sources to collect logs from Oracle Cloud Infrastructure services, see Oracle-Defined Sources and search for sources with title OCI...

  • Types of service logs you can collect: For the types of logs you can collect from the Oracle Cloud Infrastructure services, the parsers, example log content, fields, and JSON path, see OCI Parser Details.

  • Filter logs collected though service connector: The service connector OCID is mapped to the field Log Origin. To view the logs flowing from that service connector to Oracle Logging Analytics, filter the logs by the field Log Origin. See Filter Logs by Pinned Attributes and Fields.

Allow Collection of Logs from OCI Logging Service

Based on the type of service logs that you want to ingest, you must create policies to enable Oracle Logging Analytics to get the information about the resources and create an entity for each resource.

After you create the policy, the entity that is created will be auto-associated with all the logs collected from that resource. If the policy is not created, then the logs are still ingested but the entity is not created.

The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements:

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <userGroup> to MANAGE serviceconnectors in tenancy
allow group <userGroup> to READ logging-family in tenancy

In the above policy statements,

  • Log_Group_Compartment_OCID: The compartment OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Log_Group_OCID: The OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector hub.

Policy for each type of service logs:

  • API Gateway Logs

    allow service loganalytics to read api-gateway-family in tenancy
  • OCI Application Performance Monitoring Logs

    allow service loganalytics to {APM_DOMAIN_READ} in tenancy
  • Event Service Logs

    allow service loganalytics to {EVENTRULE_READ} in tenancy
  • Function Logs

    allow service loganalytics to read functions-family in tenancy
  • Load Balancer Logs

    allow service loganalytics to {LOAD_BALANCER_READ} in tenancy
  • Media Flow Logs

    allow service loganalytics to {MEDIA_WORKFLOW_READ} in tenancy
  • Network Firewall Logs

    allow service loganalytics to {NETWORK_FIREWALL_READ} in tenancy
  • Object Storage Logs

    allow service loganalytics to {BUCKET_READ} in tenancy
  • OCI DevOps Logs

    allow service loganalytics to {DEVOPS_DEPLOYMENT_READ} in tenancy
    allow service loganalytics to {DEVOPS_DEPLOY_PIPELINE_READ} in tenancy
    allow service loganalytics to {DEVOPS_DEPLOY_STAGE_READ} in tenancy
  • OCI DevOps Build Logs

    allow service loganalytics to {DEVOPS_BUILD_PIPELINE_READ} in tenancy
    allow service loganalytics to {DEVOPS_BUILD_PIPELINE_STAGE_READ} in tenancy
    allow service loganalytics to {DEVOPS_BUILD_RUN_READ} in tenancy
  • OCI Email Delivery Logs

    allow service loganalytics to {APPROVED_SENDER_READ} in tenancy
  • Oracle Operator Access Control Logs

    allow service loganalytics to read operator-control-family in tenancy
  • OCI Site-to-Site VPN Logs (IPSec Connection)

    allow service loganalytics to {IPSEC_CONNECTION_READ} in tenancy
  • OCI WAF Logs

    allow service loganalytics to {WEB_APP_FIREWALL_READ} in tenancy
  • Virtual Cloud Network Logs

    allow service loganalytics to {VNIC_READ} in tenancy

Set Up the Service Connector to Ingest Logs

Before you set up the service connector to ingest logs, ensure that the compartment and log group are identified for the logs that you want to ingest.

In the following example, the steps show you how to collect VCN service logs from Oracle Cloud Infrastructure Logging service:

  1. This is a suggestive step to show you how to enable logs in the Oracle Cloud Infrastructure Logging service.

    Go to Oracle Cloud Infrastructure Logging service > Go to Logs.

    Click Enable Resource Log to enable VCN service logs. The dialog box opens.

    1. Select the resource compartment.
    2. Select the service, for example, Virtual Cloud Network (subnets).
    3. Select the resource, for example, the VCN resource.
    4. Under Configure Log, select the log category, for example, Flow Logs, and the log name.
    5. Under Log Location, select the compartment and log group that Oracle Logging Analytics will refer the logs from.

    Click Enable Log.

  2. Set up the service connector by specifying the source service of the logs and the target as Oracle Logging Analytics. You can either set it up from the source service that has integrated with Oracle Cloud Infrastructure Service Connector Hub, for example, Oracle Cloud Infrastructure Logging service, or directly from Oracle Cloud Infrastructure Service Connector Hub.

    Go to Oracle Cloud Infrastructure Logging service > Go to Service Connectors > Click Create Connector.

    Alternatively, go to Oracle Cloud Infrastructure Service Connector Hub service > Click Create Service Connector.

    The Create Service Connector page opens.

    1. Enter a name for the connector and provide a description.
    2. Select the resource compartment where the connector resource must be created.
    3. Under Configure Service Connector, specify Logging as the Source service, and Logging Analytics as the Target service.
    4. Under Configure Source Connection, provide the details of the logs to collect from the service, for example, the VCN service logs.

      Select the compartment name, the log group to which the logs belong, and the name of the logs that you had configured in step 1.

    You can configure the same service connector to collect more logs. Click Another Log and repeat step 2-d.

    Optionally, you can create filters under Configure Task.

    Click Create Connector.

After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.

Allow Cross-Tenancy Log Collection from OCI Logging Service

Let Source_Tenant be the tenant of the source service such as Oracle Cloud Infrastructure Logging from which logs are collected. Let Target_Tenant be the tenant in which the service connector is created. The service connector is configured with Oracle Logging Analytics as the target for the logs that are collected from the source service. It is assumed that the service connector hub and Oracle Logging Analytics are available on the same target tenant.

Set the following policies to configure the log collection from a tenancy that is different from the tenancy the service connector is created in.

Policies To Be Added in the Source Tenant

Here is an example of policy statements which allow any user of the service connector hub tenancy to have READ access to the Logging service:

define tenancy <Target_Tenant> as <Target_Tenant_OCID>
define group <Common_User_Group> as <Common_User_Group_OCID>
admit any-user of tenancy <Target_Tenant> to read logging-family IN TENANCY WHERE ALL {request.principal.type = 'serviceconnector'}
admit group <Common_User_Group> of tenancy <Target_Tenant> to read logging-family IN TENANCY

Ensure to set the policy for the type of service logs that must be collected from the source service. See Allow Collection of Logs from OCI Logging Service.

Policies To Be Added in the Target Tenant

Here is an example of policy statements which allow any user to access the Logging service through the service connector hub, and the target IAM group Common_User_Group to have MANAGE access to the service connector hub:

define tenancy <Source_Tenant> as <Source_Tenant_OCID>
endorse any-user to read logging-family IN tenancy <Source_Tenant> WHERE ALL {request.principal.type = 'serviceconnector'}
endorse group <Common_User_Group> to read logging-family IN tenancy <Source_Tenant>

The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. Make sure to manually create the policy that includes the following policy statements:

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <Common_User_Group> to MANAGE serviceconnectors in tenancy

In the above policy statements,

  • Log_Group_OCID: The OCID of the Oracle Logging Analytics log group.

  • Log_Group_Compartment_OCID: The OCID of the compartment where the Oracle Logging Analytics log group is located.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector.

  • Common_User_Group: The user group that creates the service connector.

Create a Connector Between the Source and Target Tenants

After the required policies are created for the source and target tenants, create a service connector using CLI. The following example CLI command specifies Logging as the source and Oracle Logging Analytics as the target for creating the cross-tenancy service connector:

oci --profile <Target_Profile> sch service-connector create 
    --display-name XTenancyConnector 
    --compartment-id <Connector_Compartment_OCID> 
    --source '{ "kind": "logging", "logSources": 
        [ { "compartmentId": "<Logging_LogGroup_Compartment_OCID>", 
            "logGroupId": "<Logging_LogGroup_OCID>" } ] }' 
    --target '{ "kind": "loggingAnalytics", "logGroupId": "<LoggingAnalytics_LogGroup_OCID>" }'

The above command is formatted for better readability. Remove characters like new line, tab and additional spaces before running it.

In the above CLI command,

  • Target_Profile: The profile in the .oci/config file that maps to the target tenancy.

  • Connector_Compartment_OCID: The OCID of the compartment where the service connector resource is created.

  • Logging_LogGroup_Compartment_OCID: The OCID of the compartment the Oracle Cloud Logging log group belongs to. This is in the source tenant.

  • Logging_LogGroup_OCID: The OCID of the Oracle Cloud Logging log group. This is in the source tenant.

  • LoggingAnalytics_LogGroup_OCID: The OCID of the Oracle Logging Analytics log group. This is in the target tenant.

For more details about the CLI command, see CLI Command Reference - Create.

After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.