Ingest Logs from Other OCI Services Using Service Connector

You can analyze the logs to troubleshoot issues, monitor health and performance and observe the operational tasks in Oracle Cloud Infrastructure services by ingesting the logs into Oracle Logging Analytics.

Use the Service Connector to identify your Oracle Cloud Infrastructure service as the source of the logs and Oracle Logging Analytics as the destination. For information on how the Service Connector Hub works, see Service Connector Hub Overview in Oracle Cloud Infrastructure Documentation.

Note

After the service connector is created, an entity is automatically created for processing the logs. To ensure proper log collection, the entity must not be deleted.

In case of Oracle Operator Access Control Logs, the entity is not automatically created. To create an entity, see Create an Entity to Represent Your Log-Emitting Resource.

Topics

Important: Oracle recommends that you use the data ingestion work flow available in Logging Analytics console to quickly ingest logs from other OCI services. Go to Logging Analytics Home or Log Explorer, click Compass, and click Add Data.

  • For all types of logs from OCI services except OCI Audit Logs and IDCS Audit Logs, expand the section Monitor OCI resources and click Configure log collection for OCI resources.
  • In case of OCI Audit Logs or IDCS Audit Logs, expand the section Security and Compliance and click the logs of your choice. In this work flow, all the required resources like policies, log group, and service connector are automatically created.

Follow the intuitive steps in the work flow to start ingesting logs. As a prerequisite, ensure that you have the required permissions to complete the steps. For a quick walk through of the steps, watch Video: How to Quickly Ingest Logs into Logging Analytics from Other OCI Services in Oracle Cloud Observability and Management Platform.

Alternatively, you can manually set up the log collection by performing the following steps:

Additional Information

  • List of Oracle-defined sources for collecting logs: For the list of Oracle-defined sources to collect logs from Oracle Cloud Infrastructure services, see Oracle-defined Sources and search for sources with title OCI...

  • Types of service logs you can collect: For the types of logs you can collect from the Oracle Cloud Infrastructure services, the parsers, example log content, fields, and JSON path, see OCI Parser Details.

  • Filter logs collected though service connector: The service connector OCID is mapped to the field Log Origin. To view the logs flowing from that service connector to Oracle Logging Analytics, filter the logs by the field Log Origin. See Filter Logs by Pinned Attributes and Fields.

Allow Collection of Logs from OCI Logging Service

Based on the type of service logs that you want to ingest, you must create policies to enable Oracle Logging Analytics to get the information about the resources and create an entity for each resource.

After you create the policy, the entity that is created will be auto-associated with all the logs collected from that resource. If the policy is not created, then the logs are still ingested but the entity is not created.

The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements:

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <userGroup> to MANAGE serviceconnectors in tenancy
allow group <userGroup> to READ logging-family in tenancy

In the above policy statements,

  • Log_Group_Compartment_OCID: The compartment OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Log_Group_OCID: The OCID of the log group in Oracle Logging Analytics where the logs must be stored.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector hub.

Note

If you enabled Oracle Logging Analytics using the onboarding UI which is available when you navigate to the service for the first time, then some policies are already created. See Policies Created While Onboarding Logging Analytics.

Policy for Each Type of Service Logs

Oracle Logging Analytics creates an entity representing the underlying OCI resource when new logs are received through the service connector. In order to obtain the necessary information from the OCI resource, you must provide Oracle Logging Analytics with a minimum of read access to the OCI resource. For example, in order to read information about a VNIC, you can write one of the following policies:

Policy statement with the READ PRIVILEGE of the OCI resource:

allow service loganalytics to {VNIC_READ} in compartment <specify_compartment>

OR

Policy statement with the read verb for the OCI RESOURCE:

allow service loganalytics to read vnics in compartment <specify_compartment>

The above policy statements restrict the read access to a compartment. To extend the access to the entire tenancy, you can change the policy statement accordingly.

The following OCI resources are supported in Oracle Logging Analytics for log collection through the service connector. You can either create the policy using read verb for the OCI resource or use the read privilege for the resource as illustrated above.

OCI Resource Description OCI Resource Read Privilege
Analytics Cloud Instance analytics-instances ANALYTICS_INSTANCE_READ
API Gateway api-gateways API_GATEWAY_READ
APM Domain apm-domains APM_DOMAIN_READ
Container Engine For Kubernetes clusters CLUSTER_READ
Data Flow (Application) dataflow-application DATAFLOW_APPLICATION_READ
Data Integration Workspace dis-workspaces DIS_WORKSPACE_READ
DevOps Build Pipeline devops-build-pipeline DEVOPS_BUILD_PIPELINE_READ
DevOps Build Pipeline Stage devops-build-pipeline-stage DEVOPS_BUILD_PIPELINE_STAGE_READ
DevOps Build Run devops-build-run DEVOPS_BUILD_RUN_READ
DevOps Deployment devops-deployment DEVOPS_DEPLOY_DEPLOYMENT_READ
DevOps Deployment Pipeline devops-deploy-pipeline DEVOPS_DEPLOY_PIPELINE_READ
DevOps Deployment Stage devops-deploy-stage DEVOPS_DEPLOY_STAGE_READ
Email Delivery Service approved-senders APPROVED_SENDER_READ
Events Service cloudevents-rules EVENTRULE_READ
Functions (FN App) fn-app FN_APP_READ
Functions (FN Function) fn-function FN_FUNCTION_READ
GoldenGate Deployment goldengate-deployments GOLDENGATE_DEPLOYMENT_READ
Instance instances INSTANCE_READ
IPSec Tunnel ipsec-connections IPSEC_CONNECTION_READ
Load Balancer load-balancers LOAD_BALANCER_READ
Media Workflow media-workflow MEDIA_WORKFLOW_READ
Media Workflow Job media-workflow-job MEDIA_WORKFLOW_JOB_READ
Network Firewall network-firewall NETWORK_FIREWALL_READ
Object Storage (Bucket) buckets BUCKET_READ
OIC Instance integration-instance INTEGRATION_INSTANCE_READ
Operator Control operator-control-family -
Service Connector serviceconnectors SERVICE_CONNECTOR_READ
VCN - VNIC vnics VNIC_READ
Web Application Firewall web-app-firewall WEB_APP_FIREWALL_READ
Note

If you enabled Oracle Logging Analytics using the onboarding UI which is available when you navigate to the service for the first time, then some policies are already created. See Policies Created While Onboarding Logging Analytics.

Set Up the Service Connector to Ingest Logs

Before you set up the service connector to ingest logs, ensure that the compartment and log group are identified for the logs that you want to ingest.

In the following example, the steps show you how to collect VCN service logs from Oracle Cloud Infrastructure Logging service:

  1. This is a suggestive step to show you how to enable logs in the Oracle Cloud Infrastructure Logging service.

    Go to Oracle Cloud Infrastructure Logging service > Go to Logs.

    Click Enable Resource Log to enable VCN service logs. The dialog box opens.

    1. Select the resource compartment.
    2. Select the service, for example, Virtual Cloud Network (subnets).
    3. Select the resource, for example, the VCN resource.
    4. Under Configure Log, select the log category, for example, Flow Logs, and the log name.
    5. Under Log Location, select the compartment and log group that Oracle Logging Analytics will refer the logs from.

    Click Enable Log.

  2. Set up the service connector by specifying the source service of the logs and the target as Oracle Logging Analytics. You can either set it up from the source service that has integrated with Oracle Cloud Infrastructure Service Connector Hub, for example, Oracle Cloud Infrastructure Logging service, or directly from Oracle Cloud Infrastructure Service Connector Hub.

    Go to Oracle Cloud Infrastructure Logging service > Go to Service Connectors > Click Create Connector.

    Alternatively, go to Oracle Cloud Infrastructure Service Connector Hub service > Click Create Service Connector.

    The Create Service Connector page opens.

    1. Enter a name for the connector and provide a description.
    2. Select the resource compartment where the connector resource must be created.
    3. Under Configure Service Connector, specify Logging as the Source service, and Logging Analytics as the Target service.
    4. Under Configure Source Connection, provide the details of the logs to collect from the service, for example, the VCN service logs.

      Select the compartment name, the log group to which the logs belong, and the name of the logs that you had configured in step 1.

    You can configure the same service connector to collect more logs. Click Another Log and repeat step 2-d.

    Optionally, you can create filters under Configure Task.

    Click Create Connector.

After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.

Allow Cross-Tenancy Log Collection from OCI Logging Service

Let Source_Tenant be the tenant of the source service such as Oracle Cloud Infrastructure Logging from which logs are collected. Let Target_Tenant be the tenant in which the service connector is created. The service connector is configured with Oracle Logging Analytics as the target for the logs that are collected from the source service. It is assumed that the service connector hub and Oracle Logging Analytics are available on the same target tenant.

Set the following policies to configure the log collection from a tenancy that is different from the tenancy the service connector is created in.

Policies To Be Added in the Source Tenant

Here is an example of policy statements which allow any user of the service connector hub tenancy to have READ access to the Logging service:

define tenancy <Target_Tenant> as <Target_Tenant_OCID>
define group <Common_User_Group> as <Common_User_Group_OCID>
admit any-user of tenancy <Target_Tenant> to read logging-family IN TENANCY WHERE ALL {request.principal.type = 'serviceconnector'}
admit group <Common_User_Group> of tenancy <Target_Tenant> to read logging-family IN TENANCY

Ensure to set the policy for the type of service logs that must be collected from the source service. See Allow Collection of Logs from OCI Logging Service.

Policies To Be Added in the Target Tenant

Here is an example of policy statements which allow any user to access the Logging service through the service connector hub, and the target IAM group Common_User_Group to have MANAGE access to the service connector hub:

define tenancy <Source_Tenant> as <Source_Tenant_OCID>
endorse any-user to read logging-family IN tenancy <Source_Tenant> WHERE ALL {request.principal.type = 'serviceconnector'}
endorse group <Common_User_Group> to read logging-family IN tenancy <Source_Tenant>

The following permissions are for uploading logs to Oracle Logging Analytics from the service connector. Make sure to manually create the policy that includes the following policy statements:

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <Common_User_Group> to MANAGE serviceconnectors in tenancy

In the above policy statements,

  • Log_Group_OCID: The OCID of the Oracle Logging Analytics log group.

  • Log_Group_Compartment_OCID: The OCID of the compartment where the Oracle Logging Analytics log group is located.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector.

  • Common_User_Group: The user group that creates the service connector.

Create a Connector Between the Source and Target Tenants

After the required policies are created for the source and target tenants, create a service connector using CLI. The following example CLI command specifies Logging as the source and Oracle Logging Analytics as the target for creating the cross-tenancy service connector:

oci --profile <Target_Profile> sch service-connector create 
    --display-name XTenancyConnector 
    --compartment-id <Connector_Compartment_OCID> 
    --source '{ "kind": "logging", "logSources": 
        [ { "compartmentId": "<Logging_LogGroup_Compartment_OCID>", 
            "logGroupId": "<Logging_LogGroup_OCID>" } ] }' 
    --target '{ "kind": "loggingAnalytics", "logGroupId": "<LoggingAnalytics_LogGroup_OCID>" }'

The above command is formatted for better readability. Remove characters like new line, tab and additional spaces before running it.

In the above CLI command,

  • Target_Profile: The profile in the .oci/config file that maps to the target tenancy.

  • Connector_Compartment_OCID: The OCID of the compartment where the service connector resource is created.

  • Logging_LogGroup_Compartment_OCID: The OCID of the compartment the Oracle Cloud Logging log group belongs to. This is in the source tenant.

  • Logging_LogGroup_OCID: The OCID of the Oracle Cloud Logging log group. This is in the source tenant.

  • LoggingAnalytics_LogGroup_OCID: The OCID of the Oracle Logging Analytics log group. This is in the target tenant.

For more details about the CLI command, see CLI Command Reference - Create.

After the service connector is created, you can verify that the selected logs are available in Oracle Logging Analytics.