You can collect log data continuously from Oracle Cloud Infrastructure (OCI)
Object Storage. To enable the log collection, create ObjectCollectionRule resource
using REST API or CLI. After the successful creation of this resource and having the
required IAM policies, the log collection will be initiated.
You can use this method of collecting logs to ingest any type of log stored
in an object storage bucket.
You can collect logs from Object Storage bucket in one of the following ways:
LIVE: For continuous collection of objects from the time of
creation of the rule ObjectCollectionRule. This is the default
method.
HISTORIC: For onetime collection of objects for a specified
time range.
HISTORIC_LIVE: For collection of all historic logs in the
bucket, and after that, continuous collection of all newly created objects
containing logs.
Oracle Logging Analytics uses
Events and Streaming services of OCI in conjunction with Object storage to
collect and process objects. When you configure a bucket for log collection, Oracle Logging Analytics creates an
Events rule to emit event notification for every new object upload to the
bucket. The notifications will be delivered to a stream owned by Oracle Logging Analytics.
Note:
For proper functioning of log collection from object storage, ensure
that the Event rules created by Oracle Logging Analytics are not tampered with.
Per bucket, you can have only one ObjectCollectionRule of
type LIVE or HISTORIC_LIVE.
You can create up to 1000 unique object collection rules per tenancy in a
region.
The object can be a single raw log file or any archive file
(.zip, .gz, .tgz,
.tar) containing multiple log files. The number of files
inside an archive file must be less than 2000, including the directories if
any.
The maximum size of the object (single file or archive file) is 1 GB. The
uncompressed size of the object must be less than 10 GB.
Prerequisites: Before enabling log collection using this approach,
ensure to:
Create a new log source or use an Oracle-defined log source that matches your log
format. See Create a Source.
Create a log group or use an existing log group where you will store
these logs to control the user access control to the logs and note the log group
OCID. See Create Log Groups to Store Your Logs.
To stop the collection of objects from the bucket, delete the
ObjectCollectionRule rule. This will only delete the associated configuration
with the bucket but will have no effect on the already collected log data or your
objects in the bucket.
Allow Log Collection from Object
Storage 🔗
The following IAM policy statements must be included in your policy to provide the
permission to the user group for performing the required operations on
ObjectCollectionRule:
allow group <group_name> to use loganalytics-object-collection-rule in compartment <object_collection_rule_compartment>
allow group <group_name> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
allow group <group_name> to {LOG_ANALYTICS_SOURCE_READ} in tenancy
allow group <group_name> to {BUCKET_UPDATE, BUCKET_READ} in compartment <object_store_bucket_compartment>
If you are creating IAM policies at Oracle Logging Analytics aggregate resources level, then the following policy statements must
be included to use object collection:
allow group <group_name> to use loganalytics-features-family in tenancy
allow group <group_name> to use loganalytics-resources-family in compartment/tenancy
allow group <group_name> to use object-family in compartment <object_store_bucket_compartment>
On the other hand, if you are creating IAM policies at the level of individual
resource-types, then the following policy statements are required to use object
collection:
allow group <group_name> to use loganalytics-object-collection-rule in compartment <object_collection_rule_compartment>
allow group <group_name> to use loganalytics-log-group in compartment <log_group_compartment>
allow group <group_name> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <entity_compartment>
allow group <group_name> to read loganalytics-source in tenancy
allow group <group_name> to use object-family in compartment <object_store_bucket_compartment>
group_name in all the above policy statements refers to the user
group that must be given the required permissions.
Note
When you delete and recreate a bucket, for the existing log collection rule to work,
set the flag Emit Objects Events for the bucket after it is recreated.
For log collection to work, along with the above permissions for Creating
ObjectCollectionRule, you must also give permission to Oracle Logging Analytics to read the objects
from the bucket in your tenancy and manage the cloud event rules in corresponding
compartment or tenancy where the object storage bucket is located. The following are the
additional IAM policies required:
allow service loganalytics to read buckets in compartment/tenancy
allow service loganalytics to read objects in compartment/tenancy
allow service loganalytics to manage cloudevents-rules in compartment/tenancy
allow service loganalytics to inspect compartments in tenancy
allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags/}
By default, Object Storage disables automatic emission of events at object
level. You can either enable emission of events or have required permissions while
creating ObjectCollectionRule. To enable event emission, see Managing Objects.
Some of the above policy statements are included in the readily
available Oracle-defined policy templates. You may want to consider using the template
for your use case. See Oracle-defined Policy Templates for Common Use Cases.
ObjectCollectionRule Operations
🔗
Using REST API or CLI, you can perform operations such as Create,
Update, Delete, List, and
Get on the ObjectCollectionRule resource.
To communicate with OCI cloud services, create an API Signing Key and
register it in your user account in OCI. To generate and register the key and to collect
the Tenancy's OCID and user's OCID, see Security Credentials - API Signing
Key.
The following mandatory properties must be provided in the json:
name: A unique name given to the ObjectCollectionRule. The
name must be unique within the tenancy, and cannot be modified.
compartmentId: The OCID of the compartment in which the
ObjectCollectionRule is located.
osNamespace: The object storage namespace.
osBucketName: The name of the object storage bucket.
logGroupId: Logging AnalyticsLogging Analytics log group OCID to associate the
processed logs with.
logSourceName: Name of the Logging Analytics source to use for the processing.
In addition to the mandatory properties, you can also optionally
specify the following properties:
collectionType: The type of collection. Allowed values are
LIVE, HISTORIC, and
HISTORIC_LIVE.
charEncoding: Example values ISO_8859_1,
UTF-16
definedTags: Defined tags for this resource. Each key is
predefined and scoped to a namespace. For example,
{"foo-namespace": {"bar-key": "value"}}
description: A string that describes the details of the rule, not
more than 400 characters long.
entityId: OCID of the Logging Analytics entity with which the collected logs will be
associated.
freeformTags: Simple key-value pair that is applied without any
predefined name, type or scope. Exists for cross-compatibility only. For
example, {"bar-key": "value"}
Update ObjectCollectionRule: Following is a sample command to update a
log source. Similarly, log group, entity, or any other resource can also be
updated.
oci log-analytics object-collection-rule list --namespace-name <Namespace> --compartment-id <compartment-OCID>
For Example:
oci log-analytics object-collection-rule list --namespace-name “My Namespace” --compartment-id ocid.compartment.oc1..exampleuniqueID
Get ObjectCollectionRule:
oci log-analytics object-collection-rule get --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID>
For Example:
oci log-analytics object-collection-rule get --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID
Add Overrides to ObjectCollectionRule:
If the ObjectCollectionRule already exists, create an
override json, for example update_override.json, with the
override conditions that you want to add to ObjectCollectionRule:
If you want to remove all the override conditions from the
ObjectCollectionRule, then create a json, for example
remove_overrides.json, that details the override
property as follows:
If the ObjectCollectionRule already exists, create a
filters json, for example filters.json, with the filters on
object names that you want to add to ObjectCollectionRule:
{
"objectNameFilters":["a/*","*audit*"]
}
Note
You can update an existing
ObjectCollectionRule with ObjectNameFilters only if it is
of the type LIVE or HISTORIC_LIVE. Type
HISTORIC is not supported for this operation.
Now update the ObjectCollectionRule to include the filters:
Allow Cross-Tenancy Log
Collection from Object Storage 🔗
Set the following policies to configure the object collection rule for collecting
logs from a bucket in a guest tenant.
Let Guest_Tenant be the guest tenant and
Guest_Compartment the compartment in that guest tenant which has
the object storage buckets from which the logs must be collected. Let
Host_Tenant be the tenant which is subscribed to Oracle Logging Analytics.
For additional information about writing policies that let your tenancy access Object
Storage resources in other tenancies, see Accessing Object Storage Resources Across
Tenancies in Oracle Cloud Infrastructure Documentation.
Policies To Be Added in the Guest Tenant
Here is an example of policy statements which allow the IAM group
Host_User_Group from the host tenant
Host_Tenant and the Oracle Logging Analytics service to access the objects in the guest compartment
Guest_Compartment in the guest tenancy:
define group <Host_User_Group> as <Host_User_Group_OCID>
define tenancy <Host_Tenant> as <Host_Tenant_OCID>
admit group <Host_User_Group> of tenancy <Host_Tenant> to use buckets in compartment <Guest_Compartment>
allow service loganalytics to read buckets in compartment <Guest_Compartment>
allow service loganalytics to read objects in compartment <Guest_Compartment>
allow service loganalytics to manage cloudevents-rules in compartment <Guest_Compartment>
allow service loganalytics to inspect compartments in tenancy
allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags /}
Policies To Be Added in the Host Tenant
Here is an example of policy statements which allow the host IAM group
Host_User_Group to have USE access to the
buckets in the guest tenancy Guest_Tenant:
define tenancy <Guest_Tenant> as <Guest_Tenant_OCID>
endorse group <Host_User_Group> to use buckets in tenancy <Guest_Tenant>
allow group <Host_User_Group> to use loganalytics-object-collection-rule in compartment <Rule_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <LogGroup_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_SOURCE_READ} in tenancy
Optionally, define this policy if ObjectCollectionRule has associated
entities:
allow group <Host_User_Group> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <Entity_Compartment>
In the above policies,
Rule_Compartment: The compartment in which
ObjectCollectionRule must be created.
LogGroup_Compartment: The compartment of Oracle Logging Analytics log group
in which the logs must be stored.
Entity_Compartment: The compartment of Oracle Logging Analytics
entity.
After the required policies are created, you can create
ObjectCollectionRule to collect the logs from guest tenancy Object
Storage. Provide the namespace osNamespace and bucket name
osBucketName of the guest tenant in the JSON file, as shown in
the following example:
{
"name": "<My_Rule>",
"compartmentId": "<Compartment_OCID>",
"osNamespace": "<Guest_Tenant_Namespace>", // Namespace of the guest tenant
"osBucketName": "<Guest_Tenant__Bucket1>", // Bucket in the guest tenant object store namespace
"logGroupId": "<Log_Group_OCID>",
"logSourceName": "<My_Log_Source>"
}
Override
ObjectCollectionRule Configuration to Process Specific Objects
🔗
When you want to process specific objects inside a bucket by using a
configuration that's different from the one defined for ObjectCollectionRule, you can
use the override feature. The override feature facilitates to provide configuration
properties based on object names matching specific patterns or prefixes or directories
inside the bucket.
The properties for which you can specify overrides:
logSourceName, charEncoding,
entityId, timezone
The match type that you can use in the override: contains
The match value is always applied on the object name
If you are creating the ObjectCollectionRule for the first time, then one of the
following examples of the create.json can be used to build the json for your use
case:
Sample
create.json with overrides to process object names containing audit
with logSourceName
myLOGANAuditSource:
Using the above rule, the objects containing
audit are processed with logSourceName
myLOGANAuditSource while all other objects in the bucket
are processed with the logSourceName
myLOGANSourceName.
Sample create.json with overrides to process object names
containing audit with logSourceName
myLOGANAuditSource and those object names containing
dir1/ with charEncoding
UTF-16:
After creating the configuration json, specify the file path when you create the
ObjectCollectionRule. See section Create ObjectCollectionRule in ObjectCollectionRule Operations.
If you have already created the ObjectCollectionRule, then create a json with the
details of the override and update the ObjectCollectionRule to include it. See
section Add Overrides to ObjectCollectionRule in ObjectCollectionRule Operations.
If you want to remove all the override conditions, then create a json as
specified in the section Remove Overrides from ObjectCollectionRule and update
the ObjectCollectionRule. See ObjectCollectionRule Operations.
Perform Selective Object
Collection by Applying Filters on Object Names 🔗
Use the Selective Objects Collection feature to collect only a subset of
objects in a given object storage bucket. The feature is supported with filters on object
names. When the filters are applied, only the objects matching the filters are collected for
processing.
The matchType property only supports exact match along with the
wildcard *. Following are some of the examples of filters using the
wildcard:
Filter objectName* specifies objects with the prefix objectName.
Filter *objectName specifies objects with the suffix objectName.
Filter *objectName* specifies objects that contain the text objectName.
If you are creating the ObjectCollectionRule for the first time, then one of the
following examples of the create.json can be used to build the json for your use
case:
Sample create.json with objectNameFilters to process object names that
have a prefix a/ and contain the text
audit:
If required, you can also use the override feature to specify different
configuration for each of the filters. For the above example, you can additionally
specify that all the objects containing the text audit must use the
source
myLOGANAuditSource:
After creating the configuration json, specify the file path when you create
the ObjectCollectionRule. See section Create ObjectCollectionRule in ObjectCollectionRule Operations.
If you have already created the ObjectCollectionRule, then create a
json with the details of the filters and update the ObjectCollectionRule to
include it. See section Add objectNameFilters to ObjectCollectionRule in ObjectCollectionRule Operations.