Manage Storage
The log data ingested into Oracle Logging Analytics is available in the active storage for analysis. You can perform the following storage related activities based on your need:
-
Archive Logs: If you want to use your old logs for analysis in the future, then enable archiving and specify the number of days from the log's timestamp after which the log data must be automatically moved from active storage to archive storage which is available at a lesser cost. You can also recall the archived log data for active use. See Archive Log Data.
-
Recall Archived Logs: After the log data is archived, you can recall the selected log data for active use. The logs are selected for recall by specifying the time range in which the timestamps of the logs are present. You can release the recalled logs back to the archive pool after active use. Note that the recalled data will count towards your active storage usage until you release it. See Recall Archived Logs.
-
Release Recalled Logs: Use this option for releasing the recalled logs back into the archive storage to optimize your storage cost. See step 8 in Recall Archived Logs.
-
-
Purge Logs: You can purge the unused or old log data to reduce the size of the active storage that you are consuming. You can perform purge on-demand or create a purge policy. See Purge Log Data.
-
View Storage Activity Report: Use this single-pane window to keep track of all your storage management activities and to perform more management tasks. See View Storage Activity Report.
Your archive policy and recall activity may not complete if the time lines overlap with the purge policy. Make sure to review your purge policy and archival setting to avoid losing log data that must be archived.
Archive Log Data
If you're using only the recent logs for your search and analysis tasks in Oracle Logging Analytics, then enable archiving so that you can optimize the storage cost.
-
You can enable archiving only after you have the minimum specified size of data in active storage. Currently, this is 1 TB.
-
The minimum Active Storage Duration (Days) for logs before they can be archived is 30 days.
-
Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.
-
The administration resources are listed in the left hand navigation pane under Resources. Click Storage.
The Storage page is displayed.
-
Click Enable Archiving. In the Enable Archiving dialog box, enter the count of the days after which the log data in the active storage must be archived in the field Active Storage Duration (Days), and click Enable.
The count is calculated based on the timestamp of the logs. For example, if your logs have the timestamp
November 4, 2020 23:43:12
, and you've specified the Active Storage Duration as30
, then the logs will be typically moved to archive storage onDecember 3, 2020
.Note
It must be noted that even if you specify the Active Storage Duration of the logs to determine the logs that must be moved to Archive storage, the log index structure is based on the buckets that are used for storing the logs. In a typical scenario, an entire bucket is moved to the archive storage when all the logs in it are older than the specified criterion.
For example, consider that the field Active Storage Duration is set to 30 days:
- Bucket_1 has logs of age 40 - 80 days: The log data is eligible and is moved to archive storage.
- Bucket_2 has logs of age 25 - 40 days: Although some of the log data is eligible for archiving, it is not archived until all the logs are suitable for the specified age.
- Bucket_3 has logs of age 0 - 25 days: None of the logs are suitable for archiving. The entire bucket is archived when all the logs become eligible.
In the above scenario, after Bucket_1 logs are archived, if more logs are collected which are older than 40 days, then they are typically appended to Bucket_2.
-
If you have enabled archiving already, and want to modify the archiving settings, then click Modify Archiving Settings. You can perform any of the following tasks:
- You can change the value of the count of the days specified for archiving under Active Storage Duration (Days).
- Click Disable Archiving to stop archiving.
Click Save Changes.
Recall Archived Logs
If you want to use the logs that are archived for viewing and analysis, then you can recall the logs. The recalled data will count towards your active storage usage until you release it.
You can recall and release your selected set of logs multiple times. However, the recall feature is enabled only if you already have archived logs.
-
Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.
-
The administration resources are listed in the left hand navigation pane under Resources. Click Storage.
The Storage page is displayed.
-
In the Storage page, on the left panel under Resources, click Archiving Recall Requests.
The Archiving Recall Requests page displays the previously initiated recall requests.
-
Click Create Recall Request. The Create Recall Request dialog box opens.
-
Specify the Purpose of recall. This can help you to identify your recall request.
-
Optionally, if you have defined log set, then you can specify one or more Log Sets to filter the recalled data. To specify multiple log sets, use comma separation.
-
Select the time range of the logs that you want to recall, by specifying the User-defined start time and User-defined end time.
-
Click Estimate Recall Log Size. The Data set recommended for analysis section opens. The size of the logs that you've selected for recall is displayed adjacent to the heading Maximum recalled data size before filtering.
Note that the start time and end time are extended to align with the log index structure based on buckets. So, when you view the list of active recalls or visit the activity tab, you may get the start and end time extended beyond your chosen time range.
If your current recall time specifications overlap with another recall activity, then they can possibly get merged into a single recall activity and the resulting start and end time can get extended.
-
An alternative time range is recommended based on the availability of data. To select the time range you specified earlier instead of the recommended time range, enable the check box Do not use recommended data set for recall
-
Specify the Query to filter the data set. Exclude the time and log set from the query.
Note that applying the filter does not impact the size of the data set estimated based on the time range.
-
Click Create Recall Request to proceed with the recall of the selected logs.
The recall activity is listed in the Archiving Recall Requests page. The table specifies the status, time range, data size, and request date and time of recall activity, user who initiated the recall, and the purpose of recall. The individual recalls that have overlapping data are combined into a single collection. In such cases, the table displays the data size of the collection instead of the data size of the underlying recalls.
Note
If you keep the recommended and default data set for each recall, then the collection time range is the super set of the time ranges of the individual recalls. Otherwise, the collection time range may not be the super set of the individual recall time ranges.
Watch the status of the recall activity. You can use the recalled logs for viewing and analysis after the recall activity is complete.
If the data size icon for a collection is displayed in orange, then new additional log data is available for recall. Click the data icon and click Recall new data to initiate the recall of the new data. The Recall new data dialog box opens. The query to filter the data set and the time range for data recall are predefined. Specify the purpose of recall and click Create Recall Request.
-
After active use of the recalled logs, if you want to release them back to the archive pool, click the actions menu icon in the row corresponding to your recalled logs, and select Release.
The recalled logs will then be released back into the archive pool. This will enable you to optimize your storage size and cost.
Note
When releasing the recalled logs using REST API, note the recall time range from console or CLI, and format the time as follows:
- Recall start time: Round down (floor) the value. If the recall
start time is
From Mon, Mar 7, 2022, 05:45:33 UTC
, then round down the time and specify it asfrom_time=2022-03-07T5:45:32.000Z
. - Recall end time: Round up (ceil) the value. If the recall end
time is
To Wed, Mar 15, 2023, 17:26:53 UTC
, then round up the time and specify it asto_time=2023-03-15T17:26:54.000Z
.
- Recall start time: Round down (floor) the value. If the recall
start time is
Purge Log Data
Oracle Logging Analytics lets you purge log events that were loaded by agent or by an on-demand upload, to reduce the index size of the log data.
There are multiple ways to purge log data.
- By purging on-demand: All log data from the specified compartment created prior to the selected time range gets purged.
- By creating a purge policy: The old log data can be purged by specifying a schedule for purging and the query to filter the data to purge. If you want to automate the purge activity, then you can create a purge policy by specifying the purge schedule, selecting the log data to purge, and enabling the policy.
Allow Users to Purge Log Data
To purge log data, first set up right permissions by creating the following IAM policies:
-
Create a dynamic group to allow purges for the compartments you want to allow purges in:
ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment ocid>'}
Alternatively, to allow purges on all compartments:
ALL {resource.type='loganalyticsscheduledtask'}
-
Create policies to allow the dynamic group to perform purge operation:
allow dynamic-group <group_name> to read compartments in tenancy allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERY_VIEW} in tenancy allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy allow dynamic-group <group_name> to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy
Note
-
For the proper functioning of the purge policy, the permissions
read compartments
,LOG_ANALYTICS_STORAGE_PURGE
, andLOG_ANALYTICS_QUERY_VIEW
must be created at tenancy level. To restrict the purge action permission to specific compartments, the permissionsLOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE
,LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS
, andLOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ
can be set at the required compartment level. -
In the above policy statements involving dynamic group, if the dynamic group is in a domain other than Default, then the policy statement must be of the format:
allow dynamic-group '<domain>'/'<group_name>' to ...
Enclose the domain name and dynamic group name in single quotes.
-
-
Additionally, ensure that the user has MANAGE permission on loganalytics-features-family and loganalytics-resources-family. If the user creating the on-demand or scheduled purge has Administrator privileges, then the required permissions are already available:
allow group <group_name> to MANAGE loganalytics-features-family in tenancy allow group <group_name> to MANAGE loganalytics-resources-family in tenancy
Some of the above policy statements are included in the readily available Oracle-defined policy templates. You may want to consider using the template for your use case. See Oracle-defined Policy Templates for Common Use Cases.
For information about dynamic groups and IAM policies, see OCI Documentation: Managing Dynamic Groups and OCI Documentation: Managing Policies.
View Storage Activity Report
You can view the summary of your archive, recall, release, and purge activities to maintain close control of your storage use and also to track the status of your key logs that have been part of the activities.
-
Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.
-
The administration resources are listed in the left hand navigation pane under Resources. Click Storage.
The Storage page is displayed.
-
In the left panel under Resources, click the Activity Report.
The page displays the summary of the storage activities initiated such as purge policy, purge on demand, archiving, archiving recall request and recall release.
-
Use the Activity Type, Status, and Time filters on the left panel to narrow down your search for the storage activities.
-
Expand the storage activity row to view more details about it.