The log data ingested into Oracle Logging Analytics is available in the active storage for analysis. You can perform
the following storage related activities based on your need:
Archive Logs: If you want to use your old logs for
analysis in the future, then enable archiving and specify the number of days
from the log's timestamp after which the log data must be automatically
moved from active storage to archive storage which is available at a lesser
cost. You can also recall the archived log data for active use. See Archive Log Data.
Recall Archived Logs: After the log data is
archived, you can recall the selected log data for active use. The
logs are selected for recall by specifying the time range in which
the timestamps of the logs are present. You can release the recalled
logs back to the archive pool after active use. Note that the
recalled data will count towards your active storage usage until you
release it. See Recall Archived Logs.
Release Recalled Logs: Use this option for
releasing the recalled logs back into the archive storage to
optimize your storage cost. See step 8 in Recall Archived Logs.
Purge Logs: You can purge the unused or old log data to reduce the size of the
active storage that you are consuming. You can perform purge
on-demand or create a purge policy. See Purge Log Data.
View Storage Activity Report: Use this single-pane window
to keep track of all your storage management activities and to perform more
management tasks. See View Storage Activity Report.
Note
Your archive policy and recall activity may not complete if
the time lines overlap with the purge policy. Make sure to review your purge policy
and archival setting to avoid losing log data that must be archived.
Archive Log Data π
If you're using only the recent logs for your search and analysis tasks in
Oracle Logging Analytics, then enable
archiving so that you can optimize the storage cost.
Note
You can enable archiving only after you have the minimum
specified size of data in active storage. Currently, this is 1
TB.
The minimum Active Storage Duration (Days) for logs
before they can be archived is 30 days.
Open the navigation
menu and click Observability & Management. Under
Logging Analytics, click
Administration. The Administration
Overview page opens.
The administration resources are listed in the left hand navigation
pane under Resources. Click
Storage.
The Storage page is displayed.
Click Enable Archiving. In the Enable
Archiving dialog box, enter the count of the days after which the log data in
the active storage must be archived in the field Active Storage
Duration (Days), and click Enable.
The count is calculated based on the timestamp of the logs. For
example, if your logs have the timestamp November 4, 2020
23:43:12, and you've specified the Active Storage Duration as
30, then the logs will be typically moved to archive
storage on December 3, 2020.
Note
It must be noted that even if you specify the Active Storage Duration of the
logs to determine the logs that must be moved to Archive storage, the log
index structure is based on the buckets that are used for storing the logs.
In a typical scenario, an entire bucket is moved to the archive storage when
all the logs in it are older than the specified criterion.
For example, consider that the field Active Storage
Duration is set to 30 days:
Bucket_1 has logs of age 40 - 80 days: The log data is eligible
and is moved to archive storage.
Bucket_2 has logs of age 25 - 40 days: Although some of the log
data is eligible for archiving, it is not archived until all the logs
are suitable for the specified age.
Bucket_3 has logs of age 0 - 25 days: None of the logs are
suitable for archiving. The entire bucket is archived when all the logs
become eligible.
In the above scenario, after Bucket_1 logs are archived,
if more logs are collected which are older than 40 days, then they are
typically appended to Bucket_2.
If you have enabled archiving already, and want to modify the
archiving settings, then click Modify Archiving Settings.
You can perform any of the following tasks:
You can change the value of the count of the days specified for archiving
under Active Storage Duration (Days).
Click Disable Archiving to stop archiving.
Click Save Changes.
Recall Archived Logs π
If you want to use the logs that are archived for viewing and analysis, then
you can recall the logs. The recalled data will count towards your active storage usage
until you release it.
You can recall and release your selected set of logs multiple times. However,
the recall feature is enabled only if you already have archived logs.
Open the navigation
menu and click Observability & Management. Under
Logging Analytics, click
Administration. The Administration
Overview page opens.
The administration resources are listed in the left hand navigation
pane under Resources. Click
Storage.
The Storage page is displayed.
In the Storage page, on the left panel under
Resources, click Archiving Recall Requests.
The Archiving Recall Requests page displays the previously initiated recall
requests.
Specify the Purpose of recall. This can help you to identify your recall
request.
Optionally, if you have defined log set, then you can specify one or more Log
Sets to filter the recalled data. To specify multiple log sets, use
comma separation.
Select the time range of the logs that you want to recall, by
specifying the User-defined start time and
User-defined end time.
Click Estimate Recall Log Size. The Data
set recommended for analysis section opens. The size of the logs that
you've selected for recall is displayed adjacent to the heading Maximum
recalled data size before filtering.
Note that the start time and end time are extended to align with the
log index structure based on buckets. So, when you view the list of active
recalls or visit the activity tab, you may get the start and end
time extended beyond your chosen time range.
If your current recall time specifications overlap with another recall activity,
then they can possibly get merged into a single recall activity and the
resulting start and end time can get extended.
An alternative time range is recommended based on the availability of data. To
select the time range you specified earlier instead of the recommended time
range, enable the check box Do not use recommended data set for
recall
Specify the Query to filter the data set. Exclude the time and log set
from the query.
Specifying the filters reduce the actual recalled data size.
However, the filter does not impact this estimate.
Note
Only search filters or regex are supported in the query. Refrain
from adding pipes, aggregates, or statistical functions to the query.
You can use any log fields to filter the data with the query, except for time and
log set.
Some of the examples of invalid searches:
Entity = βtest1β | search βxyzβ
Instead, you can use Entity = 'test1' and 'xyz' which is
valid. Similarly, more valid examples include 'Entity =
'test1' and 'Log Source' = 'AVDF Alert Linux
Syslog' and 'xyz'.
Entity = βtest1β | stats count
The above query does not have a workaround because it has a statistical
function and pipe. However, using only the entity filter is valid
Entity = βtest1β.
Click Create Recall Request to proceed with
the recall of the selected logs.
The recall activity is listed in the Archiving Recall Requests page.
The table specifies the status, time range, data size, and request date and time
of recall activity, user who initiated the recall, and the purpose of recall.
The individual recalls that have overlapping data are combined into a single
collection. In such cases, the table displays the data size of the collection
instead of the data size of the underlying recalls.
Note
If you keep the recommended and default data set for each
recall, then the collection time range is the super set of the time ranges
of the individual recalls. Otherwise, the collection time range may not be
the super set of the individual recall time ranges.
Watch the status of the recall activity. You can use the recalled
logs for viewing and analysis after the recall activity is complete.
If the data size icon for a collection is displayed in orange, then new additional
log data is available for recall. Click the data icon and click Recall new data to initiate the recall of the new
data. The Recall new data dialog box opens. The query to filter the data set and
the time range for data recall are predefined. Specify the purpose of recall and
click Create Recall Request.
After active use of the recalled logs, if you want to release them
back to the archive pool, click the actions menu icon in the row corresponding to your recalled logs, and select
Release.
The recalled logs will then be released back into the archive pool.
This will enable you to optimize your storage size and cost.
Note
When releasing the recalled logs using REST API, note the recall time range
from console or CLI, and format the time as follows:
Recall start time: Round down (floor) the value. If the recall
start time is From Mon, Mar 7, 2022, 05:45:33 UTC, then
round down the time and specify it as
from_time=2022-03-07T5:45:32.000Z.
Recall end time: Round up (ceil) the value. If the recall end
time is To Wed, Mar 15, 2023, 17:26:53 UTC, then round
up the time and specify it as
to_time=2023-03-15T17:26:54.000Z.
Purge Log Data π
Oracle Logging Analytics lets you
purge log events that were loaded by agent or by an on-demand upload, to reduce
the index size of the log data.
Purging enables you to bring down your usage to reduce overage charges. Oracle Logging Analytics can purge log data automatically per a set
schedule or manually based on your need. Before you purge log data,
create IAM policies to set up permissions for the task. See Allow Users to Purge Log Data.
There are multiple ways to purge log data.
By purging on-demand: All log data
from the specified compartment created prior to
the selected time range gets purged.
By creating a purge policy: The old
log data can be purged by specifying a schedule
for purging and the query to filter the data to
purge. If you want to automate the purge activity,
then you can create a purge policy by specifying
the purge schedule, selecting the log data to
purge, and enabling the policy.
When
you use CLI or REST API to create a purge policy,
the value of the parameter task-type must
be set to PURGE.
Open the navigation
menu and click Observability & Management. Under
Logging Analytics, click
Administration. The Administration
Overview page opens.
The administration resources are
listed in the left hand navigation pane under
Resources. Click
Storage.
The Storage page is displayed.
In the Storage page, you can purge log data in
one of the following methods:
Perform on-demand Purge:
Click Purge
Logs. The Purge Logs dialog box
is displayed.
Select the compartment in which
the logs must be purged.
Specify if the subcompartments also
must be included in the Purge.
Select the date and time prior to
which the log data that must be purged was
collected.
Purge action is performed on the
log data from all the buckets in the selected
compartment which were collected prior to the
specified time period. For example, if you specify
the date and time as November 2, 2020
12:00:00 and compartment
Analyze, then the log data with
the time stamp older than November 2, 2020
12:00:00 stored in the compartment
Analyze is deleted.
In the Query
field, enter the query to select a specific set of
log data. For example, to select the logs from the
entities of the type Linux Host, specify
the query 'Entity Type'='Host
(Linux)'.
Click Estimate
Reclaimed Storage to determine the
size of the storage that can be reclaimed based on
the selection you made in the previous fields.
Click
Purge.
Create a purge policy to purge
logs based on a query or age:
Under Purge Policies, click
Create. The Create Purge
Policy dialog box opens.
Enter a name for the new purge
policy.
Select the log group compartment
to query for the logs. You can optionally specify
if the subcompartments must also be queried for
the specified logs.
Under Purge Logs Older
than, select the time period from when
the log data must be purged.
Under Schedule
Interval, select the periodicity, and
time of the purge action.
In the Query
field, enter the query to select a specific set of
log data. For example, to select the logs from the
source Apache HTTP Server Access
Logs, specify the query 'Log
Source'='Apache HTTP Server Access
Logs'.
Click Estimate
Reclaimed Storage to determine the
size of the storage that would be reclaimed if the
selections you made in the previous fields were to
be applied now.
Optionally, click Show Advanced
Options and add tags to your purge policy.
Click
Create.
The purge policy is created
and will be run periodically as set in the
previous steps.
Note
If a purge policy is paused due to lack of
permissions, after modifying the policy statements
as required, manually Resume the purge
task.
To delete a policy, click
Actions icon next to the policy name, and click
Delete.
To view the purge activities
performed, in the Storage page, under
Resources, click Activity Report. The
Activity Report page is displayed which summarizes
all the storage activities. Use the Status
and Time filters to view the preferred
purge activities.
Allow Users to Purge Log
Data π
To purge log data, first set up right permissions by creating the following
IAM policies:
Create a dynamic group to allow purges for the compartments you want
to allow purges in:
ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment ocid>'}
Alternatively, to allow purges on all compartments:
ALL {resource.type='loganalyticsscheduledtask'}
Create policies to allow the dynamic group to perform purge
operation:
allow dynamic-group <group_name> to read compartments in tenancy
allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy
allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERY_VIEW} in tenancy
allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy
allow dynamic-group <group_name> to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy
allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy
Note
For the proper functioning of the purge policy, the
permissions read compartments,
LOG_ANALYTICS_STORAGE_PURGE, and
LOG_ANALYTICS_QUERY_VIEW must be created at
tenancy level. To restrict the purge action permission to specific
compartments, the permissions
LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE,
LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS, and
LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ can be
set at the required compartment level.
In the above policy statements involving dynamic group, if the
dynamic group is in a domain other than Default, then the
policy statement must be of the format:
allow dynamic-group '<domain>'/'<group_name>' to ...
Enclose the domain name and dynamic group name in single quotes.
Additionally, ensure that the user has MANAGE permission on
loganalytics-features-family and
loganalytics-resources-family. If the user creating the on-demand or
scheduled purge has Administrator privileges, then the required
permissions are already available:
allow group <group_name> to MANAGE loganalytics-features-family in tenancy
allow group <group_name> to MANAGE loganalytics-resources-family in tenancy
Some of the above policy statements are included in the readily
available Oracle-defined policy templates. You may want to consider using the template
for your use case. See Oracle-defined Policy Templates for Common Use Cases.
Provide simple filter query to identify the log data that must be purged. In
case of wild card characters in the query such as *, ?,
and %, refrain from using them in purge policy. Oracle recommends using
Extended Field Definitions for future data in purge tasks.
For guidelines on creating queries for filtering log data, see Query Search.
Delete All Data older than 30 Days every Sunday at
midnight:
Purge Logs Older Than: 30 Days
Schedule Interval: Every Week, Day:
Sunday, Time: 00:00, Timezone:
Asia/Calcutta
Query: *
Delete logs from source OCI Audit Logs older than 2 months:
Purge Logs Older Than: 2 Months
Query: 'Log Source' = 'OCI Audit Logs'
Purge log for a log source and specific entities associated with that source
older than 1 year:
Purge Logs Older Than: 1 Year
Query: 'Log Source' = 'OCI VCN Flow Unified Schema
Logs' and Entity in ('Entity1', 'Entity2')
View Storage Activity
Report π
You can view the summary of your archive, recall, release, and purge
activities to maintain close control of your storage use and also to track the
status of your key logs that have been part of the activities.
Open the navigation
menu and click Observability & Management. Under
Logging Analytics, click
Administration. The Administration
Overview page opens.
The administration resources are listed in the left
hand navigation pane under Resources.
Click Storage.
The Storage page is displayed.
In the left panel under
Resources, click the
Activity Report.
The page displays the summary of the storage
activities initiated such as purge policy, purge on demand,
archiving, archiving recall request and recall release.
Use the Activity Type, Status, and
Time filters on the left panel to narrow down
your search for the storage activities.
Expand the storage activity row to view more details about
it.