Advanced Configuration Options for Management Agent
Presented here are some use cases where you can make changes to Management Agent configuration to customize the log collection.
Manually Specify Time Zone and Character Encoding for Files
You can manually specify the properties for the log files by editing the configuration properties in the agent installation folder.
You can perform this task for all the logs except the Windows events.
-
If you want to apply the properties on specific patterns and / or log sources, then make a note of the pattern ID and the source ID. Open the config xml file under the agent installation folder agent_inst/state/laStorage/os_file/os_file.xml.
- Pattern ID example:
<Pattern id="495071102827757094" name="/tmp/w*.mgr" include="true">
- Source ID example:
<LogSource id="-2574377491167724513" name="SS Concurrent Manager Logs" sourceType="os_file"/>
The agent installation folder location varies depending on the source type. For example, the location is agent_inst/state/laStorage/database_sql/database_sql.xml for SQL Database.
- Pattern ID example:
-
Edit the properties file emd.properties in the location agent_inst/config/emd.properties and add the following property to override the default configuration of Oracle Logging Analytics:
loganalytics.src.override_config=true
-
Specify the time zone
tz
and character encodingenc
properties in the file emd.properties by selecting from one of the following examples:-
Apply the properties for all the sources and patterns:
loganalytics.src.addl_src_ptn_configs=tz=UTC,enc=EUC-JP
-
Apply the properties only for specific log sources:
loganalytics.src.addl_src_ptn_configs=srcid=-2574377491167724513,tz=UTC,enc=EUC-JP;srcid=-2574377491167724512,enc=UTF-8
In this example, the time zone
UTC
and character encodingEUC-JP
properties are applied for the source-2574377491167724513
, and character encoding propertyUTF-8
is applied for the source-2574377491167724512
. -
Apply the properties only for specific patterns:
loganalytics.src.addl_src_ptn_configs=ptnid=495071102827757094,tz=UTC,enc=EUC-JP;ptnid=495071102827757095,enc=UTF-8
In this example, the time zone
UTC
and character encodingEUC-JP
properties are applied for the pattern495071102827757094
, and character encoding propertyUTF-8
is applied for the pattern495071102827757095
. -
Apply the properties only for a combination of specific patterns and sources:
loganalytics.src.addl_src_ptn_configs=srcid=-2574377491167724513,ptnid=495071102827757094,tz=UTC,enc=EUC-JP;srcid=-2574377491167724513,ptnid=495071102827757095,enc=UTF-8
In this example, the time zone
UTC
and character encodingEUC-JP
properties are applied for logs with the pattern495071102827757094
and the source-2574377491167724513
, and character encoding propertyUTF-8
is applied for logs with the pattern495071102827757095
and the source-2574377491167724513
.
-
Configure Database SQL Collection Schedule
You can customize the SQL collection schedule for each of your database log sources by specifying the periodicity in a CSV file in the format specified below.
-
Navigate to the Management Agent or Oracle Cloud Agent install directory on your database host, for example
agent_inst
. Create a directorylaconfig
in the agent install directory. -
In
laconfig
directory, create a CSV filelogan_schedule_database_sql_*.csv
. This is the schedule configuration file. -
In the schedule configuration file, provide the details of the source schedule in the following format:
NAME,RECURRENCE_PATTERN_UNIT,RECURRENCE_PATTERN_VALUE,START_HOUR <Source-Name>,DAY,1,1
The header row must have the prescribed content:
NAME,RECURRENCE_PATTERN_UNIT,RECURRENCE_PATTERN_VALUE,START_HOUR
In the subsequent rows, specify the schedule for each source, one per row.
The header row parameters and their values:
Header Row Parameter Description Allowed Values NAME
The name of the database log source defined in Oracle Logging Analytics.
Any
RECURRENCE_PATTERN_UNIT
This is the unit of cadence of SQL collection, for example, minutes or days.
WEEK and MONTH can have multiple values separated by
:
- MIN: minutes
- HOUR: hours
- DAY: days
- WEEK: weeks
- MONTH: months
RECURRENCE_PATTERN_VALUE
The value of the pattern unit you selected.
- MIN: interval 5 min, 15 min
- HOUR: interval 1 hour
- DAY: daily running at 1, or 12
- WEEK: SUN, MON, TUE, WED, THU, FRI, SAT
- MONTH: date 1-31, LAST day of the month
START_HOUR
The start hour for SQL collection, for example, 0100 hrs or 1200 hrs.
If this value is not specified, then the SQL collection begins immediately after source-entity association. Note that, even if the value is not specified, the preceding comma must be present in the row for the schedule to be valid.
1 or 12.
-
After saving the CSV file, restart the agent.
Some of the example values for the schedule configuration:
Source Row Content | Description |
---|---|
|
Schedule the SQL query every 5 min starting from source-entity association time. |
|
Schedule the SQL query every hour starting from source-entity association time. |
|
Schedule the SQL query every day at 0100 hrs system time. |
|
Schedule the SQL query every day at 1200 hrs system time. |
|
Schedule the SQL query every week on Friday at 0100 Hrs system time. |
|
Schedule the SQL query every week on both Wednesday and Thursday at 0100 Hrs system time. |
|
Schedule the SQL query every month on Date 7 at 0100 Hrs system time. |
|
Associate the Source1 first with the entity and wait for log collection to start and then update the configuration by adding the association of Source2 with the same entity |
Enable Log Collection from Large Folders
Folders containing more than 10,000 files may cause high resource (memory / storage / CPU) usage by the Management Agent which may lead to slow log collection, affect other functionalities of the Management Agent, and may also slow down the host machine.
When large folders are encountered by the Management Agent Logging Analytics plug-in, a
message similar to the following example message is added to the Management Agent
mgmt_agent_logan.log
file:
2020-07-30 14:46:51,653 [LOG.Executor.2388 (LA_TASK_os_file)-61850] INFO - ignore large dir /u01/service/database/logs. set property loganalytics.enable_large_dir to enable.
We recommend avoiding large folders. Utilize a cleanup mechanism to remove files soon after they are collected so that the Management Agent would have sufficient time to collect them again.
However, if you want to continue monitoring logs in large folders, then you can enable the support by making the following changes.
You can manually edit the configuration property
loganalytics.enable_large_dir
to true
in the agent
installation folder to enable the collection of logs from folders having more than 10000
files. By default, the property value is false
.
Edit the properties file emd.properties in the location agent_inst/config/emd.properties and add the above property value to override the default configuration of Oracle Logging Analytics:
sudo -u mgmt_agent echo "loganalytics.enable_large_dir=true" >> INSTALL_DIRECTORY/agent_inst/config/emd.properties
Replace INSTALL_DIRECTORY
with the path to the
agent_inst
folder. Restart the management agent after editing the
file emd.properties.
You may have to make some configuration changes on your host agent to enable this support. Try the new settings in a development or test environment before making them in production. Determine the increase for the following factors by using a representative environment to test them. The actual required increase will depend on factors such as the number of files, rate of file creation, and the other types of collection that the Management Agent is doing.
- Increase the heap size of the Management Agent. For directories with a large number of files, the required heap size increases with the number of files. See Management Agent documentation.
- Ensure that sufficient disk space and inodes are available for handling the large number of state files that the Management Agent may have to keep. This depends on the type of log source and parser used. If your parser uses the Header-Detail function, then the agent creates and stores the header in a cache file as long as the original log file exists.
- Ensure that the operating system setting for the number of open files can support the Management Agent reading the large folder and potentially large number of state files.
Use Macro for Case-Insensitive Match of Log File Path
Use the ignore-case macro (?i)
in the log file path pattern
to interpret it in the case-insensitive form. This is applicable for log file path pattern
provided in log sources of the type File .
The macro (?i)
will be applied only when it is in the start of the
element of the path.
Consider the example:
/tmp/(?i)FOO/*.log
In the above example, the element of the path which follows the macro is
FOO
. This is interpreted in the case-insensitive form.
If the value of a property contains the slash character
/
, then care must be taken to ensure the proper use of
ignore-case macro. For example, if the entity property is
logdir=/tmp/logs/
, the log source file path pattern is
(?i){logdir}/foo.log
, and the ignore-case macro is
applied on it, then the log source file path pattern is interpreted as
(?i)/tmp/logs/foo.log
which is not the desired
result.
In such cases, add multiple properties that are case-insensitive and build the log pattern path from them using the ignore-case macro.