Advanced Configuration Options for Management Agent

Presented here are some use cases where you can make changes to Management Agent configuration to customize the log collection.

Manually Specify Time Zone and Character Encoding for Files

You can manually specify the properties for the log files by editing the configuration properties in the agent installation folder.

You can perform this task for all the logs except the Windows events.

  1. If you want to apply the properties on specific patterns and / or log sources, then make a note of the pattern ID and the source ID. Open the config xml file under the agent installation folder agent_inst/state/laStorage/os_file/os_file.xml.

    • Pattern ID example: <Pattern id="495071102827757094" name="/tmp/w*.mgr" include="true">
    • Source ID example: <LogSource id="-2574377491167724513" name="SS Concurrent Manager Logs" sourceType="os_file"/>

    The agent installation folder location varies depending on the source type. For example, the location is agent_inst/state/laStorage/database_sql/database_sql.xml for SQL Database.

  2. Edit the properties file emd.properties in the location agent_inst/config/emd.properties and add the following property to override the default configuration of Oracle Logging Analytics:

    loganalytics.src.override_config=true
        
  3. Specify the time zone tz and character encoding enc properties in the file emd.properties by selecting from one of the following examples:

    • Apply the properties for all the sources and patterns:

      loganalytics.src.addl_src_ptn_configs=tz=UTC,enc=EUC-JP
    • Apply the properties only for specific log sources:

      loganalytics.src.addl_src_ptn_configs=srcid=-2574377491167724513,tz=UTC,enc=EUC-JP;srcid=-2574377491167724512,enc=UTF-8

      In this example, the time zone UTC and character encoding EUC-JP properties are applied for the source -2574377491167724513, and character encoding property UTF-8 is applied for the source -2574377491167724512.

    • Apply the properties only for specific patterns:

      loganalytics.src.addl_src_ptn_configs=ptnid=495071102827757094,tz=UTC,enc=EUC-JP;ptnid=495071102827757095,enc=UTF-8

      In this example, the time zone UTC and character encoding EUC-JP properties are applied for the pattern 495071102827757094, and character encoding property UTF-8 is applied for the pattern 495071102827757095.

    • Apply the properties only for a combination of specific patterns and sources:

      loganalytics.src.addl_src_ptn_configs=srcid=-2574377491167724513,ptnid=495071102827757094,tz=UTC,enc=EUC-JP;srcid=-2574377491167724513,ptnid=495071102827757095,enc=UTF-8

      In this example, the time zone UTC and character encoding EUC-JP properties are applied for logs with the pattern 495071102827757094 and the source -2574377491167724513, and character encoding property UTF-8 is applied for logs with the pattern 495071102827757095 and the source -2574377491167724513.

Configure Database SQL Collection Schedule

You can customize the SQL collection schedule for each of your database log sources by specifying the periodicity in a CSV file in the format specified below.

  1. Navigate to the Management Agent or Oracle Cloud Agent install directory on your database host, for example agent_inst. Create a directory laconfig in the agent install directory.

  2. In laconfig directory, create a CSV file logan_schedule_database_sql_*.csv. This is the schedule configuration file.

  3. In the schedule configuration file, provide the details of the source schedule in the following format:

    NAME,RECURRENCE_PATTERN_UNIT,RECURRENCE_PATTERN_VALUE,START_HOUR
    <Source-Name>,DAY,1,1

    The header row must have the prescribed content: NAME,RECURRENCE_PATTERN_UNIT,RECURRENCE_PATTERN_VALUE,START_HOUR

    In the subsequent rows, specify the schedule for each source, one per row.

    The header row parameters and their values:

    Header Row Parameter Description Allowed Values

    NAME

    The name of the database log source defined in Oracle Logging Analytics.

    Any

    RECURRENCE_PATTERN_UNIT

    This is the unit of cadence of SQL collection, for example, minutes or days.

    WEEK and MONTH can have multiple values separated by :

    • MIN: minutes
    • HOUR: hours
    • DAY: days
    • WEEK: weeks
    • MONTH: months

    RECURRENCE_PATTERN_VALUE

    The value of the pattern unit you selected.

    • MIN: interval 5 min, 15 min
    • HOUR: interval 1 hour
    • DAY: daily running at 1, or 12
    • WEEK: SUN, MON, TUE, WED, THU, FRI, SAT
    • MONTH: date 1-31, LAST day of the month

    START_HOUR

    The start hour for SQL collection, for example, 0100 hrs or 1200 hrs.

    If this value is not specified, then the SQL collection begins immediately after source-entity association. Note that, even if the value is not specified, the preceding comma must be present in the row for the schedule to be valid.

    1 or 12.

  4. After saving the CSV file, restart the agent.

Some of the example values for the schedule configuration:

Source Row Content Description

<Source-Name>,MIN,5,

Schedule the SQL query every 5 min starting from source-entity association time.

<Source-Name>,HOUR,1,

Schedule the SQL query every hour starting from source-entity association time.

<Source-Name>,DAY,1,1

Schedule the SQL query every day at 0100 hrs system time.

<Source-Name>,DAY,1,12

Schedule the SQL query every day at 1200 hrs system time.

<Source-Name>,WEEK,FRI,1

Schedule the SQL query every week on Friday at 0100 Hrs system time.

<Source-Name>,WEEK,WED:THU,1

Schedule the SQL query every week on both Wednesday and Thursday at 0100 Hrs system time.

<Source-Name>,MONTH,7,1

Schedule the SQL query every month on Date 7 at 0100 Hrs system time.

<Source1-Name>,MIN,5,

<Source2-Name>,DAY,1,1

Associate the Source1 first with the entity and wait for log collection to start and then update the configuration by adding the association of Source2 with the same entity

Enable Log Collection from Large Folders

Folders containing more than 10,000 files may cause high resource (memory / storage / CPU) usage by the Management Agent which may lead to slow log collection, affect other functionalities of the Management Agent, and may also slow down the host machine.

When large folders are encountered by the Management Agent Logging Analytics plug-in, a message similar to the following example message is added to the Management Agent mgmt_agent_logan.log file:

2020-07-30 14:46:51,653 [LOG.Executor.2388 (LA_TASK_os_file)-61850] INFO - ignore large dir /u01/service/database/logs. set property loganalytics.enable_large_dir to enable.
Note

We recommend avoiding large folders. Utilize a cleanup mechanism to remove files soon after they are collected so that the Management Agent would have sufficient time to collect them again.

However, if you want to continue monitoring logs in large folders, then you can enable the support by making the following changes.

You can manually edit the configuration property loganalytics.enable_large_dir to true in the agent installation folder to enable the collection of logs from folders having more than 10000 files. By default, the property value is false.

Edit the properties file emd.properties in the location agent_inst/config/emd.properties and add the above property value to override the default configuration of Oracle Logging Analytics:

sudo -u mgmt_agent echo "loganalytics.enable_large_dir=true" >> INSTALL_DIRECTORY/agent_inst/config/emd.properties

Replace INSTALL_DIRECTORY with the path to the agent_inst folder. Restart the management agent after editing the file emd.properties.

You may have to make some configuration changes on your host agent to enable this support. Try the new settings in a development or test environment before making them in production. Determine the increase for the following factors by using a representative environment to test them. The actual required increase will depend on factors such as the number of files, rate of file creation, and the other types of collection that the Management Agent is doing.

  • Increase the heap size of the Management Agent. For directories with a large number of files, the required heap size increases with the number of files. See Management Agent documentation.
  • Ensure that sufficient disk space and inodes are available for handling the large number of state files that the Management Agent may have to keep. This depends on the type of log source and parser used. If your parser uses the Header-Detail function, then the agent creates and stores the header in a cache file as long as the original log file exists.
  • Ensure that the operating system setting for the number of open files can support the Management Agent reading the large folder and potentially large number of state files.

Use Macro for Case-Insensitive Match of Log File Path

Use the ignore-case macro (?i) in the log file path pattern to interpret it in the case-insensitive form. This is applicable for log file path pattern provided in log sources of the type File .

The macro (?i) will be applied only when it is in the start of the element of the path.

Consider the example:

/tmp/(?i)FOO/*.log

In the above example, the element of the path which follows the macro is FOO. This is interpreted in the case-insensitive form.

Note

If the value of a property contains the slash character /, then care must be taken to ensure the proper use of ignore-case macro. For example, if the entity property is logdir=/tmp/logs/, the log source file path pattern is (?i){logdir}/foo.log, and the ignore-case macro is applied on it, then the log source file path pattern is interpreted as (?i)/tmp/logs/foo.log which is not the desired result.

In such cases, add multiple properties that are case-insensitive and build the log pattern path from them using the ignore-case macro.