Using the Data Filters tab when editing or creating a source, you can mask IP addresses, user ID, host name, and other sensitive information with replacement strings, drop specific keywords and values from a log entry, and also hide an entire log entry.

You can add data filters when creating a log source, or when editing an existing source. See Customize an Oracle-Defined Source to learn about editing existing log sources.

If the log data is sent to Oracle Logging Analytics using On-demand Upload or collection from object store, then the masking will happen on the cloud side before the data is indexed. If you are collecting logs using the Management Agent, then the logs are masked before the content leaves your premises.

Topics:

• Masking Log Data

• Hash Masking the Log Data

• Dropping Specific Keywords or Values from Your Log Records

• Dropping an Entire Log Entry Based on Specific Keywords

Masking Log Data

Masking is the process of taking a set of existing text and replacing it with other static text to hide the original content.

If you want to mask any information such as the user name and the host name from the log entries:

1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

2. The administration resources are listed in the left hand navigation pane under Resources. Click Sources.

3. Click the name of the source that you want to edit. The source details page opens. Click Edit to edit the source.

4. Click the Data Filters tab and click Add.

5. Enter the mask Name, select Mask as the Type, enter the Find Expression value, and its associated Replace Expression value.

   Find Expression value can be plain text search or standard regular expression. The value that will be replaced with the Replace Expression should be surrounded by quotes ( ).

   Name	Find Expression	Replace Expression
   mask username	User=(\S+)s+	confidential
   mask host	Host=(\S+)s+	mask_host

   Note
   
   

   The syntax of the replace string should match the syntax of the string that’s being replaced. For example, a number shouldn’t be replaced with a string. An IP address of the form 123.45.67.89 should be replaced with 000.000.000.000 and not with 000.000. If the syntaxes don’t match, then the parsers may break.

6. Click Save.

When you view the masked log entries for this log source, you’ll find that Oracle Logging Analytics has masked the values of the fields that you’ve specified.

• User = confidential

• Host = mask_host

Hash Masking the Log Data

When you mask the log data using the mask as described in the previous section, the masked information is replaced by a static string provided in the Replace Expression. For example, when the user name is masked with the string confidential, then the user name is always replaced with the expression confidential in the log records for every occurrence. By using hash mask, you can hash the found value with a unique hash. For example, if the log records contain multiple user names, then each user name is hashed to a unique value. So, if the string user1 is replaced with the text hash ebdkromluceaqie for every occurrence, then the hash can still be used to identify that these log entries are for the same user. However, the actual user name will not be visible.

Risk Associated: Because this is a hash, there is no way to recover the actual value of the masked original text. However, taking a hash of any string, you arrive at the same hash every time. Ensure that you consider this risk while hash masking the log data. For example, the string oracle has the md5 hash of a189c633d9995e11bf8607170ec9a4b8. Every time someone tries to create an md5 hash of the string oracle, it will always be the same value. Although you cannot take this md5 hash and reverse it back to get the original string oracle, if someone tries to guess and forward hash the value oracle, they will see that the hash matches the one in the log entry.

To apply the hash mask data filter on your log data:

1. Go to Create Source page. For steps, see Create a Source.

2. You can also edit a source that already exists. For steps to open an Edit Source page, see Edit Source.

3. Click the Data Filters tab and click Add.

4. Enter the mask Name, select Hash Mask as the Type, enter the Find Expression value, and its associated Replace Expression value.

   Name	Find Expression	Replace Expression
   Mask User Name	User=(\S+)s+	Text Hash
   Mask Port	Port=(\d+)s+	Numeric Hash

5. Click Save.

If you want to use hash mask on a field that is string based, you can use Text or Numeric hash as a string field. But if your data field is numeric, such as an integer, long, or floating point, then you must use Numeric hash. If you do not use numeric hash, then the replace text will cause your regular expressions which depend on this value to be a number, to break. The value will also not be stored.

This replacement happens before the data is parsed. Typically, when the data must be masked, it's not clear if it is always numeric. Therefore, you must decide the type of hash while creating the mask definition.

As the result of the above example hash masking, each user name is replaced by a unique text hash, and each port number is replaced by a unique numeric hash.

You can utilize the hash mask when filtering or analyzing your log data. See Filter Logs by Hash Mask.

Dropping Specific Keywords or Values from Your Log Records

Oracle Logging Analytics lets you search for a specific keyword or value in log records and drop the matched keyword or value if that keyword exists in the log records.

Consider the following log record:

ns5xt_119131: NetScreen device_id=ns5xt_119131  [Root]system-notification-00257(traffic): start_time="2017-02-07 05:00:03" duration=4 policy_id=2 service=smtp proto=6 src zone=Untrust dst zone=mail_servers action=Permit sent=756 rcvd=756 src=192.0.2.1 dst=203.0.113.1 src_port=44796 dst_port=25 src-xlated ip=192.0.2.1 port=44796 dst-xlated ip=203.0.113.1 port=25 session_id=18738

If you want to hide the keyword device_id and its value from the log record:

1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

2. The administration resources are listed in the left hand navigation pane under Resources. Click Sources.

3. Click the name of the source that you want to edit. The source details page opens. Click Edit to edit the source.

4. Click the Data Filters tab and click Add.

5. Enter the filter Name, select Drop String as the Type, and enter the Find Expression value such as device_id=\S*

6. Click Save.

When you view the log records for this source, you’ll find that Oracle Logging Analytics has dropped the keywords or values that you’ve specified.

Dropping an Entire Log Entry Based on Specific Keywords

Oracle Logging Analytics lets you search for a specific keyword or value in log records and drop an entire log entry in a log record if that keyword exists.

Consider the following log record:

ns5xt_119131: NetScreen device_id=ns5xt_119131  [Root]system-notification-00257(traffic): start_time="2017-02-07 05:00:03" duration=4 policy_id=2 service=smtp proto=6 src zone=Untrust dst zone=mail_servers action=Permit sent=756 rcvd=756 src=198.51.100.1 dst=203.0.113.254 src_port=44796 dst_port=25 src-xlated ip=198.51.100.1 port=44796 dst-xlated ip=203.0.113.254 port=25 session_id=18738

Let’s say that you want to drop entire log entry if the keyword device_id exists in them:

1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

2. The administration resources are listed in the left hand navigation pane under Resources. Click Sources.

3. Click the name of the source that you want to edit. The source details page opens. Click Edit to edit the source.

4. Click the Data Filters tab and click Add.

5. Enter the filter Name, select Drop Log Entry as the Type, and enter the Find Expression value such as .*device_id=.*

   It is important that the regular expression must match the entire log entry. Using .* in front of and at the end of the regular expression ensures that it match all other text in the log entry.

6. Click Save.

When you view the log entries for this log source, you’ll find that Oracle Logging Analytics has dropped all those log entries that contain the string device_id in them.

If you use Automatic parse time only option in your source definition instead of creating a parser, then the only field that will be available for creating Extended Field Definitions will be the Original Log Content field since no other fields will be populated by the parser. See Use the Automatic Time Parser.

Oracle Logging Analytics enables you to search for the extended fields that you’re looking for. You can search based on the how it was created, the type of base field, or with some example content of the field. Enter the example content in the Search field, or click the down arrow for the search dialog box. In the search dialog box, under Creation Type, select if the extended fields that you’re looking for are Oracle-defined or user-defined. Under Base Field, you can select from the options available. You can also specify the example content or the extraction field expression that can be used for the search. Click Apply Filters.

Topics:

• Use Ingest-Time Lookups in the Source

• Use the Geolocation Field for Grouping Logs

• Add More Data to Your Log Entries at Processing Time

• Use Substitution Function to Replace an Expression in a Field

When a log entry matches the condition that you have defined, a label is populated with that log entry. That label is available in your log explorer visualizations as well as for searching and filtering log entries.

You can use Oracle-defined or user created labels in the sources. To create a custom label to tag a specific log entry, see Create a Label.

1. To use labels in an existing source, edit that source. For steps to open an Edit Source page, see Edit Source.

2. Click the Labels tab.

3. To add a conditional label, click Add conditional label.

   In the Conditions section:

   1. Select the log field on which you want to apply the condition from the Input Field list.

   2. Select the operator from the Operator list.

   3. In the Condition Value field, specify the value of the condition to be matched for applying the label.

      Note
      
      

      To add the Log Group as the input field, provide its OCID for the value instead of the name.

   4. To add more conditions, click Add Condition icon , and repeat the steps 3a through 3c. Select the logical operation to apply on the multiple conditions. Select from AND, OR, NOT AND, or NOT OR.

      To add a group of conditions, click the Group Condition icon , and repeat the steps 3a through 3c to add each condition. A group of conditions must have more than one condition. Select the logical operation to apply on the group of conditions. Select from AND, OR, NOT AND, or NOT OR.

      To remove a condition, click Remove Condition icon .

      To view the list of conditions in the form of statement, click Show Condition Summary.

4. Under Actions, select from the already available Oracle-defined or user created labels. If required, you can create a new label by clicking Create Label.

   Select the Enabled check box.

5. Click Add.

Oracle Logging Analytics enables you to search for the labels that you’re looking for in the Log Explorer. You can search based on any of the parameters defined for the labels. Enter the search string in the Search field. You can specify the search criteria in the search dialog box. Under Creation Type, select if the labels that you’re looking for are Oracle-defined or user-defined. Under the fields Input Field, Operator, and Output Field, you can select from the options available. You can also specify the condition value or the output value that can be used for the search. Click Apply Filters.

You can now search log data based on the labels that you’ve created. See Filter Logs by Labels.

Use the Conditional Fields to Enrich the Data Set

Optionally, if you want to select any arbitrary field and write a value to it, you can use the conditional fields. Populating a value in an arbitrary field using the conditional fields functionality is very similar to using Lookups. However, using the conditional fields provides more flexibility in your matching conditions and is ideal to use when dealing with a small number of conditions - field population definitions. For example, if you have a few conditions to populate a field, then you can avoid creating and managing a lookup by using conditional fields.

The steps to add the conditional fields are similar to those in the workflow above for adding conditional labels.

• In the step 3, instead of clicking Add conditional label, click Add conditional field. The rest of the step 3 to select the conditions remains the same as the above workflow.

• In step 4 above,

  1. For the Output Field, select from the already available Oracle-defined or user created fields from the menu. If required, you can create a new field by clicking Create New Field.

  2. Enter an Output Value to write for the output field when the input condition is true.

     For example, the source can be configured to attach the authentication.login output value for the Security Category output field when the log record contains the input field Method set to the value CONNECT .

     Select the Enabled check box.

This is particularly helpful when you’re not sure about how to parse your logs or how to write regular expressions to parse your logs, and you just want to pass the raw log data to perform analysis. Typically, a parser defines how the fields are extracted from a log entry for a given type of log file. However, the generic parser in Oracle Logging Analytics can:

• Detect the time stamp and the time zone from log entries.

• Create a time stamp using the current time if the log entries don’t have any time stamp.

• Detect whether the log entries are multiple lined or single lined.

• Time stamp:

  • When a log entry doesn’t have a time stamp, then the generic parser creates and displays the time stamp based on the time when the log data was collected.

  • When a log record contains a time stamp, but the time zone isn’t defined, then the generic parser uses the management agent’s time zone.

    When using Management Agent, if the timezone is not detected properly, then you can manually set the timezone in the agent configuration files. See Manually Specify Time Zone and Character Encoding for Files.

    When uploading logs using on-demand upload, you can specify the timezone along with your upload to force the timezone if we cannot properly detect it. If you're using CLI, see Command Line Reference: Logging Analytics - Upload. If you're using REST API, then see Logging Analytics API - Upload.

  • When a log file has log records with multiple time zones, the generic parser can support up to 11 time zones.

  • When a log file displays some log entries with a time zone and some without, then the generic parser uses the previously found timezone for the ones missing a timezone.

  • When you ingest logs using management agent, if the time zone or the time zone offset is not indicated in the log records, then Oracle Logging Analytics compares the last modified time of the OS with the timestamp of the last log entry to determine the proper time zone.

• Multiple lines: When a log entry spans multiple lines, the generic parser can captures the multiline content correctly.