The Create Parser page opens in the Guided mode, by default. Continue in this mode if you want Logging Analytics to generate the regular expression to parse the logs after you select the fields. For Guided mode, see Video: Using RegEx Parser Builder. If you want to write regular expression for parsing, then switch to Advanced mode and continue with the following steps:

1. In the Name field, enter the parser name. For example, enter Custom Audit Logs.

2. Optionally, provide a suitable description to the parser for easy identification.

3. In the Example Log Content field, paste the contents from a log file that you want to parse, such as the following:

   Jun 20 2020 15:19:29 hostabc rpc.gssd: ERROR: can't open clnt5aa9: No such file or directory
   Jul 29 2020 11:26:28 hostabc kernel: FS-Cache: Loaded
   Jul 29 2020 11:26:28 hostxyz kernel: FS-Cache: Netfs 'nfs' registered for caching
   Aug 8 2020 03:20:01 slc08uvu rsyslogd-2068: could not load module '/lib64/rsyslog/lmnsd_gtls.so', rsyslog error -2078
   [try http://www.rsyslog.com/e/2068 ]
   Aug 8 2020 03:20:36 0064:ff9b:0000:0000:0000:0000:1234:5678 rsyslogd-2066: could not load module '/lib64/rsyslog/lmnsd_gtls.so', dlopen: /lib64/rsyslog/lmnsd_gtls.so: cannot open shared object file: Too many open files
   [try http://www.rsyslog.com/e/2066 ]
   Sep 13 2020 03:36:06 hostnam sendmail: uAD3a6o5009542: from=root, size=263, class=0, nrcpts=1, msgid=<201611130336.uAD3a6o5009542@hostname.example.com>, relay=root@localhost
   Sep 13 2020 03:36:06 hostnam sendmail: uAD3a6o5009542: to=root, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30263, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (uAD3a6KW009543 Message accepted for delivery)
   Sep 20 2020 08:11:03 hostnam sendmail: STARTTLS=client, relay=userv0022.example.com, version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-GCM-SHA384, bits=256/256]]>

   This is a multi-line log entry because entry 4 and 5 span multiple lines.

4. In the Parse Regular Expression field, enter the regular expression to parse the fields. Each pair of parentheses indicate the regular expression clause to extract to a single field.

   The parse expression is unique to each log type and depends on the format of the actual log entries. In this example, enter:

   (\w+)\s+(\d+)\s+(\d+)\s+(\d+):(\d+):(\d+)\s+([^\s]+)\s+([^:]+):\s+(.*)

   Note
   
   

   • Don’t include any spaces before or after the content.

   • The regular expression must match the complete text of the log record.

   • If you’ve included hidden characters in your parse expression, then the Create Parser interface issues a warning message:

     Parser expression has some hidden control characters.

     To disable this default response, uncheck the Show hidden control characters check box when the error message appears.

   To learn more about creating parse expressions, see Sample Parse Expressions.

5. Select the appropriate Log Record Span.

   The log entry can be a single line or multiple lines. If you chose multiple lines, then enter the log record’s start expression.

   You must define the regular expression to be the minimum rule needed to successfully identify the start of a new log entry. In the example, the start expression can be:

   (\w+)\s+(\d+)\s+(\d+)\s+

   This indicates that the start of a new log entry occurs whenever a new line starts with the text that matches this entry start expression.

   Optionally, you can enter the end expression too. Use End Expression to indicate the end of the log record. When a log record is written to the file over a long period of time and you want to ensure that the entire log entry is collected, then you use the end regular expression. The agent waits till the end regex pattern is matched to collect the log record. The same format rules apply for the end expression as that of Entry Start Expression.

   Select the Handle entire file as a single log entry check box, if required. This option lets you parse and store an entire log file as a single log entry. This is particularly useful when parsing log sources such as Java Hotspot Dump logs, RPM list of packages logs, and so on. If the log entry has a lot of text that you want searchable, then you might want to consider selecting the check box Enable raw-text searching on this content. This option enables you to search the log records with the Raw Text field. When enabled, you can view the original log content with the raw text.

6. In the Fields tab, select the relevant field to store each captured value from the regular expression.

   To create a new field, click icon. The Create User Defined Field dialog box opens. Enter the values for Name, Data Type, Description and use the check box Multi-Valued Field as applicable for your logs. For information about the values to enter, see Create a Field. To remove the selected field in the row, click icon.

   For each captured value, select the field that you want to store the data into. The first captured field in the example can be entered as follows:

   • Field Name: Month (Short Name)

   • Field Data Type: STRING

   • Field Description: Month component of the log entry time as short name, such as Jan

   • Field Expression: (\w{3})

   In the above log example, you would map the fields in following order:

   1. Month (short name)
   2. Day
   3. Year (4 digits)
   4. Host Name (Server)
   5. Service
   6. Message

7. In the Functions tab, click Add Function to optionally add a function to pre-process log entries. See Preprocess Log Entries.

8. Click the Parser Test tab to view how your newly created parser extracts values from the log content.

   You can view the log entries from the example content that you provided earlier. Use this to determine if the parser that you defined works successfully.

   If the regular expression fails for one log entry, the match status shows which part of the expression worked in green and the part that failed in red. This helps you to identify which part of the regular expression needs to be fixed.

   You can view the list of events that failed the parser test and the details of the failure. You can also view the step count which can give an idea of how expensive the regular expression is. Ideally this value should be as low as possible. Anything under 1000 can be acceptable, but sometimes a more expensive regex is necessary. If the regex takes too many steps, it may result in higher CPU usage on the agent host and slower processing of the logs. This will delay the availability of the log entries in the Log Explorer. In extreme cases where the regular expression is too expensive, we will skip parsing the log entry and any other entries collected in the same collection period.

Click Save to save the new parser that you just created.

To abort creating the parser, click Cancel. When you click Cancel, you will lose any progress already made in the creation of Regex type parser.

To create a JSON parser:

1. In the Parser field, enter the parser name. For example, enter Custom API Log.

2. Select the parser type as JSON.

3. Optionally, provide a suitable description to the parser for easy identification.

4. In the Example Log Content field, paste the contents from a log file that you want to parse, such as the following:

   {
     "name": "helloapi",
     "hostname": "banana.local",
     "pid": 40442,
     "level": 30,
     "req": {
       "method": "GET",
   	"url": "/hello?name=paul",
   	"headers": {
   	  "user-agent": "curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1/2/3",
   	  "host": "0.0.0.0:8080",
   	  "accept": "*/*"
   	},
   	"remoteAddress": "127.0.0.1",
   	"remotePort": 59834
     },
     "msg": "start",
     "time": "2012-03-28T17:39:44.880Z",
     "v": 0
   }

   Based on the example log content, the fields are picked and displayed in the Fields tab, as in the following example:

   
   
   
   
   
5. In the Fields tab, for the specific JSON path, select the field name from the available defined fields.

   To create a new field, click icon. The Create User Defined Field dialog box opens. Enter the values for Name, Data Type, Description and use the check box Multi-Valued Field as applicable for your logs. For information about the values to enter, see Create a Field. To remove the selected field in the row, click icon.

   The default root path selected is $. If you want to change the JSON root path, expand the Advanced Options section, and select the Log Entry JSON Path from the menu.

6. In the Functions tab, click Add Function to optionally add a function to pre-process log entries with the header parser function. See Preprocess Log Entries.
7. After the fields are selected, go to Parser Test tab to view the match status, and the fields picked from the example log content.
8. Click Save to save the new parser that you just created.

To abort creating a JSON type parser and to switch to creating a parser of regex type, under Type, select Regex.

1. In the Parser field, enter the parser name. For example, enter Custom Database Audit Log.

2. Select the parser type as XML.

3. Optionally, provide a suitable description to the parser for easy identification.

4. In the Example Log Content field, paste the contents from a log file that you want to parse, such as the following ExampleA:

   <AuditRecord><Audit_Type>1</Audit_Type><Session_ld>201073</Session_ld><Statementid>13</Statementid><Entryld>6</Entryld>
   <Extended_Timestamp>2016-09-09T01:26:07.832814Z</Extended_Timestamp><DB_User>SYSTEM</DB_User><OS_User>UserA</OS_User><Userhost>host1</Userhost>
   <OS_Process>25839</OS_Process><Terminal>pts/4</Terminal><Instance_Number>0</Instance_Number><Object_Schema>HR</Object_Schema>
   <Object_Name>EMPLOYEES</Object_Name><Action>17</Action>
   <Returncode>0</Returncode><Sen>§703187</Sen><AuthPrivileges>--------¥-----</AuthPrivileges>
   <Grantee>SUMON</Grantee>
   <Priv_Used>244</Priv_Used><DBID>2791516383</DBID><Sql_Text>GRANT SELECT ON hr.employees TO sumon</Sql_Text></AuditRecord>

   Based on the example log content, Oracle Logging Analytics automatically identifies the list of XML elements that represent the log records.

5. Optionally, enable the check box Ignore namespace to ignore the namespace in the XML path.

6. Optionally, enable the check box Enable positional awareness to detect multiple values of an element under an XML path, like in the case of array of elements. For example, for the following example log content ExampleB:

   <ListOfTrackingFields>
       <TrackingField>
           <Name>NameA</Name>
       </TrackingField>
       <TrackingField>
           <Name>NameB</Name>
       </TrackingField>
       <TrackingField>
           <Name>NameC</Name>
       </TrackingField>
       <TrackingField>
           <Name>NameD</Name>
       </TrackingField>
       <TrackingField>
           <Name>NameE</Name>
       </TrackingField>
   </ListOfTrackingFields>

   If positional awareness is not enabled, you would be restricted to the XML path /ListOfTrackingFields/TrackingField/Name, which would detect only the first value of Name. If you enable positional awareness, then the following 5 values of Name under the element TrackingField are detected: NameA, NameB, NameC, NameD, NameE, if all the corresponding XML paths are configured.

7. From Log Entry XML Path menu, select the XML element suitable for the log records of interest. For the above example log content ExampleA, select /AuditRecord.

   Based on the selection of the XML path, the fields are picked and displayed in the Fields tab. In the above example log content, the highlighted elements qualify for the fields. For the example XML path /AuditRecord, the XML path of the field would be /AuditRecord/<highlighted_element>, for example /AuditRecord/Audit_Type.

8. In the Fields tab, for the specific XML path, select the field name from the available defined fields.

   To create a new field, click icon. The Create User Defined Field dialog box opens. Enter the values for Name, Data Type, Description and use the check box Multi-Valued Field as applicable for your logs. For information about the values to enter, see Create a Field. To remove the selected field in the row, click icon.

9. After the fields are selected, go to Parser Test tab to view the match status.

   
   
   
   
   

10. Click Save to save the new parser that you just created.

To abort creating a XML type parser and to switch to creating a parser of regex type, under Type, select Regex.

1. Enter the parser Name. For example, enter Comma Separated Custom App Log.

2. Optionally, provide a suitable description to the parser for easy identification.

3. Select the parser type as Delimited.

4. Under Delimiter, select the type of delimiter used in the logs to separate the fields, for example, comma, semicolon, space, or tab. In case it is different from the options listed, then select Other, and specify your delimiter character in the provided space.

   Consider the following example log content:

   helloapi,"host,local", 65427, 29, start, 2021-08-28T17:39:44.880Z

   In the above example, comma (,) is the delimiter.

   Ensure that the Delimiter character you specify is different from the Qualifier you select next.

5. Optionally, select the Qualifier which is used for wrapping the text for a field in your logs. Select Double Quote or Single Quote.

   Text qualifier is used when the delimiters are contained within the field data. Consider a scenario where field1 has the value host,local which contains a delimiter character (,) and a text qualifier is not used. Then the data that occurs after the delimiter (local) will be considered as the value of the next field.

   In the example log content from the previous step, double quote (") is the qualifier.

   Note
   
   For the values of the fields of the data type string, the space character before and after the qualifier will also be considered as part of the string. Ensure that the qualifier is positioned right next to the delimiter so that the string value matches. For example, in ,"host,local",, the space character after the first delimiter is removed. Also, the space character before the last delimiter is removed. Therefore, the effective value of the matching string field is host,local as can be seen in the parser test result later.

6. Optionally, under Header, paste the header line of your log which can then be mapped to the Header Column in the Fields tab. For example,

   name, hostname, pid, level, msg, time

   When you specify a Header, any log lines in the actual logs which exactly match the header are filtered out.

7. In the Example Log Content field, paste the contents from a log file that you want to parse, such as the following:

   helloapi,"host,local", 65427, 29, start, 2021-08-28T17:39:44.880Z

   Based on the example log content, the fields are picked and displayed in the Fields tab, as in the following example:

8. In the Fields tab, for the header column, select the field name from the available defined fields. However, you can selectively map for only your preferred header columns, and it's not necessary that you map field for every header column.
   
   
   
   

   To create a new field, click icon. The Create User Defined Field dialog box opens. Enter the values for Name, Data Type, Description and use the check box Multi-Valued Field as applicable for your logs. For information about the values to enter, see Create a Field. To remove the selected field in the row, click icon.

9. After the fields are selected, go to Parser Test tab to view the match status, and the fields picked from the example log content.
   
   
   
   
10. Click Create Parser to save the new parser that you just created.

To abort creating a Delimited type parser and to switch to creating a different type of parser, select it under Type.

If your original log content is wrapped with additional information, then you can map the base field which contains the original log content to a parser and extracts the fields in that original log record.

Let's consider that your original log content is as shown below:

####<Feb 01, 2021 10:28:27 AM PST> <Info> <Diagnostics> <soaapps.example.com> <excstg_soa> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <kernel_version> <123456> <[severity-value:64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320145> <Size based data retirement operation completed on archive HarvestedDataArchive. Retired 9,720 records in 11,242 ms.>

When the same log content is sent from OCI Logging through Connector Hub to Oracle Logging Analytics, then a JSON wrapper is added to the original log content. For example,

{
  "data": {
    "Labels": "LogPath:/u01/APPLTOP/instance/wlslogs/FADomain/servers/ServiceServer_1/logs/ServiceServer_1.log;ServiceName:Host;ServiceType:Oracle_Host;LogSet:TEST_POD;HostName:examplehost.oraclecloud.com;IP:192.0.2.1;WLS:NA;Domain:NA;AppName:myApp;Customer:CompanyA, (U.S.) Corp;CustomerType:NonHypercare;PODType:TEST;PODSize:S0;DC:CDC;Version:11.13.20.10.0;",
    "action": "ACTION",
    "msg": "####<Feb 01, 2021 10:28:27 AM PST> <Info> <Diagnostics> <soaapps.example.com> <excstg_soa> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <kernel_version> <123456> <[severity-value:64] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-320145> <Size based data retirement operation completed on archive HarvestedDataArchive. Retired 9,720 records in 11,242 ms.>"
  },
  "id": "123123123",
  "oracle": {
    "compartmentid": "ocid1.compartment.oc1..aaaaaaaauniqueID",
    "ingestedtime": "2021-02-01T18:32:49.204Z",
    "loggroupid": "ocid1.loggroup.oc1.iad.amaaaaaauniqueID",
    "logid": "ocid1.log.oc1.iad.amaaaaaayuniqueID",
    "tenantid": "ocid1.tenancy.oc1..aaaaaaaaetuniqueID"
  },
  "source": "FA-APPS",
  "specversion": "1.0",
  "time": "2021-02-01T11:48:43.091Z",
  "type": "FA Unified Apps Logs"
}

In the above example, the original log content is present in the base field msg.

Note that in the example discussed, the log content has a JSON wrapper. However, you can follow these steps even if your original log content and wrappers are in any supported parser format, and the logs are ingested in any supported method.

To extract the fields in the original log content, follow these steps:

1. Based on the type of original log content, identify or create a parser to pick up the relevant fields from the original log content. See Create a Parser.

   This parser is going to be used as a mapped parser within the wrapper parser that you will create next.

2. Based on the type of wrapper content, create the parser for it by following the steps in Create a Parser.

   This parser is going to be referred to as a wrapper parser in the steps that follow.

3. For the wrapper parser, provide example log content which has a wrapper over your original log content.

   In case of JSON parser, the default root path selected is $. If you want to change the JSON root path, expand the Advanced Options section, and select the Log Entry JSON Path from the menu.

4. In the Fields tab, corresponding to the base field that has your original log content, click Actions menu , and select Map fields through a parser. The dialog box where you can select the parser opens.

   In the above example, the row for which a mapped parser must be added is $.data.msg.

   In a wrapper parser, you can map only one base field to a mapped parser.

5. In the Map fields through a parser dialog box, select the parser using which the fields from the base field must be extracted. Click Map fields.

   The Fields tab now shows that the base field is mapped to the selected mapped parser and the list of fields extracted.

   It is possible that the mapped parser you select may also have a mapped parser included in it. Oracle Logging Analytics supports two levels of nested mapped parsers, for example, Wrapper-parser contains mapped-parser-1 contains mapped-parser-2.

6. After the remaining fields are selected, in the Parser Test tab, run the parser test to verify that the match status, and the fields picked from the example log content.

Oracle Logging Analytics provides the following functions which allows you to preprocess log entries and enrich the resultant log entries:

• Header Detail Function

• Find Replace Function

• Time Offset Function

To preprocess log entries while creating a parser, click the Functions tab and then click the Add Function button.

In the resultant Add Function dialog box, enter a name for the function, select the required function, and specify the relevant field values.

To test the result of applying the function on the example log content, click Test. A comparative result is displayed to help you determine the correctness of the field values.