Record APM Spans as JFR Events
APM Tracer and APM Agent support recording APM spans as JFR Events.
Currently JDK 11 or higher are supported.
For information about recording APM spans as JFR events when using APM Agent, see Record APM Spans as JFR Events under Provision and Deploy APM Java Agents on Application Servers.
Java Flight Recorder (JFR) is a tool for collecting diagnostic and profiling data about a running Java application. For information about Java Flight Recorder (JFR) and Java Mission Control (JMC), see JDK Mission Control.
Capture APM Events in JFR Recordings
To capture APM events in JFR recordings when using the APM Tracer, follow these steps:
-
Ensure APM Tracer is working properly.
Open Trace Explorer and confirm you are able to view all the traces. For information about opening and using Trace Explorer, see Monitor Traces in Trace Explorer.
-
Add a dependency to the Maven project which will let the APM Tracer add events into the JFR recordings made by the Helidon JVM.
<dependency> <groupId>com.oracle.apm.agent.java</groupId> <artifactId>apm-java-agent-jfr11</artifactId> <version>[1.0.1389,)</version> </dependency>
-
Initiate a recording.
To test that everything is working properly, you need to initiate a recording using your preferred method (It could be using JDK Mission Control UI or JCMD comand line utility). After a recording has been initiated, trace events are displayed on JDK Mission Control UI when the recording is viewed.
To verify that APM trace events are properly added to JFR recordings, connect to JDK Mission Control and confirm the APM traces are visible under Event Browser: Click on APM OpenTracing and confirm that APM trace events are visible.
Configure JFR Events in APM Tracer
(Optional) This is an optional step if you want to customize the reporting of the events.
To simplify the JFR recording events analysis, each event is organized into a category or an event type.
Default Category
In the APM Tracer, the default category for APM JFR events is obtained by using the contents of theSpan Operation Name
(in a
field called name
within the Span). The component
tag may
not be populated therefore it cannot be used as a default category.
Events
An event type allows the events to get organized and classified inside the JMC user interface. When the APM Tracer gets a span, it can generate an event for this span, but it needs to determine the event type of that specific event.
Examples of event type are servlet, jdbc and jax-rs.
Event Type Configuration
There are several ways to do the configuration of an event type or category. See below the different use cases:
-
Case 1:
component
tagThis is the simplest way to classify an event. This is the recommended method to use.
In most scenarios, the Opentracing span will contain a tag called
component
. The value ofcomponent
tag is used to determine the event type.For example, see the following span:
{ "id":"e926ff3d06013045", "trace-id":"49abd92e69786853e926ff3d06013045", "parent-id":"6ef54ec2c524c99a", "name":"content-write", "ts-micros":1614190751360177, "td-micros":219348, "attributes":{ "server":"Helidon SE", "http.url":"http://localhost:8079/health", "http.status_code":200, "component":"helidon-webserver", "http.method":"GET", "organization":"development" }, "events":[ ] },
In this case the event generated will get assigned a
helidon-webserver
event type since that is the value of thecomponent
tag. -
Case 2: Other tag
If your application does not use the
component
tag, but it uses another tag or there is a library which it's generating the tags for you then the APM Tracer can be configured to read the event type from that other tag.For example, see the following span where you can read the event type from the tag called
span.type
:{ "id":"e926ff3d06013045", "trace-id":"49abd92e69786853e926ff3d06013045", "parent-id":"6ef54ec2c524c99a", "name":"content-write", "ts-micros":1614190751360177, "td-micros":219348, "attributes":{ "server":"Helidon SE", "http.url":"http://localhost:8079/health", "http.status_code":200, "span.type":"helidon-webserver", "http.method":"GET", "organization":"development" }, "events":[ ] },
In this case, you can get the APM Tracer to read the event type from the
span.type
tag by using the following system property when starting the java application:java -Dcom.oracle.apm.agent.jfr.category.tag=span.type ...
-
Case 3: Use Operation Name when no tags are available
If your application does not use any tag that has useful categories then it is possible to use the
Operation Name
to determine the event type, but this method is not recommended.The JFR system requires that the name of the event type be a valid Java identifier, but the
Operation Name
often has spaces and also, other illegal characters. APM Tracer will convert theOperation Name
into a valid identifier by replacing all illegal characters with an underscore.java -Dcom.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES
Note
TheOperation Name
will only be used if the category tag is missing or empty.This is not a simple solution because the
Operation Name
may contain unique tokens like IDs which will make every event into it's own event type. In addition to that, the conversion of theOperation Name
into a valid identifier will cause further problems in configuring the event type to setup which fields are needed in the event, and also, the configuration will need to be done based on the convertedOperation Name
otherwise you would need to configure many different variants of theOperation Name
in the same way. -
Case 4: There are no tags available and all of the Operation Names are unique
In this case there is no information available to classify the event types. This is the worst case scenario where all of the events will end up as the same event type, or everyone ends up unique if they are configured using use case 3 scenario as seen above. The only thing possible to do is to allow you to choose the name of this default event type and this can be configured as shown below.
java -Dcom.oracle.apm.agent.jfr.category.default=Unknown
Note
The default will only be used if the category tag is missing or empty.In this case, all the events are placed in the event type called
Unknown
or any custom value selected.
For every event type, we can add 1 or more fields which contain data specific to that event.
- Name.
- Type (int, float, string, thread).
- Label.
- Description.
- name: traceID
type: String
label: New Trace ID
description: New Opentracing Trace ID
Create an ACML File
This is an optional step. If the default settings are working as you expected, you can skip this step.
You can create an ACML file if you want to change the default settings and customize the recordings.
The following example shows the format of an ACML file configuring event logging to JFR by overwriting the built-in default settings for the Helidon microservices container:
caseSensitiveFieldName: false
recordCallStack: true
eventTypes:
helidon-webserver:
name: helidon-webserver
label: Helidon Webserver Events
description: All events based on helidon webserver
recordCallStack: true
# All fields must be provided here.
# Fields are NOT incremental since that would make them impossible to remove
fields:
- name: traceID
type: String
label: New Trace ID
description: New Opentracing Trace ID
- name: spanID
type: String
label: New Span ID
description: New Opentracing Span ID
- name: http.url
type: String
label: New HTTP URL
description: The New URL for the HTTP endpoint being hit
- name: organization
type: String
label: Organization
description: The New Organization value of this call
- name: startThread
type: Thread
label: Start Thread
description: The thread on which this event started
- name: http.status_code
type: int
label: Status Code
description: The HTTP Response status code
- name: http.method
type: String
label: HTTP Method
description: The HTTP Method
You can use the above format of the ACML file when using APM Agent and APM Tracer. For APM Tracer, it will work for any container like Helidon, Springboot and Micronaut.
Frequently Asked Questions
-
When you have both settings provided:
com.oracle.apm.agent.jfr.category.tag
andcom.oracle.apm.agent.jfr.category.default
properties, Which setting does it take priority?If they are both given a value then we will first look in the span for the category tag. If it's found, that will be used. If it is missing, the default will be used.
-
For APM Tracer, Is a configuration file mandatory? Can I just specify the properties without file?
A configuration file is not mandatory. If the default works for you then you don't need a configuration file, and you can still specify the properties.
-
In some cases, Why does it look like entries are duplicated? For example, the following snippet from the ACML configuration file has the
helidon-webserver
entry twice:eventTypes: helidon-webserver: name: helidon-webserver
The first
helidon-webserver
entry is what we read/deduce from the tags or the operation name. If the event type is clean, easy to ready and intuitive, like the above example, then you may use the same event type to send to the JFR recording, and that how it will be get displayed in JMC.However, sometimes when using theoperation name
you may get an event type like the following:q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A=
which represents a JDBC query from a db client. In this case you would want to change the name sent to the JFR recording to something intuitive therefore the below example can be used in those cases:eventTypes: q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A=: name: jdbc-select-query
-
If
recordCallStack
can be specified for each event type, Why is there also a need to have top levelrecordCallStack
?We support two use cases for maximum flexibility in recording of the call stack. Call stack recording can be globally turned off and then it can selectively get turned on for specific event types. Another option is to globally turned it on and then it can selectively get turned off for specific event types. Both use cases are supported, therefore the setting are at both the top level, and at the event level.
-
I have some spans which were not configured at all and they are being ignored. Is it possible to still capture those events without configuring them?
Yes. If you set
allowUnconfiguredEvents: true
in the main top level configuration then those events will get captured. When this happens, we don't know which fields to collect. Only the following set of default fields are collected:traceID
,spanID
,parentSpanID
,startThread
andoperationName
. -
If I used
com.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES
for using categories defined by operation name, Does the operation name have to match exactly to the event type name in the configuration file? If it does, and if I have unique operation names, Do I need to specify an event type for each operation?Yes, it must match the event type name in the configuration file. Even if you have a lot of unique operation names, you will need to specify an event type for each operation because each of those event types are likely to have a different set of fields that need to be configured.
-
My Operation Name is getting converted into a Java identifier therefore I cannot configure its event type, How can I determine what my Operation Name was converted to?
All illegal characters have been replaced with an underscore. If you need a sample Java code to convert your strings into identifiers then you can use the below code whicht it's the same routine that APM uses to convert the strings.// Convert an arbitrary string into a valid java identifier by replacing non-identifier chars with an underscore public static String getJavaIdentifier(String inString) { if (inString == null || inString.length() == 0) return inString; StringBuilder stringBuilder = new StringBuilder(); char[] allChars = inString.toCharArray(); if(Character.isJavaIdentifierStart(allChars[0])) stringBuilder.append(allChars[0]); else stringBuilder.append("_"); for (int i = 1; i < allChars.length; i++) { if (Character.isJavaIdentifierPart(allChars[i])) stringBuilder.append(allChars[i]); else stringBuilder.append("_"); } return stringBuilder.toString(); }
-
Can the name matching be RegEx?
No. RegEx is not supported.