Record APM Spans as JFR Events

APM Tracer and APM Agent support recording APM spans as JFR Events.

Currently JDK 11 or higher are supported.

For information about recording APM spans as JFR events when using APM Agent, see Record APM Spans as JFR Events under Provision and Deploy APM Java Agents on Application Servers.

Note

Java Flight Recorder (JFR) is a tool for collecting diagnostic and profiling data about a running Java application. For information about Java Flight Recorder (JFR) and Java Mission Control (JMC), see JDK Mission Control.

Capture APM Events in JFR Recordings

To capture APM events in JFR recordings when using the APM Tracer, follow these steps:

  1. Ensure APM Tracer is working properly.

    Open Trace Explorer and confirm you are able to view all the traces. For information about opening and using Trace Explorer, see Monitor Traces in Trace Explorer.

  2. Add a dependency to the Maven project which will let the APM Tracer add events into the JFR recordings made by the Helidon JVM.

    <dependency>
        <groupId>com.oracle.apm.agent.java</groupId>
        <artifactId>apm-java-agent-jfr11</artifactId>
        <version>[1.0.1389,)</version>
    </dependency>
  3. Initiate a recording.

    To test that everything is working properly, you need to initiate a recording using your preferred method (It could be using JDK Mission Control UI or JCMD comand line utility). After a recording has been initiated, trace events are displayed on JDK Mission Control UI when the recording is viewed.

    To verify that APM trace events are properly added to JFR recordings, connect to JDK Mission Control and confirm the APM traces are visible under Event Browser: Click on APM OpenTracing and confirm that APM trace events are visible.

Configure JFR Events in APM Tracer

(Optional) This is an optional step if you want to customize the reporting of the events.

To simplify the JFR recording events analysis, each event is organized into a category or an event type.

Default Category

In the APM Tracer, the default category for APM JFR events is obtained by using the contents of the Span Operation Name (in a field called name within the Span). The component tag may not be populated therefore it cannot be used as a default category.

Events

An event type allows the events to get organized and classified inside the JMC user interface. When the APM Tracer gets a span, it can generate an event for this span, but it needs to determine the event type of that specific event.

Examples of event type are servlet, jdbc and jax-rs.

Event Type Configuration

There are several ways to do the configuration of an event type or category. See below the different use cases:

  • Case 1: component tag

    This is the simplest way to classify an event. This is the recommended method to use.

    In most scenarios, the Opentracing span will contain a tag called component. The value of component tag is used to determine the event type.

    For example, see the following span:

    {  "id":"e926ff3d06013045",
      "trace-id":"49abd92e69786853e926ff3d06013045",
      "parent-id":"6ef54ec2c524c99a",
      "name":"content-write",
      "ts-micros":1614190751360177,
      "td-micros":219348,
      "attributes":{
        "server":"Helidon SE",
        "http.url":"http://localhost:8079/health",
        "http.status_code":200,
        "component":"helidon-webserver",
        "http.method":"GET",
        "organization":"development"
      },
      "events":[
      ]
    },

    In this case the event generated will get assigned a helidon-webserver event type since that is the value of the component tag.

  • Case 2: Other tag

    If your application does not use the component tag, but it uses another tag or there is a library which it's generating the tags for you then the APM Tracer can be configured to read the event type from that other tag.

    For example, see the following span where you can read the event type from the tag called span.type:

    {  "id":"e926ff3d06013045",
      "trace-id":"49abd92e69786853e926ff3d06013045",
      "parent-id":"6ef54ec2c524c99a",
      "name":"content-write",
      "ts-micros":1614190751360177,
      "td-micros":219348,
      "attributes":{
        "server":"Helidon SE",
        "http.url":"http://localhost:8079/health",
        "http.status_code":200,
        "span.type":"helidon-webserver",
        "http.method":"GET",
        "organization":"development"
      },
      "events":[
      ]
    },

    In this case, you can get the APM Tracer to read the event type from the span.type tag by using the following system property when starting the java application:

    java -Dcom.oracle.apm.agent.jfr.category.tag=span.type ...
  • Case 3: Use Operation Name when no tags are available

    If your application does not use any tag that has useful categories then it is possible to use the Operation Name to determine the event type, but this method is not recommended.

    The JFR system requires that the name of the event type be a valid Java identifier, but the Operation Name often has spaces and also, other illegal characters. APM Tracer will convert the Operation Name into a valid identifier by replacing all illegal characters with an underscore.

    java -Dcom.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES
    Note

    The Operation Name will only be used if the category tag is missing or empty.

    This is not a simple solution because the Operation Name may contain unique tokens like IDs which will make every event into it's own event type. In addition to that, the conversion of the Operation Name into a valid identifier will cause further problems in configuring the event type to setup which fields are needed in the event, and also, the configuration will need to be done based on the converted Operation Name otherwise you would need to configure many different variants of the Operation Name in the same way.

  • Case 4: There are no tags available and all of the Operation Names are unique

    In this case there is no information available to classify the event types. This is the worst case scenario where all of the events will end up as the same event type, or everyone ends up unique if they are configured using use case 3 scenario as seen above. The only thing possible to do is to allow you to choose the name of this default event type and this can be configured as shown below.

    java -Dcom.oracle.apm.agent.jfr.category.default=Unknown
    Note

    The default will only be used if the category tag is missing or empty.

    In this case, all the events are placed in the event type called Unknown or any custom value selected.

Fields for the event type

For every event type, we can add 1 or more fields which contain data specific to that event.

The list of fields that can be added to the event type needs to be configured in the ACML file and only after the fields can be published to JFR. For each field, the following information is required:
  • Name.
  • Type (int, float, string, thread).
  • Label.
  • Description.
For example, see the following field definition for traceID:
- name: traceID
  type: String
  label: New Trace ID
  description: New Opentracing Trace ID

Create an ACML File

This is an optional step. If the default settings are working as you expected, you can skip this step.

You can create an ACML file if you want to change the default settings and customize the recordings.

The following example shows the format of an ACML file configuring event logging to JFR by overwriting the built-in default settings for the Helidon microservices container:

caseSensitiveFieldName: false
recordCallStack: true

eventTypes:
  helidon-webserver:
    name: helidon-webserver
    label: Helidon Webserver Events
    description: All events based on helidon webserver
    recordCallStack: true
    # All fields must be provided here.
    # Fields are NOT incremental since that would make them impossible to remove 
    fields:
      - name: traceID
        type: String
        label: New Trace ID
        description: New Opentracing Trace ID
      - name: spanID
        type: String
        label: New Span ID
        description: New Opentracing Span ID
      - name: http.url
        type: String
        label: New HTTP URL
        description: The New URL for the HTTP endpoint being hit
      - name: organization
        type: String
        label: Organization
        description: The New Organization value of this call
      - name: startThread
        type: Thread
        label: Start Thread
        description: The thread on which this event started
      - name: http.status_code
        type: int
        label: Status Code
        description: The HTTP Response status code
      - name: http.method
        type: String
        label: HTTP Method
        description: The HTTP Method

You can use the above format of the ACML file when using APM Agent and APM Tracer. For APM Tracer, it will work for any container like Helidon, Springboot and Micronaut.

Frequently Asked Questions

  1. When you have both settings provided: com.oracle.apm.agent.jfr.category.tag and com.oracle.apm.agent.jfr.category.default properties, Which setting does it take priority?

    If they are both given a value then we will first look in the span for the category tag. If it's found, that will be used. If it is missing, the default will be used.

  2. For APM Tracer, Is a configuration file mandatory? Can I just specify the properties without file?

    A configuration file is not mandatory. If the default works for you then you don't need a configuration file, and you can still specify the properties.

  3. In some cases, Why does it look like entries are duplicated? For example, the following snippet from the ACML configuration file has the helidon-webserver entry twice:
    eventTypes:
      helidon-webserver:
        name: helidon-webserver

    The first helidon-webserver entry is what we read/deduce from the tags or the operation name. If the event type is clean, easy to ready and intuitive, like the above example, then you may use the same event type to send to the JFR recording, and that how it will be get displayed in JMC.

    However, sometimes when using the operation name you may get an event type like the following: q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A= which represents a JDBC query from a db client. In this case you would want to change the name sent to the JFR recording to something intuitive therefore the below example can be used in those cases:
    eventTypes:
      q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A=:
        name: jdbc-select-query
  4. If recordCallStack can be specified for each event type, Why is there also a need to have top level recordCallStack?

    We support two use cases for maximum flexibility in recording of the call stack. Call stack recording can be globally turned off and then it can selectively get turned on for specific event types. Another option is to globally turned it on and then it can selectively get turned off for specific event types. Both use cases are supported, therefore the setting are at both the top level, and at the event level.

  5. I have some spans which were not configured at all and they are being ignored. Is it possible to still capture those events without configuring them?

    Yes. If you set allowUnconfiguredEvents: true in the main top level configuration then those events will get captured. When this happens, we don't know which fields to collect. Only the following set of default fields are collected: traceID, spanID, parentSpanID, startThread and operationName.

  6. If I used com.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES for using categories defined by operation name, Does the operation name have to match exactly to the event type name in the configuration file? If it does, and if I have unique operation names, Do I need to specify an event type for each operation?

    Yes, it must match the event type name in the configuration file. Even if you have a lot of unique operation names, you will need to specify an event type for each operation because each of those event types are likely to have a different set of fields that need to be configured.

  7. My Operation Name is getting converted into a Java identifier therefore I cannot configure its event type, How can I determine what my Operation Name was converted to?

    All illegal characters have been replaced with an underscore. If you need a sample Java code to convert your strings into identifiers then you can use the below code whicht it's the same routine that APM uses to convert the strings.
    // Convert an arbitrary string into a valid java identifier by replacing non-identifier chars with an underscore
    public static String getJavaIdentifier(String inString)
    {
        if (inString == null || inString.length() == 0)
           return inString;
        StringBuilder stringBuilder = new StringBuilder();
        char[] allChars = inString.toCharArray();
    
        if(Character.isJavaIdentifierStart(allChars[0]))
            stringBuilder.append(allChars[0]);
        else
            stringBuilder.append("_");
    
        for (int i = 1; i < allChars.length; i++)
        {
            if (Character.isJavaIdentifierPart(allChars[i]))
                stringBuilder.append(allChars[i]);
            else
                stringBuilder.append("_");
        }
        return stringBuilder.toString();
    }
  8. Can the name matching be RegEx?

    No. RegEx is not supported.