Configure Thread Snapshots in APM Tracer

The thread snapshots feature allows thread stack snapshots collection which can be used for deep dive analysis.

The thread snapshots allows collection of traces like thread stack snapshots, thread memory overhead, thread cpu overhead and gc impact on trace. It collects the thread stack snapshots and stack related data like state and locks, at a specific interval.

Users can enable thread stack snapshots collection for taking stack dump of spans. It provides user code stack visibility.

Thread snapshots collection default time interval is 250 ms.

APM Tracer 1.4 version or higher supports thread snapshots and it's disabled by default.

APM Tracer and APM Agent support thread snapshots. For details about configuring thread snapshots in APM Agent, see Configure Thread Snapshots in APM Agent.

This section covers the following:

Enable Thread Snapshots in APM Tracer

Thread Snapshot is integrated with ApmTracer. It will get initialized as part of ApmTracer object creation.

If you are using Helidon, skip to Thread Snapshots in Helidon.

The below example shows many configuration methods including the one for thread snapshots feature: withCollectDeepDiveTraceSnapshot with value (true). Thread snapshots will get called from the build function of ApmTracer:


ApmTracer tracer = (ApmTracer) new ApmTracer.Builder(AGENT_TRACER_NAME,serviceName)
                      .withProperty(PROP_NAME_HYBRID_TRACER_STARTER, PROP_VALUE_HYBRID_TRACER_STARTER)                      
                      .withMicrosecondAccurateTimestamp(microsecondAccurate)                      
                      .withObject(agentCommon)                       
                      .withTraceListener(traceListener)                      
                      .withCollectThreadId(collectThreadId)                      
                      .withCollectThreadName(collectThreadName)                      
                      .withCollectDeepDiveTraceSnapshot(true)
                      .build();

For information about how to instantiate APM Tracer, see Instantiate APM Tracer.

Thread Snapshots Configuration Methods

The following configuration methods are available:

Method	Description	Unit	Default Value	Supported Values
withMaxPercentTraceExportToCollectorPerMinute	Maximum percentage of per minute incoming traces to export them to collector.	Percentage	100	Minimum:1 Maximum: 100
withMaxTraceExportToCollectorPerMinute	Maximum number of top traces per minute to export them to collector.	Number	200	Minimum:1 Maximum: 1500
withSnapshotCollectionIntervalInMS	Interval at which thread snapshot data will be collected.	Milliseconds	250	Minimum:250 Maximum: 300000
withMaxSnapshotsPerTrace	Maximum number of snapshots allowed per trace.	Number	16	Minimum: 1 Maximum: 16
withMaxStackTraceDepthPerSnapshot	Maximum stack trace frames collected for any thread.	Number	256	Minimum: 1 Maximum: 256
withLogLevel	Print deep dive logs, which are equal or above to this property value.	String	INFO	DEBUG, INFO, WARN, SEVERE
withMaxProcessCpuUsagePercent	Maximum process CPU usage. It's recommended to maintain overall process CPU usage under this threshold.	Number	90	Minimum: 1 Maximum: 90
withDeepDiveCpuUsagePercent	Minimum threshold to maintain overall deepdive CPU usage.	Percentage	3	Minimum: 1 Maximum: 5
	Maximum threshold to maintain overall deepdive CPU usage.	Percentage	5	Minimum: 5 Maximum: 20
withTraceLifeInMS	Minimum threshold for a trace.	Milliseconds	50	Minimum: 50 Maximum: 60000
	Maximum threshold for a trace.	Milliseconds	60000	Minimum: 60000 Maximum: 600000

For example:

DeepDiveConfiguration deepDiveConfiguration = DeepDiveConfigurationBuilder.builder()
                                              .withMaxPercentTraceExportToCollectorPerMinute(100) 
                                              .withMaxTraceExportToCollectorPerMinute(200) 
                                              .withSnapshotCollectionIntervalInMS(250)     
                                              .withMaxSnapshotsPerTrace(16)               
                                              .withMaxStackTraceDepthPerSnapshot(256)                   
                                              .withLogLevel("INFO")                             
                                              .withMaxProcessCpuUsagePercent(90)                              
                                              .withDeepDiveCpuUsagePercent(Range.RangeBuilder.builder()
                                              .withMinThreshold(3).withMaxThreshold(5).build()) 
                                              .withTraceLifeInMS(Range.RangeBuilder.builder()
                                              .withMinThreshold(50).withMaxThreshold(60000).build()) 
                                              .build();

ApmTracer tracer = (ApmTracer) new ApmTracer.Builder(AGENT_TRACER_NAME, serviceName)
                            .withObject(deepDiveConfiguration)
                            .withCollectDeepDiveTraceSnapshot(true)
                            .build();

Thread Snapshots in Helidon

Thread Snapshots is supported in Helidon. It will get initialized as part of the ApmTracer object creation inside Helidon.

To enable thread snapshots, update the YAML file of Helidon and add the collect-deepdive-trace-snapshots property and set it to true.

For example:

tracing:
    name: "Helidon APM Tracer"
    service: "helidonse-http"
    data-upload-endpoint:<endpoint-url>
    private-data-key: <private-data-key>
    collect-metrics:true
    collect-resources: true
    collect-deepdive-trace-snapshots:true

For information about using APM Tracer in Helidon, see Use APM Tracer in Helidon.

Configuration Properties for Helidon

To configure thread snapshots in Helidon, update the YAML file of Helidon using the following properties:

Property Name	Description	Unit	Default Value	Supported Values
com.oracle.apm.agent.deepdive.maxPercentTraceExportToCollectorPerMinute	Maximum percentage of per minute incoming traces to export them to collector.	Percentage	100	Minimum:1 Maximum: 100
com.oracle.apm.agent.deepdive.maxTraceExportToCollectorPerMinute	Maximum number of top traces per minute to export them to collector.	Number	200	Minimum:1 Maximum: 1500
com.oracle.apm.agent.deepdive.snapshotCollectionIntervalInMS	Interval at which thread snapshot data will be collected.	Milliseconds	250	Minimum:250 Maximum: 300000
com.oracle.apm.agent.deepdive.maxSnapshotsPerTrace	Maximum number of snapshots allowed per trace.	Number	16	Minimum: 1 Maximum: 16
com.oracle.apm.agent.deepdive.maxStackTraceDepthPerSnapshot	Maximum stack trace frames collected for any thread.	Number	256	Minimum: 1 Maximum: 256
com.oracle.apm.agent.deepdive.logLevel	Print deep dive logs, which are equal or above to this property value.	String	INFO	DEBUG, INFO, WARN, SEVERE
com.oracle.apm.agent.deepdive.maxProcessCpuUsagePercent	Maximum process CPU usage. It's recommended to maintain overall process CPU usage under this threshold.	Number	90	Minimum: 1 Maximum: 90
com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.minThreshold	Minimum threshold to maintain overall deepdive CPU usage.	Percentage	3	Minimum: 1 Maximum: 5
com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.maxThreshold	Maximum threshold to maintain overall deepdive CPU usage.	Percentage	5	Minimum: 5 Maximum: 20
com.oracle.apm.agent.deepdive.traceLifeInMS.minThreshold	Minimum threshold for a trace.	Milliseconds	50	Minimum: 50 Maximum: 60000
com.oracle.apm.agent.deepdive.traceLifeInMS.maxThreshold	Maximum threshold for a trace.	Milliseconds	60_000	Minimum: 60000 Maximum: 600000

For example:

deepDiveProperties:
 com.oracle.apm.agent.deepdive.maxPercentTraceExportToCollectorPerMinute: 100
 com.oracle.apm.agent.deepdive.maxTraceExportToCollectorPerMinute: 200
 com.oracle.apm.agent.deepdive.snapshotCollectionIntervalInMS: 250
 com.oracle.apm.agent.deepdive.maxSnapshotsPerTrace: 16
 com.oracle.apm.agent.deepdive.maxStackTraceDepthPerSnapshot: 256
 com.oracle.apm.agent.deepdive.logLevel: INFO
 com.oracle.apm.agent.deepdive.maxProcessCpuUsagePercent: 90
 com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.minThreshold: 3
 com.oracle.apm.agent.deepdive.deepDiveCpuUsagePercent.maxThreshold: 5
 com.oracle.apm.agent.deepdive.traceLifeInMS.minThreshold: 50
 com.oracle.apm.agent.deepdive.traceLifeInMS.maxThreshold: 60000

Verify Thread Snapshots

To verify that thread snapshots feature has been configured properly and data is being uploaded to Application Performance Monitoring, use the Snapshot view option from View Trace or Span Details.

Oracle Cloud Infrastructure Documentation