Configure APM Sampling

APM Sampling is an optional configuration for the APM Java Agent and APM Tracer to monitor a fraction of the spans based on the sampling rule.

APM Sampling is a root span based sampling implementation allowing sampled traces to yield representation of all traces. Sampling decision is made based on sampling rules. When a root span (same as trace) is decided to be sampled, the root span and all children spans are also sampled. When a root span is not sampled, then all children span are not sampled.

Rules Available

The following rules are available for APM sampling:

Rule Rule Name Rule Parameter Sample Parameter Description
Constant constant integer

(0 or above)

50 Sample constantly once per the specified parameter value.

For example, if the parameter is set to 50, the first one of every 50 traces is sampled. If the parameter is set to 0, no trace is sampled. If the parameter is set to 1, every trace is sampled.

Probabilistic probabilistic decimal number between 0 and 1 0.2 Sample decided on the specified probability.

For example, if the parameter is set to 0.2, there is a 20% chance a trace is sampled.

Limited Rate limited-rate <limit>/<seconds> 500/60 Sample up to the specified limit for every specified seconds window.

For example, if the parameter is set to 500/60, the first 500 traces are sampled of every 60 seconds.

Per Operation Limited Rate per-operation-rate <limit>/<seconds> 10/60 Sample up to the specified limit for every specified seconds window per operation name.

For example, if the parameter is set to 10/60, the first 10 traces of each operation name are sampled of every 60 seconds.

Operation Name operation <op1>,<op2>, ...<opN> /status_order,/status_ship Sample only the specified operation name. Multiple operations can be specified delimited by comma.
Operation Name in Regex operation-regex <regex> /status_(order|ship) Sample only spans with operation that matches the regex.

Configuration Methods

The following methods are available to configure APM sampling:
  • Single Rule properties: An easy way to define one rule applicable to all traces.
  • Configuration File: A comprehensive way to define samplings including branching rules and operation priority rules.

If both methods are configured, the single rule properties takes precedence over the configuration file therefore the single rule properties method is the one being used.

There is no default sampling configuration. If there's no single rule properties or no configuration file was specified then all new traces started by APM Agent or APM Tracer are sampled.

Single Rule Properties

Single rule properties allows one sampling rule be defined and applied to all traces in an easy way.

The following properties are available:

Properties Type and Description Supported by Properties Example

AgentConfig.properties

Update the AgentConfig.properties file located under oracle-apm-agent/config/<version> directory.

APM Agent
com.oracle.apm.agent.sampling.rule
com.oracle.apm.agent.sampling.param
com.oracle.apm.agent.sampling.rule=limited-rate
com.oracle.apm.agent.sampling.param=1000/60

Tracer Builder

Update the Tracer Builder.

APM Tracer
com.oracle.apm.agent.sampling.rule
com.oracle.apm.agent.sampling.param
new ApmTracer.Builder(...)
    ...
    .withProperty(com.oracle.apm.agent.sampling.rule, "limited-rate")
    .withProperty(com.oracle.apm.agent.sampling.param, "1000/60")
    ...
    .build();

System Properties

Update the system properties.

APM Agent and APM Tracer
com.oracle.apm.agent.sampling.rule
com.oracle.apm.agent.sampling.param
java -Dcom.oracle.apm.agent.sampling.rule=limited-rate -Dcom.oracle.apm.agent.sampling.param=1000/60 -javaagent:... <main_class>

Environment Variables

Update the environment variables.

APM Agent and APM Tracer
com_oracle_apm_agent_sampling_rule
com_oracle_apm_agent_sampling_param
For Windows:
set com_oracle_apm_agent_sampling_rule=limited-rate
set com_oracle_apm_agent_sampling_param=1000/60
For Linux:
export com_oracle_apm_agent_sampling_rule=limited-rate
export com_oracle_apm_agent_sampling_param=1000/60

Properties precedence: If more than one property are set, the order of precedence from highest to lowest is as follows: System Properties, Environment Variables and AgentConfig.properties.

Configuration File

The configuration file allows all sampling features for all services to be specified in a single file with the following characteristics:

  • The configuration file is in ACML (subset of YAML) format to be specified in Sampling.acml.
  • Multiple sampling configurations for different services can be defined and shared from a single configuration file.
  • Sampling rule is not limited to one. Sample rules can be branched out to other rules to form a rule tree. Each rule evaluation has 2 outcomes: true or false. True means the trace is sampled, false otherwise. In a branched rule, the terminal leaf outcome is used for sampling decision.
  • Rules with specific names can be prioritized with their own sampling rule.

For the APM Tracer, there is no default configuration location. The sampling configuration file must be specified with the tracer property, system property or environment variable using the com.oracle.apm.agent.sampling.file property. The property value is the full path of the sampling configuration file.

The com.oracle.apm.agent.sampling.file property is only applicable to APM Tracer. For APM Agent, the property is not needed. You only require to place the Sampling.acml sampling file in oracle-apm-agent/config/<version> directory.

Properties Type and Description Supported by Property Example
Tracer Builder

Update the tracer builder.

APM Tracer com.oracle.apm.agent.sampling.file
new ApmTracer.Builder(...)
    ...
    .withProperty(com.oracle.apm.agent.sampling.file, "/home/user/apm/config/Sampling.acml")
    ...
    .build();
System Property

Update the system properties.

APM Agent and APM Tracer com.oracle.apm.agent.sampling.file -Dcom.oracle.apm.agent.sampling.file=/home/user/apm/config/Sampling.acml
Environment Variable

Update the environment variable.

APM Agent and APM Tracer com_oracle_apm_agent_sampling_file

For Windows:

set com_oracle_apm_agent_sampling_file=c:\apm\config\Sampling.acml

For Linux:

export com_oracle_apm_agent_sampling_file=/home/user/apm/config/Sampling.acml

The com.oracle.apm.agent.sampling.file property value is the full path of the sampling configuration file. When it's specified, the recommended sampling configuration file name is Sampling.acml, but you can use any other preferred name.

There are 2 sampling strategies:
  • Sampling: It uses the sampling entry and it defines a rule tree for trace evaluation.
  • Operation Priority: It uses the operation_priority entry and when it's defined, it overrides sampling if trace root span operation name matched defined name. Each defined operation name has its own decision rule tree.

Pre-Sampling Decision

Pre-sampling decision is a sampling feature that allows the sampler to make a decision based on the parent span before applying the sampling rule.

Example:

sampling_config:
    -
        service:
            -   s1
            -   s2
        pre_sampling_decision:
            ignore: rum
        sampling:
            rule: constant
            param: 1000
            true:
                rule: per-operation-rate
                param: 1000/60

For details of the pre_sampling_decision syntax, see Sampling.acml File Format.

Sampling.acml File Format

The following is the Sampling.acml file format:

Sampling.acml Description

sampling_config:

Node representing an array of different sampling config.

|_-

Array element of sampling config.

|____service:

Node of service names this sampling config is applied to.

|_______- <service-name>

Service name array element.

|____pre_sampling_decision:

(Optional) Node of pre-sampling decision.

Decision is evaluated in the order of override → enforce → ignore.

In case a source is defined in more than one decision, the first decision having the matching source is used. When no parent type matches any decision, then APM Agent follows the parent span sampled flag for local span.

If there was no parent span, then sampling rule is evaluated for the local span.

All decision nodes are optional. If no pre-sampling decision is needed, then pre_sampling_decision should not be defined.

|_______override: <source>

Sample local APM Agent span regardless the parent sampled flag of specified source.

If the parent span is not sampled, then a new trace is started with the local span as root.

For source values, see Source Values.

|_______enforce: <source>

Force sampling re-evaluation regardless the parent sampled flag of specified source.

If the parent span is not sampled, and local APM Agent span is evaluated to be sampled, then a new trace is started.

If parent span is sampled, and local APM Agent span is evaluated to be sampled, then the local APM Agent span continues on the trace of the parent span.

For source values, see Source Values.

|_______ignore: <source>

Ignore sampling of local APM Agent span of specified parent.

For source values, see Source Values.

|____sampling:

Node of base sampling.

It's optional if operation_priority node is used.

|_______<rule>

Sampling rule tree of this config.

|____operation_priority:

Array of operation names with rules overriding base sampling rule.

|_______- name: <operation-name>

Name to match trace root span operation name.

|_______<rule>

Rule to be applied if matched.

|_______- regex: <operation-name-regex>

Regex to match trace root span operation name.

|_______<rule>

Rule to be applied if matched.
  • If the service node (service:) is not specified, then the sampling config is applied to all services without dedicated sampling config. When multiple configs have the same service name, then only the last config with that service name is used.

    If a file has configs with and without this service node, the config with the service name matching the APM Agent or APM Tracer service name is prioritized.

  • When sampling (sampling:) and operation_priority (operation_priority:) nodes are both specified, the operation_priority is prioritized. If the trace root span name matched the operation name from operation_priority node, it is evaluated with the rule associated with the operation name. If the trace root span name did not match any operation names from the operation_priority node, then the sampling node is evaluated.

  • If sampling node (sampling:) is not specified, or no rule is specified, then the trace is not sampled when no operation_priority rule is applicable.

  • When the operation_priority node (operation_priority:) is specified, if the operation name of the trace root span matches any operation_priority operation names, then the rule of matching name is used for evaluation instead of the base sampling rule.

Rule Node

The rule node from the Sampling.acml file has the following format:

<rule> node Description
rule: <rule_name> Name of sampling rule.
param: <rule_parameter> Parameter of the rule.
true: True outcome of rule evaluation. Value can be another <rule>, or empty.

If this node is not specified or the node value is empty, then the true evaluation is also the final sampling evaluation result and the trace is sampled.

|___<rule> Branch <rule> node on true evaluation.
false: False outcome of rule evaluation. Value can be another <rule> or empty.

If this node is not specified or the node value is empty, then the false evaluation is also the final evaluation result and the trace is not sampled.

|___<rule> Branch <rule> node on false evaluation.
Note

Changes to Sampling.acml sampling configuration file does not require to restart the APM Agent.

Source Value

If using pre_sampling_decision, you have the option to provide the source information.

The <source> value from the Sampling.acml file can be empty or it can contain multiple sources. For supported values, see the below table.

Source Name Description
apm APM Agent source.
rum RUM (Real User Monitoring) source.

It's not APM Agent span and always the parent span of the Java Agent.

synthetic Synthetic source.

By default, all synthetic requests are sampled.

* Any source.
? Any source type not defined in <source>.
null Any parent without identifiable source.

Sampling.acml File Examples

  1. Rule that apply to all services
    sampling_config:
      -
         sampling:
           # Every trace has 2% chance to be sampled.
           rule: probabilistic
           param: 0.02
  2. Probabilistic with minimum count
    sampling_config:
      -
         sampling:
          # For every 60 seconds duration, 5 root spans of each operation name are sampled,
          # followed by 10% chance of sampling.
          rule: per-operation-rate
          param: 5/60
          true:
          false:
            rule: probabilistic
            param: 0.1
  3. Sample only specified operations
    sampling_config:
      -
         service:
            - order_dept
            - shipping_dept
         operation_priority:
           # For /order operation trace root span, 1 of every 10 traces is sampled.
           - name: "/order"
             rule: constant
             param: 10
           # For /ship operation trace root span, 1 of every 50 traces is sampled.
           - name: "/ship"
             rule: constant
             param: 50
           # For /status_order or /status_ship operations, there is a 50% sampling chance.
           - regex: "/status_(order|ship)"
             rule: probabilistic
             param: 0.5
  4. Other sample example
    sampling_config:
      -
        # With service node, this config is only applied to service s1 and s2.
        service:
          - s1
          - s2
        sampling:
          # 5 root span of any operation name are sampled every 60 seconds. Starting from
          # the 6th span of an operation name, there is a 20% chance the span is sampled.
          rule: per-operation-rate
          param: 5/60
          false:
            rule: probabilistic
            param: 0.2
        operation_priority:
          # For /order operation, 1 of every 10 spans is sampled.
          - name: "/order"
            rule: constant
            param: 10
          # For /ship operation, first 3 are sampled per 60 seconds and then 10% chance after
          - name: "/ship"
            rule: limited-rate
            param: 3/60
            false:
              rule: probabilistic
              param: 0.01
      -
       # With service node, this config is only applied to service s3 and s4.
       service:
         - s3
         - s4
       sampling:
         # There is a 5% chance any trace is sampled.
         rule: probabilistic
         param: 0.05
      -
       # Without service node, this config is applied to any service except s1, s2, s3 and s4.
       sampling:
         # There is a 10% chance any trace is sampled.
         rule: probabilistic
         param: 0.1

Disable Sampling

Sampling is implicitly enabled when either single rule properties or the configuration file is specified. It can then be explicitly disabled by using the following property:

Properties Type Supported by Property Example
AgentConfig.properties APM Agent com.oracle.apm.agent.sampling.enabled com.oracle.apm.agent.sampling.enabled=false
Tracer Builder APM Tracer com.oracle.apm.agent.sampling.enabled
new ApmTracer.Builder(...)
    ...
    .withProperty(com.oracle.apm.agent.sampling.enabled, "false")
    ...
    .build();
System Property APM Agent and APM Tracer com.oracle.apm.agent.sampling.enabled -Dcom.oracle.apm.agent.sampling.enabled=false
Environment Variable APM Agent and APM Tracer com_oracle_apm_agent_sampling_enabled

For Windows:

set com_oracle_apm_agent_sampling_enabled=false

For Linux:

export com_oracle_apm_agent_sampling_enabled=false

Span Tag

If sampling rule properties or configuration file is specified and sampling is not disabled, every sampled trace root span has a tag SamplingEvaluation of evaluation flow.

See below an example of a rule tree used for evaluation:
rule: per-operation-rate
param: 5/60
false:
     rule: probabilistic
     param: 0.2

SamplingEvaluation Tag Value

See below the value of SamplingEvaluation tag of sampled root span:

SamplingEvaluation Tag Value Description
per-operation-rate(5/60):true The first 5 sampled traces of every 60 seconds.
per-operation-rate(5/60):false->probabilistic(0.2):true The 20% traces sampled after 5 sampled traces of every 60 seconds.