Configure APM Sampling
APM Sampling is an optional configuration for the APM Java Agent and APM Tracer to monitor a fraction of the spans based on the sampling rule.
APM Sampling is a root span based sampling implementation allowing sampled traces to yield representation of all traces. Sampling decision is made based on sampling rules. When a root span (same as trace) is decided to be sampled, the root span and all children spans are also sampled. When a root span is not sampled, then all children span are not sampled.
Rules Available
The following rules are available for APM sampling:
Rule | Rule Name | Rule Parameter | Sample Parameter | Description |
---|---|---|---|---|
Constant | constant | integer
(0 or above) |
50 | Sample constantly once per the specified parameter
value.
For example, if the parameter is set to 50, the first one of every 50 traces is sampled. If the parameter is set to 0, no trace is sampled. If the parameter is set to 1, every trace is sampled. |
Probabilistic | probabilistic | decimal number between 0 and 1 | 0.2 | Sample decided on the specified probability.
For example, if the parameter is set to 0.2, there is a 20% chance a trace is sampled. |
Limited Rate | limited-rate | <limit>/<seconds> | 500/60 | Sample up to the specified limit for every specified
seconds window.
For example, if the parameter is set to 500/60, the first 500 traces are sampled of every 60 seconds. |
Per Operation Limited Rate | per-operation-rate | <limit>/<seconds> | 10/60 | Sample up to the specified limit for every specified
seconds window per operation name.
For example, if the parameter is set to 10/60, the first 10 traces of each operation name are sampled of every 60 seconds. |
Operation Name | operation | <op1>,<op2>, ...<opN> | /status_order,/status_ship | Sample only the specified operation name. Multiple operations can be specified delimited by comma. |
Operation Name in Regex | operation-regex | <regex> | /status_(order|ship) | Sample only spans with operation that matches the regex. |
Configuration Methods
- Single Rule properties: An easy way to define one rule applicable to all traces.
- Configuration File: A comprehensive way to define samplings including branching rules and operation priority rules.
If both methods are configured, the single rule properties takes precedence over the configuration file therefore the single rule properties method is the one being used.
There is no default sampling configuration. If there's no single rule properties or no configuration file was specified then all new traces started by APM Agent or APM Tracer are sampled.
Single Rule Properties
Single rule properties allows one sampling rule be defined and applied to all traces in an easy way.
The following properties are available:
Properties Type and Description | Supported by | Properties | Example |
---|---|---|---|
AgentConfig.properties Update the |
APM Agent |
|
|
Tracer Builder Update the Tracer Builder. |
APM Tracer |
|
|
System Properties Update the system properties. |
APM Agent and APM Tracer |
|
|
Environment Variables Update the environment variables. |
APM Agent and APM Tracer |
|
For Windows:
For Linux:
|
Properties precedence: If more than one property are set, the order of precedence from highest to lowest is as follows: System Properties, Environment Variables and AgentConfig.properties.
Configuration File
The configuration file allows all sampling features for all services to be specified in a single file with the following characteristics:
- The configuration file is in ACML (subset of YAML) format to be
specified in
Sampling.acml
. - Multiple sampling configurations for different services can be defined and shared from a single configuration file.
- Sampling rule is not limited to one. Sample rules can be branched out to other rules to form a rule tree. Each rule evaluation has 2 outcomes: true or false. True means the trace is sampled, false otherwise. In a branched rule, the terminal leaf outcome is used for sampling decision.
- Rules with specific names can be prioritized with their own sampling rule.
For the APM Tracer, there is no default configuration location. The
sampling configuration file must be specified with the tracer property, system
property or environment variable using the
com.oracle.apm.agent.sampling.file
property. The property value
is the full path of the sampling configuration file.
The com.oracle.apm.agent.sampling.file
property is only applicable
to APM Tracer. For APM Agent, the property is not needed. You only require to place
the Sampling.acml
sampling file in
oracle-apm-agent/config/<version>
directory.
Properties Type and Description | Supported by | Property | Example |
---|---|---|---|
Tracer Builder
Update the tracer builder. |
APM Tracer | com.oracle.apm.agent.sampling.file |
|
System Property Update the system properties. |
APM Agent and APM Tracer | com.oracle.apm.agent.sampling.file |
-Dcom.oracle.apm.agent.sampling.file=/home/user/apm/config/Sampling.acml |
Environment Variable
Update the environment variable. |
APM Agent and APM Tracer | com_oracle_apm_agent_sampling_file |
For Windows:
For Linux:
|
The com.oracle.apm.agent.sampling.file
property value is the full
path of the sampling configuration file. When it's specified, the recommended
sampling configuration file name is Sampling.acml
, but you can use
any other preferred name.
- Sampling: It uses the
sampling
entry and it defines a rule tree for trace evaluation. - Operation Priority: It uses the
operation_priority
entry and when it's defined, it overrides sampling if trace root span operation name matched defined name. Each defined operation name has its own decision rule tree.
Pre-Sampling Decision
Pre-sampling decision is a sampling feature that allows the sampler to make a decision based on the parent span before applying the sampling rule.
Example:
sampling_config:
-
service:
- s1
- s2
pre_sampling_decision:
ignore: rum
sampling:
rule: constant
param: 1000
true:
rule: per-operation-rate
param: 1000/60
For details of the pre_sampling_decision
syntax, see
Sampling.acml File Format.
Sampling.acml File Format
The following is the Sampling.acml
file format:
Sampling.acml | Description |
---|---|
|
Node representing an array of different sampling config. |
|_ |
Array element of sampling config. |
|____ |
Node of service names this sampling config is applied to. |
|_______ |
Service name array element. |
|____ |
(Optional) Node of pre-sampling decision.
Decision is evaluated in the order of override → enforce → ignore. In case a source is defined in more than one decision, the first decision having the matching source is used. When no parent type matches any decision, then APM Agent follows the parent span sampled flag for local span. If there was no parent span, then sampling rule is evaluated for the local span. All decision nodes are optional. If no
pre-sampling decision is needed, then
|
|_______ |
Sample local APM Agent span regardless the parent sampled flag of specified source. If the parent span is not sampled, then a new trace is started with the local span as root. For |
|_______ |
Force sampling re-evaluation regardless the parent sampled flag of specified source. If the parent span is not sampled, and local APM Agent span is evaluated to be sampled, then a new trace is started. If parent span is sampled, and local APM Agent span is evaluated to be sampled, then the local APM Agent span continues on the trace of the parent span. For |
|_______ |
Ignore sampling of local APM Agent span of specified parent. For |
|____ |
Node of base sampling.
It's
optional if |
|_______ |
Sampling rule tree of this config. |
|____ |
Array of operation names with rules overriding base sampling rule. |
|_______ |
Name to match trace root span operation name. |
|_______ |
Rule to be applied if matched. |
|_______ |
Regex to match trace root span operation name. |
|_______ |
Rule to be applied if matched. |
-
If the service node (
service:
) is not specified, then the sampling config is applied to all services without dedicated sampling config. When multiple configs have the same service name, then only the last config with that service name is used.If a file has configs with and without this service node, the config with the service name matching the APM Agent or APM Tracer service name is prioritized.
-
When sampling (
sampling:
) and operation_priority (operation_priority:
) nodes are both specified, the operation_priority is prioritized. If the trace root span name matched the operation name from operation_priority node, it is evaluated with the rule associated with the operation name. If the trace root span name did not match any operation names from the operation_priority node, then the sampling node is evaluated. -
If sampling node (
sampling:
) is not specified, or no rule is specified, then the trace is not sampled when no operation_priority rule is applicable. -
When the operation_priority node (
operation_priority:
) is specified, if the operation name of the trace root span matches any operation_priority operation names, then the rule of matching name is used for evaluation instead of the base sampling rule.
Rule Node
The rule node from the Sampling.acml
file has the
following format:
<rule> node | Description |
---|---|
rule:
<rule_name> |
Name of sampling rule. |
param:
<rule_parameter> |
Parameter of the rule. |
true: |
True outcome of rule evaluation. Value can be
another <rule>, or empty.
If this node is not specified or the node value is empty, then the true evaluation is also the final sampling evaluation result and the trace is sampled. |
|___ <rule> |
Branch <rule> node on true evaluation. |
false: |
False outcome of rule evaluation. Value can be
another <rule> or empty.
If this node is not specified or the node value is empty, then the false evaluation is also the final evaluation result and the trace is not sampled. |
|___ <rule> |
Branch <rule> node on false evaluation. |
Changes to Sampling.acml
sampling configuration
file does not require to restart the APM Agent.
Source Value
If using pre_sampling_decision
, you have the option to
provide the source information.
The <source>
value from the Sampling.acml
file
can be empty or it can contain multiple sources. For supported values, see the below
table.
Source Name | Description |
---|---|
apm | APM Agent source. |
rum | RUM (Real User Monitoring) source.
It's not APM Agent span and always the parent span of the Java Agent. |
synthetic | Synthetic source.
By default, all synthetic requests are sampled. |
* | Any source. |
? | Any source type not defined in <source>. |
null | Any parent without identifiable source. |
Sampling.acml File Examples
- Rule that apply to all
services
sampling_config: - sampling: # Every trace has 2% chance to be sampled. rule: probabilistic param: 0.02
- Probabilistic with minimum
count
sampling_config: - sampling: # For every 60 seconds duration, 5 root spans of each operation name are sampled, # followed by 10% chance of sampling. rule: per-operation-rate param: 5/60 true: false: rule: probabilistic param: 0.1
- Sample only specified
operations
sampling_config: - service: - order_dept - shipping_dept operation_priority: # For /order operation trace root span, 1 of every 10 traces is sampled. - name: "/order" rule: constant param: 10 # For /ship operation trace root span, 1 of every 50 traces is sampled. - name: "/ship" rule: constant param: 50 # For /status_order or /status_ship operations, there is a 50% sampling chance. - regex: "/status_(order|ship)" rule: probabilistic param: 0.5
- Other sample
example
sampling_config: - # With service node, this config is only applied to service s1 and s2. service: - s1 - s2 sampling: # 5 root span of any operation name are sampled every 60 seconds. Starting from # the 6th span of an operation name, there is a 20% chance the span is sampled. rule: per-operation-rate param: 5/60 false: rule: probabilistic param: 0.2 operation_priority: # For /order operation, 1 of every 10 spans is sampled. - name: "/order" rule: constant param: 10 # For /ship operation, first 3 are sampled per 60 seconds and then 10% chance after - name: "/ship" rule: limited-rate param: 3/60 false: rule: probabilistic param: 0.01 - # With service node, this config is only applied to service s3 and s4. service: - s3 - s4 sampling: # There is a 5% chance any trace is sampled. rule: probabilistic param: 0.05 - # Without service node, this config is applied to any service except s1, s2, s3 and s4. sampling: # There is a 10% chance any trace is sampled. rule: probabilistic param: 0.1
Disable Sampling
Sampling is implicitly enabled when either single rule properties or the configuration file is specified. It can then be explicitly disabled by using the following property:
Properties Type | Supported by | Property | Example |
---|---|---|---|
AgentConfig.properties | APM Agent | com.oracle.apm.agent.sampling.enabled |
com.oracle.apm.agent.sampling.enabled=false |
Tracer Builder | APM Tracer | com.oracle.apm.agent.sampling.enabled |
|
System Property | APM Agent and APM Tracer | com.oracle.apm.agent.sampling.enabled |
-Dcom.oracle.apm.agent.sampling.enabled=false |
Environment Variable | APM Agent and APM Tracer | com_oracle_apm_agent_sampling_enabled |
For Windows:
For Linux:
|
Span Tag
If sampling rule properties or configuration file is specified and
sampling is not disabled, every sampled trace root span has a tag
SamplingEvaluation
of evaluation flow.
rule: per-operation-rate
param: 5/60
false:
rule: probabilistic
param: 0.2
SamplingEvaluation Tag Value
See below the value of SamplingEvaluation
tag of sampled
root span:
SamplingEvaluation Tag Value
|
Description |
---|---|
per-operation-rate(5/60):true | The first 5 sampled traces of every 60 seconds. |
per-operation-rate(5/60):false->probabilistic(0.2):true | The 20% traces sampled after 5 sampled traces of every 60 seconds. |