Application Performance Monitoring Metrics
You can monitor the health, capacity, and performance of your applications by using metrics, alarms, and notifications.
- Overview
- Prerequisites
- Available Metrics: oracle_apm_rum
- Available Metrics: oracle_apm_synthetics
- Available Metrics: oracle_apm_monitoring for APM Java Agent
- Available Metrics: oracle_apm_monitoring for APM Dotnet Agent
- Available Metrics: oracle_apm_agent for APM Java Agent
- Available Metrics: oci_apm
Overview
Application Performance Monitoring data sources such as APM Java agents and APM Browser agents collect and upload key application metrics to the Oracle Cloud Infrastructure Monitoring service. For information on Application Performance Monitoring data sources, see Application Performance Monitoring Data Sources.
Here are the Monitoring service metric namespaces related to Application Performance Monitoring:
- oracle_apm_rum: Metrics related to Real User Monitoring (RUM), specifically session metrics for all configured web applications as well as metrics for all monitored user activities (page activities, AJAX calls, and script errors).
- oracle_apm_synthetics: Metrics related to Synthetic Monitoring, specifically the monitor's availability, the number of connections established, the number of DNS lookups, and so on.
- oracle_apm_monitoring: All other Application Performance Monitoring metrics, such as the metrics stemming from application servers and custom metrics.
For information on the Monitoring service and its features, see Monitoring Overview.
Prerequisites
IAM policies: To monitor resources, you must be given the required type of access in a policy written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool. The policy must give you access to the monitoring services as well as the resources being monitored. If you try to perform an action and get a message that you don’t have permission or are unauthorized, confirm with your administrator the type of access you've been granted and which compartment you should work in. For more information on user authorizations for monitoring, see the Authentication and Authorization section for the related service: Monitoring or Notifications.
Span Enrichment
About Span Enrichment
- Provide users with the ability to enrich spans with custom attributes or change, combine or remove data from existing attributes.
- Enrichment operations are defined through span rules which contain one or more actions to be applied in a particular order on a given span.
- There are two types of span rules: Custom and Oracle-provided.
- Span enrichment is configured at APM Domain level.
- Removal of Personal Identifiable Information (PII) from spans.
- Combine data from multiple attributes into a single one.
- Extract data from an attribute. For example, a partial path from a longer URL.
- Apply mathematical functions on data. For example, convert a unit from seconds to milliseconds.
- Map an attribute value to a new value. For example, map the value '1' to the color 'blue', value '2' to the color 'red'.
- Distribute numeric values into buckets. For example, a temperature value range that is mapped into 'cold', 'warm' and 'hot' buckets.
Span Enrichment Usage
Span Enrichment is configured through Span Rules which operate on a span and are used to modify the span's attributes. The Span Rules toolbox provides a variety of actions that can be performed as part of a rule.
Span Rules are managed within groups. A group is a collection of rules that are applied in a specified order. Each rule can apply one or more actions on a span and the output of each rule can be used as input to follow-up rules. Rule groups require a Span Filter that is used to select the spans on which to apply the rules.
Multiple groups can be defined. This is useful when you need to maintain different types of rules for different types of Spans as it creates logical entities with functionality designed for a specific purpose. This reduces the overall complexity of your Rule Set.
If multiple groups have Span Filters set up that match the same span, then all of them will apply their rules to that span. The order in which this is performed is determined by the alphabetical order of the group names. Keep this in mind when you name or rename your groups. Oracle recommends prefixing the names with a number to avoid the group names getting sorted alphabetically. For example, "1. Remove Location Data", "2. Convert Length Units".
Keep in mind that when multiple groups are operating on the same span, this means that each group will be operating on a version of the span that may have been modified by a previous group in the chain. This can be useful, as you can write rules in follow-up groups that rely on modifications created by previous groups. On the other hand, it can also cause follow-up rules to fail if an attribute that a rule relies on, is removed or modified by a previous group.
The above is only important when a span matches more than one group.
If your groups have clearly disjoint filters, then a span will always be modified by a single group and the above considerations become irrelevant.
Attributes in Span Events
Spans may contain a list of events which are also referred to as log messages. These are typically used to store information about important moments during the execution of the span. For example, if a span describes a database query, then events could be generated to mark the moment in time when the connection was open, then when the query request was actually submitted and when the response was received.
An event is identified by a name
, timestamp
and a set of attributes
(key-value pairs) that provide further context for the recorded information.
event reference
syntax to identify the proper event attribute:event['NameOfEvent']:'NameOfAttribute'.
Event attributes cannot be used as Span Filters.
Create and Manage Span Rules
You can create and manage Span Rules depending on your reporting needs.
-
Go to the APM Domains page and click on the desired APM domain.
The Domain Details page is displayed.
-
Go to Resources, located at the bottom left, and click Span Enrichment.
The Span Enrichment pane is displayed.
-
Click on Create group to create a new span rule group.
The Create span rule group pane is displayed.- Enter the Name of the new group.
- Select a Span Filter to use with this group.
- Optionally, provide a Description for the new group.
- Click Next to proceed to Rule actions pane to start adding span rules to the new group depending on the span rule type: Custom span rule or Oracle provided span rule.
- Enter the Rule name.
- Optionally, select a Rule span filter to further narrow
down the scope of the rule's operation.
- Optionally, provide a Rule description.
- Select a span rule type.
-
If you select Oracle Provided span rule, you are offered a list of rules to choose from.
The rules have already been created by Oracle in order to assist the users. The list includes the following:
- Adds the client IP address to a
span.
It adds the client IP address to a span as an attribute named
ClientIP
. - EBS suite.
It provides the E-Business Suite template.
- OpenTelemetry to APM naming
conversion.
It converts span attribute names from the OpenTelemetry naming convention to the APM naming convention. For information about using OpenTelemetry with APM, see Configure OpenTelemetry Data Sources .
Click on a rule from the list to add it to the group.
Note
You cannot configure any actions since they are managed internally by the rule itself. Also, you cannot see what the actions are for an Oracle provided rule. - Adds the client IP address to a
span.
- If you select to create a Custom
span rule, you need to define one or
more actions for it. (A rule always contains at least
one action).
Before you can choose an Action, you have to select an Action target and Action type.
- The Action
target specifies the part of the span
on which the action operates. There are two
targets available:
- Entity: Target the entire span.
- Attribute: Target a single attribute.
- The Action type specifies the type. It is mainly used to group actions logically together. For example, string or numeric.
- Now select the appropriate action
from the Available Actions
drop-down.
Since actions have various contexts, the window will change accordingly with each action type. For more details about actions, see Working with Actions.
For more information about the available actions for Action target and Action type, see Available Actions.
-
Use the Add Action button to add more actions to the rule.
-
Use the action item menu at the top right of each Action to move it up or down in the rule ordering, or delete an action.
- The Action
target specifies the part of the span
on which the action operates. There are two
targets available:
-
Working with Actions
Actions that have a target of Attribute always
require you to provide the name of the attribute whose value will be used as input
to the action. The configuration window for each action refers to it as
Input Attribute
.
Some actions also require you to provide the name of an attribute in
which to store the result of the operation. This is referred to as the
Output Attribute
. If the attribute exists on the span, any
value it has at that time will be overwritten by the result of the action. If it
does not yet exist, it will be added to the span with the new value. You can use
this mechanism to replace the value of any attribute on the span (even the input
attribute) with the result of the action's operation.
Using this, you can feed the results of any action to a follow-up action (or even a rule within the same rule group).
You can also use it to store values into temporary attributes that can be used by follow up rules or actions, but without the need to keep them as part of the final result. Just make sure to remove the temporary attributes as a last step using the Remove Attribute action.
A Block action rejects the span and you cannot use it later on, even from another Rule Group.
Available Actions
Action Target | Action Type | Available Actions |
---|---|---|
Entity |
Passthrough Actions that control whether the span in its entirety is passed through or blocked. Blocked spans are rejected from the processing pipeline. Rejected spans will no longer be processed by APM, will not be stored anywhere, and will not be available for viewing later on in other APM tools or dashboards. |
Block: Blocks this span unconditionally. Any follow up actions and rules will not be applied to it and the span will no longer be available in APM OCI tools and products. Sample: Samples (accepts) spans based on a sample percentage. For example, a sample percentage of 25% will (randomly) accept 25% of all spans, and will block the rest. Spans blocked here are treated the same as spans that were blocked by the Block action. |
Attribute |
String Actions that allow various operations on the value of attribute which is a string. For example, changing the case, replacing a value or extracting a portion of the string. |
Append: Appends a static string to the value of an attribute using an optional separator string. Concatenate: Concatenates two attribute values. An optional separator string can be provided on which the values will be joined. The result is returned as a new attribute. Extract: Extracts a portion of an attribute's value based on a regular expression. The result is returned as a new attribute. The regular expression must contain a capturing group operator which identifies the portion of the string to capture. If the regular expression doesn't match, or there is no capture group, the action will produce no output. You can optionally provide a fallback value, which will be used as the output of the action in case the extract itself produced no result. Lowercase: Converts the value of an attribute to lower case. Search & Replace: Replaces one or more portions of an attribute's value using a regular expression. The amended string is returned as a new attribute. The regular expression may contain one capture group. Without a capture group, the full content of what the expression matches will be replaced. When a capture group is present, only the content that is captured by the group will be replaced. The optional match occurrence can be used if only a specific occurrence of a match must be replaced. If this is not set, or set to 0, all matched occurrences will be replaced. This is the default behavior. Trim: Trims any leading and trailing spaces from an attribute's value. Uppercase: Converts the value of an attribute to upper case. |
Attribute |
Numeric Actions that allow various mathematical operations on the
attribute values (provided they are numbers). Numeric actions
support both actual numbers and numbers represented as strings
in the attribute value, that is the following two are
equivalent:
Numeric actions can take as input either a string
representing the name of an attribute whose value to use in the
operation, or a number which will be used directly in the
computation.
All numeric actions have an option
where you can apply rounding to the output value. The rounding
can be one of the following:
|
Absolute: Returns the absolute value of the input value. Add: Sums two values. Bucketize: Maps a numeric attribute value into bucket names. If the attribute value falls within the range of a bucket, the bucket name will be returned. If no bucket matches, a fallback value will be returned. Divide: Divides the first by the second value. If the second value is 0, the result will be 0. Maximum: Returns the maximum of the two values. Minimum: Returns the minimum of the two values. Multiply: Multiplies the two values. Negate: Negates (changes the sign) of the value. Subtract: Subtracts the second from the first attribute. |
Attribute |
Generic Generic actions provide attribute level operations, such as removing an attribute from the span. |
Map attribute value: Takes the value of an attribute and maps it to a new value. Remove attribute: Removes the attribute from the span. If the attribute is an event name reference, Rename attribute: Renames the attribute. Set attribute: Sets an attribute to a static value. If the attribute doesn't exist, it is added, otherwise it is assigned the new value, overwriting the existing one. |
Examples of Action Configuration
This section provides examples of the configuration steps required for some individual actions.- Bucketize Example:
This action uses a mini-editor to set up a number of buckets and their threshold values. You can use the control on the right to create, update and remove buckets.
Buckets are automatically ordered based on their threshold values. A fallback value is used when a value is too large for the last bucket. The following image shows an example with five buckets and a fallback value:
- Map Example:
This action uses a mini-editor to set up map of input / output value pairs. Both values can be numbers or strings. Use the control on the right to create, update and/or remove a mapping.
The Export button allows you to download any mappings to a file (in CSV format) to your computer.The Import button allows you to populate the mappings from a CSV file on your computer. This is particularly useful if you have a large number of mappings.
Span Enrichment Rules Validation
Span enrichment configurations can be complex to write and it can sometimes be difficult to know beforehand if a rule or rule group will work as expected. To help with this, the Span Enrichment Configuration Editor provides an option to test a rule or rule group on a sample span before saving it.
- Select a group, and then click on Edit to open the Edit span rule group panel. Another way is to create a new group and the Edit panel get displayed.
- Click through to the Configure rule actions step.
- Click Test to test the entire rule group. Alternatively, to test an individual rule, select the rule and click Test rule.
Note
Test rule is not available when the rule group contains only one rule. - The Test span rule group panel is displayed.
The rules to test will be shown with a checkbox next to them. You can toggle those as desired.
- Enter a sample span in the Span sample text box.
- Click Test span to perform a test of the selected rules on the given span.
Figure 8-1 Span Enrichment Rule Validation
The test run will report whether the span is matching the configured span filters or not.
If the filters match, then the rules will be applied to the span and the resulting differences (if any) will be displayed in the table below. Any attributes that were added as a result of the rule will be highlighted in green. Attributes that were removed will be highlighted in red. Modified attributes will be highlighted in yellow.
Trace Explorer: To get accurate samples of spans from your traffic, click Trace Explorer located at the top right to open Trace Explorer and see the actual spans that have been processed. You can select a span, open the context menu on the right (the three vertical dots) and click Copy root span data.
A popup is displayed with the contents of the span. You can copy the span data and use it during testing.
Another option is to paste your own spans or enter them manually, as long as they conform to the span format as reported by Trace Explorer.
Span Property Aliases
Few of the actual span properties are referred to by a different name when used in Span Enrichment or Span Filters features. Other than the visual difference, this has no impact on how you setup your rules and filters. For example, you should always use the alias when referring to the given property.
Here are the properties that currently have an alias:
Span Property | Span Enrichment Alias | Remarks |
---|---|---|
td-micros |
SpanDuration |
td-micros is in microseconds, while SpanDuration is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
|
name |
OperationName |
The name of the span represents the operation performed |
ts-micros |
StartTime |
ts-micros is in microseconds, while StartTime is in milliseconds. Time unit conversion is applied automatically by Span Enrichment and Span Filters.
|
id |
SpanId |
|
parent-id |
ParentId |
|
trace-id |
TraceId |
Available Metrics: oracle_apm_rum
The RUM metrics are available after an APM Browser agent is configured and uploads user experience-related data. You don't need to enable monitoring on the APM domain to get these metrics.
RUM metrics include the following dimensions:
- Generic Dimensions
- ApdexLevel: The level of user satisfaction. For example, Satisfied or Frustrated.
- ApmrumType: The type of monitored activity. For example, Page, AJAX call or Script Error.
- ApmrumPageUpdateType: The subtype of the ApmrumType dimension, which is related to user activity. For example, Page Load, Full Update or Click.
- User Session-Based Dimensions
- BrowserName: The name of the web browser derived from the user agent. For example, Chrome.
- DeviceType: The type of device used by the user. For example, Personal Computer.
- OsFamily: The Operating System family derived from the user agent. For example, Linux.
- ResourceId: The OCID of the APM domain.
- Type: The type of metric. For example, Gauge or Counter.
- WebApplicationName: The name of the web application as specified in instrumentation. If no value is specified for this dimension, then "Default WebApp" is assigned.
When building metric queries for the oracle_apm_rum namespace, you must include the general dimension,
MetricGroupName = "No Group"
to ensure correctness of data.
Similarly, the WebApplicationName
dimension with the value
All Web Applications
or the name of the required web
application must be added. For information on adding dimensions, see Build Application Performance Monitoring Metric Queries.
The following table lists RUM metrics:
Metric | Description | Unit |
---|---|---|
ActiveSessionsPerMin |
The number of active sessions per minute (in the context of web applications). |
– |
AjaxCalls |
The number of AJAX calls observed. |
– |
AjaxDownloadTime |
The time taken to download a response. |
Milliseconds |
AjaxErrors |
The number of AJAX errors observed. |
– |
AjaxFirstByteTime |
The time taken to receive the first byte of an AJAX call response after the request is sent to the server. This is mainly impacted by network latency and server response time. |
Milliseconds |
AjaxInitTime |
The time taken to initialize an AJAX request inside
the browser, from the creation of |
Milliseconds |
AjaxResponseTime |
The time taken to process the AJAX call response, which is the sum of AjaxInitTime, AjaxFirstByteTime, and AjaxDownloadTime. |
Milliseconds |
ApdexScore |
The Apdex value between 0 and 1. |
– |
ConnectCount |
The number of connections established. |
– |
ConnectTime |
The average time taken to establish server connections. |
Milliseconds |
DNSLookups |
The number of DNS lookups. |
– |
DNSTime |
The average time taken to perform DNS lookups. |
Milliseconds |
FrustratedPageViews |
The number of page views with a 'frustrating' performance. |
– |
PageClicks |
The number of clicks on a page. |
– |
PageDownloadTime |
The time from the browser sending a request for the page URL till the full content is received. |
Milliseconds |
PageFirstByteTime |
The time from the browser sending the request for the page URL till the first byte of response from the server is sent for the page URL. This is mainly impacted by server response time and network latency. |
Milliseconds |
PageInitTime |
The time from the browser receiving the navigation trigger till the request for the page URL is started, which includes connection setup time. This time is calculated for Page Loads and Page Updates. |
Milliseconds |
PageInteractiveTime |
The time taken for the page to become interactive. |
Milliseconds |
PageRenderTime |
The time from the browser receiving full HTML till the content is rendered. |
Milliseconds |
PageResponseTime |
The time taken from the beginning of navigation till the loading of script is complete. This is the sum of PageInitTime, PageFirstByteTime, PageDownloadTime, and PageRenderTime. |
Milliseconds |
PageViews |
The number of page views or page updates. |
– |
RedirectTime |
The average time spent in handling HTTP redirects. |
Milliseconds |
SatisfiedPageViews |
The number of page views with a 'satisfying' performance. |
– |
ScriptErrors |
The number of JavaScript errors observed. |
– |
SSLTime |
The average time taken to establish secure server connections (as part of ConnectTime). |
Milliseconds |
ToleratingPageViews |
The number of page views with a 'tolerable' performance. |
– |
Available Metrics: oracle_apm_synthetics
The Synthetic Monitoring metrics are emitted as soon as the monitor is created, although there may be a delay of a few minutes before the metrics are displayed initially. You don't need to enable monitoring on the APM domain to get these metrics.
Synthetic Monitoring metrics include the following dimensions:
- CustomMarker: Indicates the custom target name using which an operation is added to a script.
- DNSConfigType: Indicates the DNS configuration type. The supported values are DNS_SERVER_CONFIG, DNS_TRACE_CONFIG and DNSSEC_CONFIG.
- ErrorCategory: The category of the error, if an error occurs when the script is executed.
- Genre: Indicates that the metric is a Synthetic Monitoring metric. The default value is Synthetics.
- Host: The host of the network timing metrics. The network timing metrics such as SSLTime and DNSTime are averaged by this host value.
- IsAPMAgentMonitored: Indicates if the application is also being monitored by another Application Performance Monitoring agent.
- IsDnsDataAvailable: Indicates if DNS data is available.
- IsHarAvailable: Indicates if the HAR
.zip
file is available. - IsLogAvailable: Indicates if the log
.zip
file is available. - IsNetworkDataAvailable: Indicates if the network
.zip
file is available. - IsRetryExecution: Indicates if monitor execution has been tried more than once.
- IsScreenshotAvailable: Indicates if screenshots are available.
- MaintenanceWindowActive: Indicates if a maintenance window is currently active (1 for active, 0 for inactive).
- MonitorId: The OCID of the monitor created in Synthetic Monitoring.
- MonitorName: The name assigned to the monitor.
- MonitorType: The type of monitor.
- RequestType: The type of request. The values are CSS, Image, JavaScript, AJAX, or Others.
- ResourceId: The OCID of the APM domain in which the monitor is created.
- SqlState: The database state function. It returns a 5-character data type of CHARACTER with a default value of '00000' .
- Target: The base URL specified in the uploaded script for the Scripted Browser or Scripted REST monitor type or specified in the user interface when creating a Browser or REST monitor type.
- UserAgent: The type of agent used. For the Browser and Scripted Browser monitor types, Chrome is the default value. For the REST and Scripted REST monitor types, the respective monitor type is the default value.
- VantagePoint: The vantage point on which the monitor is running.
- VantagePointDisplayName: The display name of the vantage point on which the monitor is running.
The following table lists the common Synthetic Monitoring metrics for all monitor types:
Metric Name | Description | Unit |
---|---|---|
Availability |
The availability of the monitor. |
1 for success, 0 for failure |
AverageBytesPerRequest |
The average number of bytes loaded per request for this request type. |
– |
AverageLoadTimePerRequest |
The average time taken to load the resources of this request type. |
Milliseconds |
CloseTime |
Time taken to close the database connection. |
Milliseconds |
CloudWalletExpiry |
Number of days remaining from the current day for the expiration of the cloud wallet. Only available if the connection type is Cloud Wallet. |
Days |
ConnectCount |
The number of connections established. |
– |
ConnectTime |
The average time taken to establish server connections. For SQL monitor, it's the time taken to create the database connection. For FTP monitor, it's the time taken to create FTP connection. |
Milliseconds |
CustomMetric |
Records the custom markers added to the scripts. |
Milliseconds |
DNSLookups |
The number of DNS lookups. |
– |
DNSTime |
The average time taken to perform DNS lookups. |
Milliseconds |
ExecutionTime |
Time taken to execute SQL query. |
Milliseconds |
Failure |
Indicates if the monitor execution failed. |
0 for success, 1 for failure |
FetchedRowCount |
Total number of rows fetched. |
- |
FinalQueryTime | The resolution time of the very last query in the trace. Applicable to DNS Trace monitor type. | Milliseconds |
HTTP4xxFailureCount |
The total number of 4xx request failures during monitor execution. |
– |
HTTP5xxFailureCount |
The total number of 5xx request failures during monitor execution. |
– |
Latency |
The average of the round-trip packet time. |
Milliseconds |
LatencyDeviation |
The standard deviation of latency. |
Milliseconds |
LoginTime |
The time taken for authentication to the FTP server. Applicable to FTP monitors. |
Milliseconds |
MonitorExecutionTime |
The total time taken to execute the monitor. |
Milliseconds |
MonitorLoadTime |
The total time taken to execute the monitor, excluding the pauses given in the script. |
Milliseconds |
OperationTime |
The time taken for the selected FTP operation (download/upload/list). |
Milliseconds |
PacketLossPercentage |
The percentage of probe packets lost. |
Percentage |
PrepareTime |
The time taken to create PreparedStatement object containing the pre-compiled SQL statement. |
Milliseconds |
RequestCount |
The total number of requests for this request type. |
– |
ResolutionTime |
The time it takes to query a specific name server for the given domain. Applicable to DNS Server monitor type. |
Milliseconds |
SSLTime |
The average time taken to establish secure server connections (as part of ConnectTime). |
Milliseconds |
Success |
Indicates if the monitor execution was successful. |
1 for success, 0 for failure |
TotalQueries | The number of queries used to produce the trace. Applicable to DNS Trace monitor type. | - |
TotalRequestFailures |
The total number of requests that failed to receive a response. |
– |
TotalSteps |
The total number of clicks or enter keys that are captured during monitor execution. |
– |
TotalTime |
The total time of ConnectTime, LoginTime and OperationTime. Applicable to FTP monitor. |
Milliseconds |
The following table lists the Synthetic Monitoring metric for the REST monitor type:
Metric Name | Description | Unit |
---|---|---|
CertificateExpiry |
The number of days left before the certificate expires. |
Days |
Available Metrics: oracle_apm_monitoring for APM Java Agent
The generic Application Performance Monitoring Java agent
metrics include default and custom (user-defined) metrics uploaded by the APM Java agent
in the oracle_apm_monitoring
namespace. You don't need to enable
monitoring on the APM domain to get these metrics.
This section provides information on the default Application Performance Monitoring metrics. For information on how to create custom metrics, see Custom Metrics.
Dimensions
Default Application Performance Monitoring metrics include the following dimensions:
- Application Server Dimensions
- Appserver: The flag to indicate if an application
server has been discovered. When the application server type is
java-jmx
orjava-no-jmx
, then this dimension is set to false. - AppserverDisplayPort: The port number of the application server. The default is the lowest HTTPS port, if available, or the lowest HTTP port.
- AppserverDomainName: The domain name of the application server.
- AppserverEngine: The name of the application server (J2EE container) used.
AppserverEngine
is set if using only Spring Boot or Dropwizard. For example: Apache Tomcat 8.5.32, Jetty 9.4.11.v20180605 or Undertow 1.4.25.Final. - AppserverId: The hash of significant application server resources, which indicates if there is resource change in the application server over time.
- AppserverName: The name of the application server or application framework.
- AppserverPorts: The port numbers of the application server. Usually, there is only one port number for the Oracle WebLogic server and multiple for the Apache Tomcat server. In case of multiple port numbers, the values are comma separated.
- AppserverServerName: The name of the application server instance.
- AppserverType: The type of application server. For example, weblogic or tomcat.
- AppserverVersion: The version of the application server.
- Appserver: The flag to indicate if an application
server has been discovered. When the application server type is
- Generic Dimensions
- ApmVersion: The version of the Application Performance Monitoring agent.
- DisplayName: The display name of the application server.
- ServiceInstanceId: The observer ID of the agent.
- ServiceName: The name you've assigned to the service.
- OraPackagedApp: The Oracle Packaged Application being used. For example: E-Business Suite (EBS), JD Edwards (JDE), Business Intelligence (OBIEE), Peoplesoft (PSFT) or Siebel CRM (SIEBEL).
- CreatedBy: Source of the trace/span creation. For example: oracle-apm-java-agent or oracle-apm-java-tracer.
- Host Dimensions
- HostAddress: The IPv4 address resolved from the host name.
- HostAddresses: The list of IPv4 addresses in the network interfaces of the host.
- Hostname: The name of the host.
- HostnameCanonical: The canonical name (FQDN) of the host.
- Hostnames: The list of host names in the network interfaces of the host.
- Java Virtual Machine (JVM) Dimensions
- GCCollectors: The garbage collection strategies used for minor and major garbage collections by JVM. For example: G1 young generation or G1 old generation.
- ProcessId: The process ID of the application server.
- VmName: The name of the virtual machine.
- VmVendor: The vendor of the virtual machine.
- VmVersion: The version of the virtual machine.
- WorkingDirectory: The working directory of the application server.
- Kubernetes Dimensions
- KubernetesNamespace: The namespace in the Kubernetes cluster the pod is running in.
- KubernetesNodeName: The name of the Kubernetes node the pod is running in.
- KubernetesPodAnnotations: The annotations you've assigned to the pod, if any.
- KubernetesPodLabels: The labels you've assigned to the pod, if any.
- KubernetesPodName: The name of the pod (container) in the Kubernetes cluster.
- Oracle Cloud Infrastructure
Dimensions
- OciAvailabilityDomain: The Oracle Cloud Infrastructure availability domain the compute instance is running in.
- OciCompartmentId: The OCID of the compartment in which the compute instance resides.
- OciComputeShape: The shape of the compute instance.
- OciDisplayName: The display name of the compute instance or Kubernetes node.
- OciFaultDomain: The name of the Oracle Cloud Infrastructure fault domain the compute instance is running in.
- OciInstanceId: The OCID of the compute instance.
- OciRegion: The Oracle Cloud Infrastructure region that contains the availability domain the compute instance is running in.
- Oracle Cloud Infrastructure Container
Engine for Kubernetes (OKE) Dimensions
- OkeClusterId: The OCID of the OKE cluster.
- OkeClusterLabel: The cluster label, which are the last 11 characters of the OCID of the OKE cluster.
- OkeCompartmentName: The Oracle Cloud Infrastructure compartment where the OKE cluster resides.
- OkeKubernetesVersion: The Kubernetes version.
- OkeNodePoolId: The OCID of the OKE node pool.
- OkeNodepoolLabel: The node pool label, which are the last 11 characters of the OCID of the OKE node pool.
- OkeTenancyId: The OCID of the OKE tenancy.
- Oracle E-Business Suite Dimensions
- EbsAkRegionAppId: The application ID of a region.
- EbsAkRegionCode: The ID of a region.
- EbsClassName: The full package name of the page that is rendered.
- EbsFunctionId: The ID of the function that is ran, as defined in the Oracle E-Business Suite Functions form.
- EbsOAFunc: The function name that is passed in the URL.
- EbsOAHP: The parameter used to change the menu context to the new Home Page and selected function.
- EbsOAPB: The function name that represents the product branding text.
- EbsOASF: The function that is selected in the current menu context.
- EbsRegionAppId: The application ID of a region.
- EbsRegionClass: The full package name of the region that is rendered.
- EbsRegionCode: The ID of a region.
- EbsRespAppId: This represents the ID of the application.
- EbsRespId: This represent the ID of the Responsibility
- Operating System Dimensions
- OsAvailableProcessors: The number of processors available to the JVM.
- OsName: The name of the Operating System.
- OsVersion: The version of the Operating System.
- Siebel Resource Dimensions
- SiebelResourceType: The category of the Siebel resource.
- SiebelResourceName: The name of the Siebel resource.
- SiebelResourceOperation:The operation invoked on the Siebel resource.
Metrics
The following table lists the default Application Performance Monitoring metrics in the
oracle_apm_monitoring
namespace:
Metric | Description | Unit |
---|---|---|
GcAfterOldCommitted |
The heap committed after old generation garbage collection. |
Bytes |
GcAfterOldUsed |
The heap used after old generation garbage collection. |
Bytes |
GcAfterYoungCommitted |
The heap committed after young generation garbage collection. |
Bytes |
GcAfterYoungUsed |
The heap used after young generation garbage collection. |
Bytes |
GcOldTotalCount |
The number of old generation garbage collections since the JVM started. |
– |
GcOldTotalTime |
The total time used for old generation garbage collection since the JVM started. |
Milliseconds |
GcYoungTotalCount |
The number of young generation garbage collections since the JVM started. |
– |
GcYoungTotalTime |
The total time used for young generation garbage collection since the JVM started. |
Milliseconds |
HeapCommitted |
The current system memory allocated for heap usage. |
Bytes |
HeapUsed |
The current heap memory usage. This value is less than or equal to the HeapCommitted value. |
Bytes |
NonHeapCommitted |
The current system memory allocated for non-heap usage. |
Bytes |
NonHeapUsed |
The current non-heap memory usage. This value is less than or equal to the NonHeapCommitted value. |
Bytes |
ProcessCpuLoad |
The current CPU load of the JVM process. |
– |
ProcessCpuTime |
The total CPU time of the JVM process. |
Nanoseconds |
SystemCpuLoad |
The current CPU load of the system. |
– |
SystemPhysicalMemoryFree |
The free memory remaining in the system. |
Bytes |
SystemSwapFree |
The free swap disk space remaining in the system. |
Bytes |
SystemVirtualMemoryCommitted |
The committed virtual memory of the JVM process. |
Bytes |
ThreadBlockedCount |
The number of threads in "BLOCKED" state in the JVM. Note: BLOCKED is the state in which a thread is blocked waiting for a monitor lock. For more information, see Thread States: BLOCKED in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification. |
– |
ThreadCount |
The total number of threads in the JVM. |
– |
ThreadDeadlockCount |
The number of deadlocked threads in the JVM. Note: DEADLOCK occurs when two or more threads form a cyclic dependency with each other. |
– |
ThreadTimedWaitingCount |
The number of threads in "TIMED_WAITING" state in the JVM. Note: TIMED_WAITING is the state in which a thread is waiting for another thread to perform a particular action for a specified waiting time. For more information, see Thread State: TIMED_WAITING in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification. |
– |
ThreadWaitingCount |
The number of threads in "WAITING" state in the JVM. Note: WAITING is the state in which a thread is waiting indefinitely for another thread to perform a particular action. For more information, see Thread State: WAITING in Java® Platform, Standard Edition & Java Development Kit Version 16 API Specification. |
– |
TotalTraceCount |
The total number of traces evaluated. |
– |
TotalSpanCount |
The total number of spans evaluated. |
– |
TotalTraceSampledCount |
The total number of traces sampled. This value will increment at the same pace as
|
– |
TotalSpanSampledCount |
The total number of spans sampled. This value will increment at the same pace as
|
– |
Uptime |
The time that has elapsed since the JVM started. |
Milliseconds |
WeblogicJDBCActiveConnectionsAverageCount | The average number of active connections in this instance of the data source. | – |
WeblogicJDBCActiveConnectionsCurrentCount | The number of connections currently in use by applications. | – |
WeblogicJDBCActiveConnectionsHighCount | The highest number of active database connections in this instance of the data source since the data source was instantiated. | – |
WeblogicJMSConnectionsCurrentCount | The current number of connections to this WebLogic Server. | – |
WeblogicJMSConnectionsHighCount | The peak number of connections to this WebLogic Server since the last reset. | – |
WeblogicJMSConnectionsTotalCount | The total number of connections made to this WebLogic Server since the last reset. | – |
WeblogicJMSDestinationsBytesCurrentCount | The current number of bytes stored in the destination. | Bytes |
WeblogicJMSDestinationsBytesHighCount | The peak number of bytes stored in the destination since the last reset. | Bytes |
WeblogicJMSDestinationsBytesReceivedCount | The number of bytes received in this destination since the last reset. | Bytes |
WeblogicJMSDestinationsMessagesHighCount | The peak number of messages in the destination since the last reset. | |
WeblogicJMSDestinationsMessagesPendingCount | The number of pending messages in the destination. | |
WeblogicJMSDestinationsMessagesReceivedCount | The number of messages received in this destination since the last reset. | |
WeblogicJMSServersCurrentCount | The current number of JMS servers that are deployed on this WebLogic Server instance. | – |
WeblogicJMSServersDestinationsCurrentCount | The current number of destinations for this JMS server. | |
WeblogicJMSServersDestinationsHighCount | The peak number of destinations on this JMS server since the last reset. | |
WeblogicJMSServersDestinationsTotalCount | The number of destinations instantiated on this JMS server since the last reset. | |
WeblogicJMSServersHighCount | The peak number of JMS servers that were deployed on this WebLogic Server instance since the server was started. | – |
WeblogicJMSServersTotalCount | The number of JMS servers that were deployed on this WebLogic Server instance since the server was started. | – |
WeblogicJTAActiveTransactionsTotalCount | The number of active transactions on the server. |
– |
WeblogicJTATransactionAbandonedTotalCount | The total number of transactions that were committed. | – |
WeblogicJTATransactionCommittedTotalCount | The total number of transactions committed since the server was started. |
– |
WeblogicJTATransactionHeuristicsTotalCount | The number of transactions that completed with a heuristic status since the server was started. |
– |
WeblogicJTATransactionRolledBackAppTotalCount | The number of transactions that were rolled back due to an application error. |
– |
WeblogicJTATransactionRolledBackResourceTotalCount | The number of transactions that were rolled back due to a resource error. | – |
WeblogicJTATransactionRolledBackSystemTotalCount | The number of transactions that were rolled back due to an internal system error. | – |
WeblogicJTATransactionRolledBackTimeoutTotalCount | The number of transactions that were rolled back due to a timeout expiration. | – |
WeblogicJVMHeapFreeCurrent | The current amount of memory that is available in the JVM heap. | Bytes |
WeblogicJVMHeapFreePercent | The percentage of the JVM heap that is free. | Percentage |
WeblogicJVMHeapSizeCurrent | The current size of the JVM heap. | Bytes |
WeblogicJVMHeapSizeMax | The maximum size of the JVM heap. | Bytes |
WeblogicServerOpenSocketsCurrentCount | The current number of sockets registered for socket muxing on this server. |
– |
WeblogicServerState |
The current state of the server as an integer. It can
be:
|
– |
WeblogicServerHealthState |
The current health state of the server as an integer. It can be:
For details, see the 5 health values defined in Class HealthState from the Weblogic documentation. |
- |
WeblogicThreadPoolCompletedRequestCount |
The number of completed requests in the priority queue. |
– |
WeblogicThreadPoolExecuteThreadIdleCount | The number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives. |
– |
WeblogicThreadPoolExecuteThreadTotalCount | The total number of threads in the pool. |
– |
WeblogicThreadPoolHoggingThreadCount | The threads that are currently being held by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary. |
– |
WeblogicThreadPoolPendingUserRequestCount |
The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests. |
– |
WeblogicThreadPoolQueueLength | The number of pending requests in the priority queue. This is the total of internal system requests and user requests. |
– |
WeblogicThreadPoolStandbyCount | The number of threads in the standby pool. Threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed. |
– |
WeblogicThreadPoolStuckCount | The number of stuck threads in the thread pool. |
– |
WeblogicThreadPoolThroughput | The mean number of requests completed per second. | Request per second |
WeblogicWebAppOpenSessionsCurrentCount | The count of the current total number of open sessions in this module. | – |
Available Metrics: oracle_apm_monitoring for APM Dotnet Agent
The Application Performance Monitoring Dotnet agent metrics
include default metrics uploaded by the APM Dotnet agent in the
oracle_apm_monitoring
metric namespace with
oracle_apm_dotnet_agent
resource group. You don't need to enable
monitoring on the APM domain to get these metrics available.
Dimensions
The APM Dotnet agent metrics include the following dimensions:
- Generic Dimensions
- ApmVersion: The version of the Application Performance Monitoring Dotnet agent. For example:
1.0.0
. - CreatedBy: The agent type. In this case, it's
oracle-apm-dotnet-agent
. - ServiceName: The name you have assigned to the service.
- ApmVersion: The version of the Application Performance Monitoring Dotnet agent. For example:
- Host Dimensions
- Hostname: The name of the host where the APM Dotnet agent is running.
- CLR Runtime Dimensions
- ProcessName: : The process name where the APM Dotnet agent is running. For example: w3wp.exe.
- OpenTelemetry Dimensions
- Telemetry-auto-version: The OpenTelemetry version of
the APM Dotnet agent. For example:
0.6.0
. - Telemetry-sdk-language: The OpenTelemetry SDK
language. For example:
dotnet
. - Telemetry-sdk-name: The OpenTelemetry SDK name. For
example:
opentelemetry
. - Telemetry-sdk-version: The OpenTelemetry SDK version.
For example:
1.4.0.687
.
- Telemetry-auto-version: The OpenTelemetry version of
the APM Dotnet agent. For example:
Metrics
- Application Metrics
The following table lists the Application Performance Monitoring Dotnet agent metrics in the
oracle_apm_monitoring
namespace withoracle_apm_dotnet_agent
resource group:Metric Description Unit http.server.duration The duration of the inbound HTTP request. It's collected from ASP.NET core component.
Milliseconds
http.client.duration The duration of the outbound HTTP request. It's collected from HTTP client component.
Milliseconds
- CLR Runtime Metrics
- Process Metrics
Available Metrics: oracle_apm_agent for APM Java Agent
The Application Performance Monitoring Java agent emits
agent health related metrics to oracle_apm_agent
namespace. You don't
need to enable monitoring on the APM domain to get these metrics.
This section provides information on the Application Performance Monitoring agent health metrics. For information on how to create custom metrics, see Custom Metrics.
Dimensions
Default Application Performance Monitoring metrics include the following dimensions:
- Application Server Dimensions
- Appserver: The flag to indicate if an application
server has been discovered. When the application server type is
java-jmx
orjava-no-jmx
, then this dimension is set to false. - AppserverDisplayPort: The port number of the application server. The default is the lowest HTTPS port, if available, or the lowest HTTP port.
- AppserverDomainName: The domain name of the application server.
- AppserverId: The hash of significant application server resources, which indicates if there is resource change in the application server over time.
- AppserverName: The name of the application server.
- AppserverPorts: The port numbers of the application server. Usually, there is only one port number for the Oracle WebLogic server and multiple for the Apache Tomcat server. In case of multiple port numbers, the values are comma separated.
- AppserverServerName: The name of the application server instance.
- AppserverType: The type of application server. For example, weblogic or tomcat.
- AppserverVersion: The version of the application server software.
- Appserver: The flag to indicate if an application
server has been discovered. When the application server type is
- Generic Dimensions
- ApmVersion: The version of the Application Performance Monitoring agent.
- DisplayName: The display name of the application server.
- ServiceInstanceId: The observer ID of the agent.
- ServiceName: The name you've assigned to the service.
- Host Dimensions
- HostAddress: The IPv4 address resolved from the host name.
- HostAddresses: The list of IPv4 addresses in the network interfaces of the host.
- Hostname: The name of the host.
- HostnameCanonical: The canonical name (FQDN) of the host.
- Hostnames: The list of host names in the network interfaces of the host.
- Java Virtual Machine (JVM) Dimensions
- ProcessId: The process ID of the application server.
- VmName: The name of the virtual machine.
- VmVendor: The vendor of the virtual machine.
- VmVersion: The version of the virtual machine.
- WorkingDirectory: The working directory of the application server.
- Kubernetes Dimensions
- KubernetesNamespace: The namespace in the Kubernetes cluster the pod is running in.
- KubernetesNodeName: The name of the Kubernetes node the pod is running in.
- KubernetesPodAnnotations: The annotations you've assigned to the pod, if any.
- KubernetesPodLabels: The labels you've assigned to the pod, if any.
- KubernetesPodName: The name of the pod (container) in the Kubernetes cluster.
- Oracle Cloud Infrastructure
Dimensions
- OciAvailabilityDomain: The Oracle Cloud Infrastructure availability domain the compute instance is running in.
- OciCompartmentId: The OCID of the compartment in which the compute instance resides.
- OciComputeShape: The shape of the compute instance.
- OciDisplayName: The display name of the compute instance or Kubernetes node.
- OciFaultDomain: The name of the Oracle Cloud Infrastructure fault domain the compute instance is running in.
- OciInstanceId: The OCID of the compute instance.
- OciRegion: The Oracle Cloud Infrastructure region that contains the availability domain the compute instance is running in.
- Oracle Cloud Infrastructure Container
Engine for Kubernetes (OKE) Dimensions
- OkeClusterId: The OCID of the OKE cluster.
- OkeClusterLabel: The cluster label, which are the last 11 characters of the OCID of the OKE cluster.
- OkeCompartmentName: The Oracle Cloud Infrastructure compartment where the OKE cluster resides.
- OkeKubernetesVersion: The Kubernetes version.
- OkeNodePoolId: The OCID of the OKE node pool.
- OkeNodepoolLabel: The node pool label, which are the last 11 characters of the OCID of the OKE node pool.
- OkeTenancyId: The OCID of the OKE tenancy.
- Operating System Dimensions
- OsAvailableProcessors: The number of processors available to the JVM.
- OsName: The name of the Operating System.
- OsVersion: The version of the Operating System.
- Agent Health Dimensions
- Probe: The name of the probe.
- DataType: The type of data. For example: span or metric.
- Circuit Breaker Dimensions
- PerformanceFactorName: The name of the performance factor.
- PerformanceFactorType: The type of the performance factor.
- Errors Dimensions
- Cause: The cause of upload error.
- Type: The type of the error observed. For example: severe or warning.
Metrics
oracle_apm_agent
namespace:
Starting APM Java agent version 1.12,
DataQueueSize
metric is no longer available. Use DataQueueUsageMaxPercent
metric instead.
Metric | Description | Unit |
---|---|---|
BrowserAgentInjectionCount | The number of times the APM browser agent is injected. | – |
CircuitBreakerActivations | Indicates activation of a circuit breaker performance
factor.
Note: This metric is emitted only when a performance factor is activated. |
- |
CircuitBreakerProbeSuspensions | Indicates suspension of a probe due to circuit
breaker.
Note: This metric is emitted only when a probe is suspended. |
- |
DataRejectCount | The number of data items rejected due to queue being full. | - |
DataQueueUsageMaxPercent | Indicates the maximum size of the data queue with respect to its capacity.
This metric is available starting with APM Java agent version 1.12. |
Percentage |
DataUploaded | The size of data uploaded. | Bytes |
DataUploadTime | The time spent in uploading data. | Miliseconds |
ErrorCount | The number of errors observed in the APM agent. | - |
SpanStartCount | The number of spans started by a probe. | - |
SpanFinishCount | The number of spans finished by a probe. | - |
SpanTimedOutCount | The number of spans that timed out. | - |
TotalSpanCount | The total number of spans evaluated. | - |
TotalSpanSampledCount | The total number of spans sampled.
This value will
increment at the same pace as the
|
- |
TotalTraceCount | The total number of traces evaluated. | - |
TotalTraceSampledCount | The total number of traces sampled.
This value
will increment at the same pace as the
|
- |
UploadErrorCount | The number of upload errors observed by the APM agent. | - |
Available Metrics: oci_apm
When spans or metrics are ingested by Application Performance Monitoring, metrics reflecting status of ingest are automatically emitted to the oci_apm namespace
.
This section provides information on the Application Performance Monitoring metrics.
Dimensions
The metrics include the following dimensions:
- Generic Dimension
- OracleApmType: Indicates the type of metric such as counter or gauge.
- ResourceId: The OCID of the APM domain for which data is ingested.
- PayloadRejections Dimensions
- ObservationType: The type of data that was rejected, such as metric, private-span, public-span.
- RejectionCause: The reason for rejecting the data such as
MISSING_DATA_KEY
,MISSING_DATA_FORMAT
,INVALID_OBSERVATION_TYPE
,OVERSIZED_PAYLOAD
,PAYLOAD_THROTTLED
,INTERNAL_SERVER_ERROR
,INVALID_DATA_KEY
and more.
Metrics
The following table lists the default Application Performance Monitoring metrics in the oci_apm
namespace:
Metric | Description | Unit |
---|---|---|
PayloadRejections | The number of observations of different types that cannot be processed due to reasons details in the RejectionCause dimension.
|
– |
SpanIngestions | The number of spans ingested during the specified period of time. The apmDomainId dimension can be used to report span counts for specific domains.
|
- |
SyntheticMonitorRuns | The number of monitor runs executed during the specified period of time. The apmDomainId dimension can be used to report on monitor executions for specific domains.
|
- |
Using the Console
Use the console to:
Build Application Performance Monitoring Metric Queries
You can build Application Performance Monitoring metric queries in Metrics Explorer and monitor your applications by various dimensions. For example, you can build a query using the oracle_apm_rum namespace, ScriptErrors metric, WebApplicationName dimension to monitor the script errors in your application for a specified period of time. You can also add another dimension, BrowserName, to determine if the script errors are browser-related.
- Sign in to the Oracle Cloud Infrastructure console.
- Open the navigation menu, click Observability &
Management. Under Monitoring, click
Metrics Explorer.
The Metrics Explorer page displays an empty chart with fields to create a query.
- Select a time period using the Start time and End time or Quick Selects fields on the top of the page.
- Scroll down and fill in the following fields for your query:
- Compartment: Ensure that the compartment in which your APM domain resides is selected.
- Metrics namespace: Select one of the
following Application Performance Monitoring
namespaces:
- oracle_apm_rum
- oracle_apm_synthetics
- oracle_apm_monitoring
- Metrics name: Select a metric within the selected Metric Namespace.
- Interval: Select an aggregation window.
- Statistic: Select an aggregation function.
- Metric dimensions: Select a dimension in
the Dimension name field and a value for the
specified dimension in the Dimension value field to
filter the metric data. Optionally, you can add an additional
dimension.
Note
In addition to the dimensions you want to select, all the metric queries for the oracle_apm_rum namespace must include the following dimensions to ensure correctness of data:MetricGroupName = "No Group"
WebApplicationName = All Web Applications
orWebApplicationName = <name of the individual web application>
- Click Update Chart.
The chart will be updated to display the metrics that have been requested.
For more information on Metrics Explorer and how to build a metric query, see Building Metric Queries.
Create an Alarm for Application Performance Monitoring Metrics
You can create an alarm using the Oracle Cloud Infrastructure Monitoring service to be notified if an Application Performance Monitoring metric breaches the specified threshold. For example, if you've built a query to monitor the JavaScript errors in browsers, you can create an alarm to be notified if the errors for any browser go over 5%.
After you create an alarm for an Application Performance Monitoring metric, you can go to the Application Performance Monitoring Home page and monitor firing alarms that require your attention in the Alarms widget and click an alarm to navigate to the Alarms Definition page.
For information on how to create a threshold alarm for Application Performance Monitoring metrics, see Using the Console to Create an Alarm.
Using the API
Use the following APIs for monitoring:
- Monitoring API for metrics and alarms
- Notifications API for notifications (used with alarms)