Oracle Cloud Infrastructure Documentation

Configure Management Agent to Collect Metrics using Prometheus Node Exporter

To be able to have the Management Agent to scrape the Prometheus metrics, a configuration file requires to be created.

The configuration file needs to be placed under the agent's detection directory which location is /opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/

Once the agent detects the existence of the configuration file in the detection direction, the scraping of Prometheus metrics take place.

It's important to use a naming convention for the configuration files: The base name of the file should be indicative of the metrics scraped, and the extension of the file should be .properties.

Each line of the file is a name/value pair with the format: propertyName=propertyValue.

The presence of a file in the detection directory indicates the current settings for the monitoring and scraping. You need to keep that file in the directory, unchanged and as long as the monitoring is needed.

Create a Configuration File

You need to create a configuration file to scrape all metrics exposed by Node Exporter to poc_prometheus namespace, specifying nodeName as a dimension on every metric datapoint. 

$ cat /opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/nodeVM1-ol7.properties
# property file scraping all metrics of the node_exporter that has been set up on 10.20.30.40
url=http://10.20.30.40:9100/metrics
namespace=poc_prometheus
nodeName=nodeVM1-ol7
metricDimensions=nodeName
allowMetrics=*
compartmentId=ocidl.compartment.ocl..aaaaaaaa...kovd6xlq
In this example, review the following:
  • The url uses IP address: 10.20.30.40. You need to replace it with the IP address or the resolvable hostname where the Node Exporter is running.
  • The compartmentId uses OCID: ocidl.compartment.ocl..aaaaaaaa...kovd6xlq. You need to replace it with the OCID of the compartment where the Prometheus metrics will get posted.

The lines starting with the hashtag symbol (#) are considered comments and they are ignored.

After the above configuration file is created and placed under the detection directory: /opt/oracle/mgmt_agent/agent_inst/discovery/PrometheusEmitter/, the Management Agent will process it in few minutes and the metrics will start flowing to the OCI Monitoring service.

Create Policies to Post Metrics to OCI Monitoring

  1. Confirm a dynamic group was created.

    Before you create policies, you first need to confirm that a required dynamic group has been created.

    For example, confirm that a Management Agent dynamic group, named Management-Agent-Dynamic-Group, has been created with the following:
    ALL {resource.type='managementagent', resource.compartment.id='<AGENT_COMPARTMENT_OCID>'}

    In this example, the <AGENT_COMPARTMENT_OCID> is the compartment OCID of the Agent.

  2. Create an IAM policy for the dynamic group to allow the Management Agent service to post the metrics collected to the OCI Monitoring service in the compartment that you have selected using the compartmentId property from the previous configuration file created. 

    allow dynamic-group Management-Agent-Dynamic-Group to use metrics in compartment <PROMETHEUS_METRIC_COMPARTMENT_NAME> where target.metrics.namespace = 'poc_prometheus'

    • The Management-Agent-Dynamic-Group is the name of the dynamic group created.

    • The <PROMETHEUS_METRIC_COMPARTMENT_NAME> is the name of the compartment which OCID was provided using the compartmentId property when creating the configuration file. See Create a Configuration File.

      This is the compartment where the Prometheus metrics will get posted.

    • The poc_prometheus is the namespace and it must match the namespace value provided when creating the configuration file.

You need to restart the agent to apply the policy changes to the agent.

Note

If the policies are not created or they are not set up properly, you will receive HTTP 404 NotAuthorized error messages in the Agent logs.

Configuration File Properties

This section lists the mandatory and optional properties available to collect metrics using Prometheus exporter.

Mandatory Properties

The following properties are required when creating the configuration file:

Name Units/Format Description
url String The url through which the Prometheus Exporter publishes its metrics. (http only)
namespace String The OCI monitoring namespace to which scraped metrics should be uploaded.
compartmentId String The OCID of the compartment to which the scraped metrics should be uploaded.

Optional Properties:

The following optional properties can be used in the configuration file:

Name Units/Format Description
allowMetrics String - comma separated metric name list. The complete set of desired scraped metrics. Use this property to limit the set of metrics uploaded if required.
connection-timeout Number in milliseconds. The timeout for connecting to the Prometheus Exporter's endpoint.
metricDimensions String - comma separated property name list. The names of other user-supplied properties expressed as fixed values to be used as dimensions for every uploaded datapoint.
proxy-url String - URL. The URL of the network proxy that provides access to the Prometheus Exporter's endpoint (URL required property).
read-data-limit Number in kilobytes. The limit on the data being sent, not to exceed the agent's fixed limit of 400 (KB).
read-timeout Number in milliseconds. The timeout for reading the response from the Prometheus Exporter's endpoint.
resourceGroup String. It includes only alphanumeric characters, periods (.), underscores (_), hyphens (-), and dollar signs ($). The OCI Monitoring resource group.

If it's specified, the resource group will be assigned to each of the emitted metrics.
scheduleMins Scraping interval in minutes If it's specified, the scraping should occur at the specified interval in minutes.

The minimum value is 1 minute.

If it is not specified, the default value is 5 minutes.

Note: Lowering the collection interval can have an impact on the CPU.

Example: scheduleMins=1