Data Storage and Logging in JMS Fleets

JMS Fleets employs a systematic approach to organizing, storing, and managing data to ensure optimal processing and extended analysis. Raw data collected from events sent by the JMS plugin from each managed instance is initially stored in the associated log objects of the fleet to which the managed instance belongs. JMS leverages the OCI Logging service to categorize this data into two main types: Inventory Logs and Operation Logs.

Inventory Logs: These Custom Logs store Java Runtime inventory and usage-related information reported from hosts by the JMS plugin. Inventory logs are mandatory for fleets to function effectively.
Operation Logs: These Custom Logs capture operations carried out through JMS Fleets. While optional, operation logs complement inventory logs by providing additional operational insights.

If fleets are created through the OCI console, log objects are created by the JMS Fleets in the same compartment as the fleet. If you're creating fleets using the createFleet API, then you'll need to provide as parameters the OCID's of log group and objects to be associated with the fleet. The log object associated with the fleet can be accessed using the Log Configuration Details present in the Fleet Properties Tab.

Both inventory and operation logs are organized under a Log Group, which facilitates advanced analytics through Logging Analytics. By default, these log objects have a retention period of one month. To preserve data indefinitely, you can establish a Connector Hub connection between the log object and an object storage bucket.

Note

Inventory and operation logs will incur storage costs. See the Oracle Cloud Cost Estimator for more details.

Raw data stored in logs is processed, and aggregated data is transferred to the Autonomous Transaction Processing (ATP) database owned by JMS service. JMS retains this aggregated data for up to one year, enabling users to analyze historical trends and gain actionable insights into their Java workloads.

Read the following for more information about logging:

Inventory Logs

Inventory logs contains information about the different event types captured by JMS Fleets. Each message is in JSON format. The message contains the type and data of the event, when it was received, ID of the log, Management Agent ID, and other data. The Management Agent writes directly to the inventory log objects associated with its JMS fleet in an interval that you configure in the Agent Settings.

The type can be one of the following:

jms.jvm.usage.log: Data about JVM invocation found during Java runtime usage reporting.
jms.jvm.installation.log: Data about JVM found during the Java runtime discovery scan.
jms.jvm.usage.attach.log: Data about JVM invocation found by Attach API during the execution of advanced features
jms.javaserver.metadata.log: Data from a Java server scan.
jms.javaserver.libraries.log: Data from a Java server libraries scan.
jms.java.libraries.log: Data from a Java libraries scan.
jms.drs.scan.log: Data from a DRS scan.
jms.agent.plugin.start.log: Generated each time the JMS plugin starts.

Sample payload for the jms.jvm.usage.log event type:

{
  "datetime": 1654089796180,
  "logContent": {
    "data": {
      "data": {
        "additionalProperties": {
          "java.runtime.name": "Java(TM) SE Runtime Environment"
        },
        "applicationName": "com.oracle.test.HelloWorld",
        "classPath": "/home/opc/HelloWorld-1.0.jar",
	  The classPath is a list of paths, but due to logging limitations, JMS provides the same information as an array of path. There is a limit 10,000 characters for string values.
        "fleetId": "ocid1.jmsfleet.oc1.iad.fleet-id",
        "javaArgs": "John",
        "javaCommand": "com.oracle.test.HelloWorld",
        "javaDistribution": "Java(TM) SE Runtime Environment",
        "javaHome": "/usr/java/jdk-11.0.14",
        "javaMajorVersion": "11",
        "javaVendor": "Oracle Corporation",
        "javaVersion": "11.0.14",
        "managedInstanceId": "ocid1.managementagent.oc1.iad.management-agent-id",
        "osArch": "amd64",
        "osName": "Linux",
        "osVersion": "5.4.17-2136.302.7.2.1.el8uek.x86_64",
        "startTime": "2022-05-27T05:02:01Z",
        "typeOfStart": "VM start"
      },
      "datacontenttype": "application/json",
      "dataschema": "1.0",
      "id": "b36806ce-564e-46e5-99fc-d50fb386dd7a",
      "source": "ocid1.managementagent.oc1.iad.management-agent-id",
      "specversion": "1.0",
      "time": "2022-06-01T13:23:16.180Z",
      "type": "jms.jvm.usage.log"
    },
    "id": "b36806ce-564e-46e5-99fc-d50fb386dd7a",
    "oracle": {
      "compartmentid": "ocid1.compartment.oc1..compartment-id",
      "ingestedtime": "2022-06-01T13:24:12.141Z",
      "loggroupid": "ocid1.loggroup.oc1.iad.log-group-id",
      "logid": "ocid1.log.oc1.iad.log-id",
      "tenantid": "ocid1.tenancy.oc1..tenant-id"
    },
    "source": "ocid1.managementagent.oc1.iad.management-agent-id",
    "specversion": "1.0",
    "subject": "JMS",
    "time": "2022-06-01T13:23:16.180Z",
    "type": "jms.jvm.usage.log"
  }
}

Sample payload for jms.jvm.installation.log event type:

{
  "datetime": 1651838678794,
  "logContent": {
    "data": {
      "data": {
        "fleetId": "ocid1.jmsfleet.oc1.iad.fleet-id",
        "javaDistribution": "Java(TM) SE Runtime Environment",
        "javaHome": "/usr/java/jdk-17.0.2",
        "javaMajorVersion": "17",
        "javaVendor": "Oracle Corporation",
        "javaVersion": "17.0.2",
        "managedInstanceId": "ocid1.managementagent.oc1.iad.management-agent-id",
        "osArch": "amd64",
        "osName": "Linux",
        "osVersion": "5.4.17-2136.304.4.1.el8uek.x86_64"
      },
      "datacontenttype": "application/json",
      "dataschema": "1.0",
      "id": "3b8f47a4-46de-476f-b786-1dc4d4ddd6ec",
      "source": "ocid1.managementagent.oc1.iad.management-agent-id",
      "specversion": "1.0",
      "time": "2022-05-06T12:04:38.794Z",
      "type": "jms.jvm.installation.log"
    },
    "id": "id",
    "oracle": {
      "compartmentid": "ocid1.compartment.oc1..compartment-id",
      "ingestedtime": "2022-05-06T12:04:59.016Z",
      "loggroupid": "ocid1.loggroup.oc1.iad.loggroup-id",
      "logid": "ocid1.log.oc1.iad.log-id",
      "tenantid": "ocid1.tenancy.oc1..tenancy-id"
    },
    "source": "ocid1.managementagent.oc1.iad.management-agent-id",
    "specversion": "1.0",
    "subject": "JMS",
    "time": "2022-05-06T12:04:38.794Z",
    "type": "jms.jvm.installation.log"
  }
}

Operation Logs

Operation logs stores events related to operations carried out through JMS Fleets. Each message is in JSON format. The message contains the type and data of the event, when it was received, ID of the log, Work Request ID, and other data. If operation logs are enabled for a fleet, the Management Agent will write to the associated operation log objects as it executes work requests initiated in the fleet.

The JMS Fleets operation log data uses the format:

type is jms.publiclogs
specversion has version 1.0
encoding is application/json
source is jms

The payload event type jms.publiclogs shows the data about the Lifecycle Management Operations. See the following logContent.data.data attributes:

{
      "datetime": 1652437153474,
      "logContent": {
            "data": {
                  "category": "EXECUTION",
                  "fleetId": "ocid1.fleet-name.oc1.iad.fleet-id",
                  "level": "INFO",
                  "managedInstanceId": "ocid1.managementagent.oc1.iad.management-agent-id",
                  "message": "Work Item 0f68ca34-1c7f-4fb2-8b6a-d0119c6a4934, 
		   part of Work Request ocid1.jmsworkrequest.oc1.iad.work-request-id, 
		  status is Retrying for delete Java Runtime 15.0.2 at 
		  /usr/java/jdk-15.0.2 in test-environment-id with 
		  OCID: ocid1.managementagent.oc1.iad.management-agent-id",
                  "timestamp": "2022-05-02T03:36:21.235Z",
                  "workRequestId": "ocid1.jmsworkrequest.oc1.iad.work-request-id"
            },
            "id": "017e880a-b33d-4f90-bdc7-5a0034b7da36",
            "oracle": {
                  "compartmentid": "ocid1.compartment.oc1..compartment-id",
                  "ingestedtime": "2022-05-13T10:19:13.676Z",
                  "loggroupid": "ocid1.loggroup.oc1.iad.log-group-id",
                  "logid": "ocid1.log.oc1.iad.log-id",
                  "tenantid": "ocid1.tenancy.oc1..tenancy-id"
            },
            "source": "jms",
            "specversion": "1.0",
            "time": "2022-05-13T10:19:13.474Z",
            "type": "jms.publiclogs"
      }
}

Advanced Search Queries

From the Explore with Log Search option, select Swtich to Advanced mode. You can search the logs for specific events by using advanced queries. The following are suggested search queries.

Use the following query to search a whole compartment:

search "COMPARTMENT_OCID" | type='jms.jvm.usage.log' or type='jms.jvm.installation.log' | sort by datetime desc

Use the following query to search events from log group:

search "COMPARTMENT_OCID/LOG_GROUP_OCID" | type='jms.jvm.usage.log' or type='jms.jvm.installation.log' | sort by datetime desc

Use the following query to search events from log object:

search "COMPARTMENT_OCID/LOG_GROUP_OCID/LOG_OBJECT_ID" | type='jms.jvm.usage.log' or type='jms.jvm.installation.log' | sort by datetime desc

Use the following query to search events associated with fleet with FLEET_OCID in compartment COMPARTMENT_OCID with associated log object LOG_OBJECT_ID and log group LOG_GROUP_OCID:

search "COMPARTMENT_OCID/LOG_GROUP_OCID/LOG_OBJECT_ID" | sort by datetime desc

Note

In the following search query, you don't need to filter on type, because only JMS Fleets related messages are in the logObject associated with the fleet.

Alternatively you can enter COMPARTMENT_OCID and filter on fleet:

search "COMPARTMENT_OCID" | type='jms.jvm.usage.log' or type='jms.jvm.installation.log' and data.data.fleetId = 'FLEET_OCID' | sort by datetime desc

Use the following query to search from an agent with MANAGEMENT_INSTANCE_OCID:

search "COMPARTMENT_OCID" | type='jms.jvm.usage.log' or type='jms.jvm.installation.log' and data.source = 'MANAGEMENT_INSTANCE_OCID' | sort by datetime desc

Oracle Cloud Infrastructure Documentation

Inventory Logs

Operation Logs

Advanced Search Queries