Data Catalog Events
Oracle Cloud Infrastructure Events are JSON files that are emitted with some service operations and carry information about that operation.
You can define rules that trigger a specific action when an event occurs. For more information, see Overview of Events and Get Started with Events. For information about managing rules for events, see Managing Rules for Events.
Data Catalog emits an event during one of the following:
- A harvest job begins.
- A harvest job ends.
- A harvest jobs fails.
- A logical data entity is updated.
- A custom property value is updated.
Event Types for Harvest Jobs
When you run a harvest job, Data Catalog extracts technical metadata from a connected data source into your data catalog repository. Data Catalog emits events when a harvest job begins and ends. For example, you can set up the Oracle Cloud Infrastructure Events service to send you an email when harvest jobs succeed.
| Friendly Name | Description | Event Type |
|---|---|---|
| Harvest - Begin |
When Data Catalog starts a harvest job, it emits an event that has information about the job such as the data asset name, compartment id, and the harvest status of IN_PROGRESS. |
com.oraclecloud.datacatalog.harvestjob.begin |
| Harvest - End |
After a harvest job ends, Data Catalog emits an event with the harvest status of either SUCCEEDED or FAILED. |
com.oraclecloud.datacatalog.harvestjob.end |
Here is a reference event for a harvest end event that completes successfully:
{
"eventType": "com.oraclecloud.datacatalog.harvestjob.end",
"eventID": "<unique_ID>",
"cloudEventsVersion": "0.1",
"eventTypeVersion": "2.0",
"source": "DataCatalog",
"eventTime": "2020-12-02T13:31:31.979Z",
"contentType": "application/json",
"data": {
"eventGroupingId": "<JobExecutionKey>",
"compartmentId": "ocid1.compartment.oc1.<unique_ID>",
"compartmentName": "<CompartmentName where the user's catalog resides>",
"resourceId": "ocid1.datacatalog.oc1.<unique_ID>",
"additionalDetails": {
"dataAssetKey": "<DataAssetKey>",
"dataAssetName": "<DataAssetName>",
"jobDefinitionKey": "<JobDefinitionKey>",
"jobDefinitionName": "<JobDefinitionName>",
"connectionKey": "<ConnectionKey>",
"connectionName": "<ConnectionName>",
"jobKey": "<JobKey>",
"jobExecutionKey": "<JobExecutionKey>",
"harvestStatus": "SUCCEEDED",
"jobUrl": "<Url to get the execution details of the job>"
"HarvestedFoldersCount" : 0,
"HarvestedEntitiesCount" : 0,
"HarvestedLogicalEntitiesCount" : 0,
"HarvestedPhysicalEntitiesCount" : 0,
"HarvestedArchivedFilesCount" : 0,
"SkippedFilesCount" : 0,
"HarvestedUnrecognizedFilesCount" : 0,
"HarvestedFoldersCountStr" : "0",
"HarvestedEntitiesCountStr" : "0",
"HarvestedLogicalEntitiesCountStr" : "0",
"HarvestedPhysicalEntitiesCountStr" : "0",
"HarvestedArchivedFilesCountStr" : "0",
"SkippedFilesCountStr" : "0",
"HarvestedUnrecognizedFilesCountStr" : "0"
}
},
"extensions": {
"compartmentId": "ocid1.compartment.oc1.<unique_ID>"
}
}Event Type for Logical Data Entities
A logical data entity is a group of Object Storage files that are derived by creating and assigning filename patterns to a data asset. Data Catalog emits events when a logical data entity is updated. For example, when you create, update, or delete a logical data entity, Data Catalog emits an event, in the form of a JSON file. You can set up the Oracle Cloud Infrastructure Events service to send you this JSON file in email or Slack when logical data entities are updated in a specific compartment.
| Friendly Name | Description | Event Type |
|---|---|---|
| Logical Entities - Change |
When you update a logical data entity, Data Catalog emits an event about the changes. The JSON file for the event includes information about the created, updated, and deleted logical data entities. |
com.oraclecloud.datacatalog.changedlogicalentities |
Here is a reference event for an update to logical data entities:
{
"eventType": "com.oraclecloud.datacatalog.changedlogicalentities",
"eventID": "<unique_ID>"
,
"cloudEventsVersion": "0.1",
"eventTypeVersion": "2.0",
"source": "DataCatalog",
"eventTime": "2020-12-02T13:31:31.979Z",
"contentType": "application/json",
"data": {
"compartmentId": "ocid1.compartment.oc1.<unique_ID>",
"compartmentName": "<CompartmentName where the user's catalog resides>",
"resourceId": "ocid1.datacatalog.oc1.<unique_ID>"
,
"additionalDetails": {
"dataAssetKey": "<DataAssetKey>",
"dataAssetName": "<DataAssetName>",
"jobDefinitionKey": "<JobDefinitionKey>",
"jobDefinitionName": "<JobDefinitionName>",
"connectionKey": "<ConnectionKey>",
"connectionName": "<ConnectionName>",
"jobKey": "<JobKey>",
"jobExecutionKey": "<JobExecutionKey>",
"harvestStatus": "<IN_PROGRESS/SUCCEEDED/FAILED>",
"createdLogicalEntities": ["LE_key1", "LE_key2"],
"updatedLogicalEntities": ["LE_key3", "LE_key4"],
"deletedLogicalEntities": ["LE_key5", "LE_key6"]
}
},
"extensions": {
"compartmentId": "ocid1.compartment.oc1.<unique_ID>"
}
}Event Type for Custom Property Value Update
Custom properties are used to capture business context for data catalog objects. Data Catalog emits an event, in the form of a JSON file, when the value of the custom property of a data catalog object is updated.
You can set up the Oracle Cloud Infrastructure Events service to send you this JSON file in an email or through Slack, whenever the value of a custom property is updated in a specific compartment.
| Friendly Name | Description | Event Type |
|---|---|---|
|
Custom Property Value - Update |
When you add, update, or delete the custom property value of a data catalog object, Data Catalog emits an event. The JSON file for the event includes information about the updated object, the custom property, and the value of the custom property. |
com.oraclecloud.datacatalog.object.customproperty.update |
Here is a reference event for an update to a custom property value:
{
"eventType" : "com.oraclecloud.datacatalog.object.customproperty.update",
"eventID" : "<unique_ID>",
"cloudEventsVersion" : "0.1",
"eventTypeVersion" : "2.0",
"source" : "DataCatalog",
"eventTime" : "2021-06-21T11:12:15Z",
"contentType" : "application/json",
"data" : {
"compartmentId" : "ocid1.compartment.oc1.<unique_ID>",
"compartmentName" : "<CompartmentName where the user's catalog resides>",
"resourceName" : "",
"resourceId" : "ocid1.datacatalog.oc1.<unique_ID>",
"additionalDetails" : {
"associatedObjectKey" : "<AssociatedObjectKey>",
"associatedObjectType" : "<AssociatedObjectType>",
"customPropertyKey" : "<CustomPropertyKey>",
"customPropertyDataType" : "<CustomPropertyDataType>",
"associatedValue" : "<AssociatedValue>"
}
},
"extensions" : {
"compartmentId" : "ocid1.compartment.oc1.<unique_ID>"
}
}
Additional Resources
Here are some resources that you can use to learn more about events in Data Catalog:
Tutorial