Receive Notifications for Data Catalog Harvest Events

Before You Begin

To successfully perform this tutorial, you must have:

An Oracle Cloud Infrastructure account. See Request and Manage Free Oracle Cloud Promotions.
Two images ready to add to your bucket later:
- <image-01>
- <image-02>

If you have administrative rights to your account, skip the rest of this section. Otherwise, have your administrator add the following policy to your account:

allow group <the-group-your-username-belongs> to manage all-resources in compartment catalog-compartment

See Common Policies for more examples.

Note

In the next section, you make a compartment for your data catalog instances, called catalog-compartment.

1. Set Up a Notification

In this section, you set up the Notifications service to register your email for a custom subscription. Later, you use this subscription, to receive notifications in your email account.

Create a Compartment

Create a Topic

Create a Subscription

2. Set Up an Event Rule

In this section, you set up a rule in the Events service to send you an email whenever a harvest job starts, succeeds or fails in your compartment.

Create a Rule

Open the navigation menu and select Observability & Management. Under Events Service, select Rules.
Click Create Rule.
Fill in the following information:
- Display Name: catalog-harvest-rule
- Description: Receive an email when a harvest job starts, succeeds or fails in the catalog-compartment.
- Rule Conditions:
  - Condition: Event Type
  - Service Name: Data Catalog
  - Event Type: Harvest - Begin
  - Event Type: Harvest - End
  Note
  
  Ensure you select both Harvest-Begin and Harvest-End.
Click + Another Condition.

Fill in the following information:

Rule Conditions:
- Condition: Attribute
- Attribute Name: CompartmentName
- Attribute Values: catalog-compartment

The Rule Logic is automatically populated:

MATCH event WHERE (
eventType EQUALS ANY OF (
com.oraclecloud.datacatalog.harvestjob.end,
com.oraclecloud.datacatalog.harvestjob.begin
)
AND (
compartmentName MATCHES ANY OF (
laleh-catalog-compartment
)
)
)

In the Actions section, fill in the following information:
- Action Type: Notifications
- Notifications Compartment: catalog-compartment
- Topic: catalog-topic
Click Create Rule.

You have successfully created an event rule for your data catalog harvest jobs.

3. Create a Bucket with Data

In this section, you create a bucket in the Object Storage service. Then you add data to the bucket.

Create a Bucket

Add Data to Bucket

Give Data Catalog Access to Bucket

In this section, you give the Data Catalog service access to Object Storage resources in the catalog-compartment. This policy allows your data catalog instances to download metadata from the catalog-bucket.

Open the navigation menu and select Identity & Security. Under Identity, select Policies.
Click Create Policy.
Fill in the following information:
- Name: catalog-policy
- Description: Allow Data Catalog to access Object Storage resources.
- In the Policy Builder section, click Customize (Advanced).

Paste the following policy in the policy builder:

allow service datacatalog to read object-family in compartment catalog-compartment

Click Create.

You have successfully created a policy for data catalog to access object storage resources including buckets.

4. Harvest the Bucket

In this section, you create a data catalog instance. Then you connect your data catalog instance to your bucket and harvest metadata from the bucket. Finally, you review your email notifications for the harvest jobs.

Gather Required Information

Prepare the information you need and copy them into a notepad.

Collect the following information from the Console:

Compartment ID: <compartment-ocid>
- In the Console search bar, enter catalog-compartment.
- Click catalog-compartment in the search results.
- Copy the OCID.
(OCI) Region: <region-identifier>
- From the top navigation bar, find <your-region>. Example: US West (Phoenix).
- On the top navigation bar, go to your Profile icon and click Tenancy: <your tenancy>.
- Click Manage Regions.
- In the list of regions, find the <region-identifier> for <your-region>. Example: us-phoenix-1.
URL: <object-storage-swift-URI>
- Copy the following string:
```
https://swiftobjectstorage.<region-identifier>.oraclecloud.com
```
- Replace <region-identifier> with the information you gathered for Region.
Namespace: <object-storage-namespace>
- In the top navigation bar, go to your Profile icon and click Tenancy: <your tenancy>.
- Copy the value for Object Storage Namespace.

Create a Data Catalog Instance

Check your email account for an email with the following subject:

OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.begin

Review the content of the email.

From: noreply@notification.<region-identifier>.oci.oraclecloud.com

Example email body:

{
  "eventType" : "com.oraclecloud.datacatalog.harvestjob.end",
  "cloudEventsVersion" : "0.1",
  "eventTypeVersion" : "2.0",
  "source" : "DataCatalog",
  "eventTime" : "<time-harvest-job-began>",
  "contentType" : "application/json",
  "data" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx",
    "compartmentName" : "catalog-compartment",
    "resourceName" : "",
    "resourceId" : "ocid1.datacatalog.oc1.iad.aaaaxxxx",
    "additionalDetails" : {
      "harvestStatus" : "IN_PROGRESS",
      "dataAssetKey" : "<data-asset-key>",
      "dataAssetName" : "object-storage-data-asset",
      "connectionKey" : "<connection-key",
      "jobKey" : "<job-key>",
      "jobUrl" : "",
      "jobDefinitionKey" : "<job-definition-key>",
      "connectionName" : "catalog-bucket-connection",
      "jobDefinitionName" : "harvest-catalog-bucket",
      "jobExecutionKey" : "<job-execution-key>"
    }
  },
  "eventID" : "<event-ocid>",
  "extensions" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx"
  }
}

Find the value for "harvestStatus".
```
"harvestStatus" : "IN_PROGRESS"
```
Check your email account for an email with the subject:
OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.end

Review the content of the email.

From: noreply@notification.<region-identifier>.oci.oraclecloud.com

Example email body:

{
  "eventType" : "com.oraclecloud.datacatalog.harvestjob.end",
  "cloudEventsVersion" : "0.1",
  "eventTypeVersion" : "2.0",
  "source" : "DataCatalog",
  "eventTime" : "<time-harvest-job-ended>",
  "contentType" : "application/json",
  "data" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx",
    "compartmentName" : "catalog-compartment",
    "resourceName" : "",
    "resourceId" : "ocid1.datacatalog.oc1.iad.aaaaxxxx",
    "additionalDetails" : {
      "harvestStatus" : "SUCCEEDED",
      "dataAssetKey" : "<data-asset-key>",
      "dataAssetName" : "object-storage-data-asset",
      "connectionKey" : "<connection-key",
      "jobKey" : "<job-key>",
      "jobUrl" : "",
      "jobDefinitionKey" : "<job-definition-key>",
      "connectionName" : "catalog-bucket-connection",
      "jobDefinitionName" : "harvest-catalog-bucket",
      "jobExecutionKey" : "<job-execution-key>"
    }
  },
  "eventID" : "<event-ocid>",
  "extensions" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx"
  }
}

Find the value for "harvestStatus".
```
"harvestStatus" : "SUCCEEDED"
```
The status is complete:
Note

The "harvestStatus" for a harvestjob.begin event is "IN_PROGRESS", and for a harvestjob.end event is either "SUCCEEDED" or "FAILED".

What's Next

Congratulations! You successfully received two notification emails, one when a harvest job started and one when it completed.

Now that you're familiar with harvesting from Oracle Object Storage, go through the following tutorials and create notifications for other types of data sources:

Oracle Cloud Infrastructure Documentation Try Free Tier

Receive Notifications for Data Catalog Harvest Events

Before You Begin 🔗

1. Set Up a Notification 🔗

2. Set Up an Event Rule 🔗

3. Create a Bucket with Data 🔗

4. Harvest the Bucket 🔗