Receive Notifications for Data Catalog Harvest Events

In this tutorial, you set up the Oracle Cloud Infrastructure Events service to send you an email whenever a Data Catalog harvest job starts, succeeds, or fails in a specific compartment. Key tasks include how to:

Create topics and subscriptions.
Write rules to receive email notifications for events.
Create buckets with data.
Harvest data in Data Catalog.

Before You Begin

To successfully perform this tutorial, you must have:

An Oracle Cloud Infrastructure account. See Request and Manage Free Oracle Cloud Promotions.
Two images ready to add to your bucket later:
- <image-01>
- <image-02>

If you have administrative rights to your account, skip the rest of this section. Otherwise, have your administrator add the following policy to your account:

allow group <the-group-your-username-belongs> to manage all-resources in compartment catalog-compartment

See Common Policies for more examples.

Note

In the next section, you make a compartment for your data catalog instances, called catalog-compartment.

1. Set Up a Notification

In this section, you set up the Notifications service to register your email for a custom subscription. Later, you use this subscription, to receive notifications in your email account.

Create a Compartment

Create a compartment for data catalog instances, events, and notifications.

Open the navigation menu and select Identity & Security. Under Identity, select Compartments.
Select Create Compartment.
Fill in the following information:
- Name: catalog-compartment
- Description: Compartment for data catalogs.
- Parent Compartment: <your-tenancy>(root)
Select Create Compartment.

You have successfully created a compartment for your data catalog resources.

Create a Topic

Open the navigation menu and select Developer Services. Under Application Integration, select Notifications.
In the leftmost panel, for Compartment, select catalog-compartment.
Refresh the leftmost navigation if the Compartment section doesn't list catalog-compartment.
Note

Ensure you select this compartment for all the resources that you create in this tutorial.
Select Create Topic.
Fill in the following information:
- Name: catalog-topic
- Description: Topic for data catalog harvest events.
Select Create.

You have successfully created a topic for your notifications.

Create a Subscription

In the Notifications service, in the list of topics, select catalog-topic.
Select Create Subscription.
Fill in the following information:
- Protocol: Email
- Email: <your-email>
Select Create.
Check your email account for an email with the subject, noreply@notification.<your-region>.oci.oraclecloud.com
Select the Confirm Subscription link in the email.
You get a page in the browser with the message:
You have subscribed <your-email> to the topic: catalog-topic.
In the Console, wait for the status of the subscription to change from pending to active.

You have successfully created an email subscription for your topic.

2. Set Up an Event Rule

In this section, you set up a rule in the Events service to send you an email whenever a harvest job starts, succeeds or fails in your compartment.

Create a Rule

Open the navigation menu and select Observability & Management. Under Events Service, select Rules.
Select Create Rule.
Fill in the following information:
- Display Name: catalog-harvest-rule
- Description: Receive an email when a harvest job starts, succeeds or fails in the catalog-compartment.
- Rule Conditions:
  - Condition: Event Type
  - Service Name: Data Catalog
  - Event Type: Harvest - Begin
  - Event Type: Harvest - End
  Note
  
  Ensure you select both Harvest-Begin and Harvest-End.
Select + Another Condition.

Fill in the following information:

Rule Conditions:
- Condition: Attribute
- Attribute Name: CompartmentName
- Attribute Values: catalog-compartment

The Rule Logic is automatically populated:

MATCH event WHERE (
eventType EQUALS ANY OF (
com.oraclecloud.datacatalog.harvestjob.end,
com.oraclecloud.datacatalog.harvestjob.begin
)
AND (
compartmentName MATCHES ANY OF (
laleh-catalog-compartment
)
)
)

In the Actions section, fill in the following information:
- Action Type: Notifications
- Notifications Compartment: catalog-compartment
- Topic: catalog-topic
Select Create Rule.

You have successfully created an event rule for your data catalog harvest jobs.

3. Create a Bucket with Data

In this section, you create a bucket in the Object Storage service. Then you add data to the bucket.

Create a Bucket

Create a bucket in the Object Storage service.

Open the navigation menu and select Storage. Under Object Storage & Archive Storage, select Buckets.
Select Create Bucket.
Fill in the following information:
- Name: catalog-bucket
- Leave the default value for other fields.
Select Create Bucket.

You have successfully created a bucket for your data.

Add Data to Bucket

In the Object Storage service, in the list of buckets, select catalog-bucket.
Select Upload.
Select the two images that you prepared in the Before You Begin section.
Select Upload and then Close.
The two images appear in the list of Objects.

You have successfully added data to your bucket.

Give Data Catalog Access to Bucket

In this section, you give the Data Catalog service access to Object Storage resources in the catalog-compartment. This policy lets your data catalog instances download metadata from the catalog-bucket.

Open the navigation menu and select Identity & Security. Under Identity, select Policies.
Select Create Policy.
Fill in the following information:
- Name: catalog-policy
- Description: Allow Data Catalog to access Object Storage resources.
- In the Policy Builder section, select Customize (Advanced).

Paste the following policy in the policy builder:

allow service datacatalog to read object-family in compartment catalog-compartment

Select Create.

You have successfully created a policy for data catalog to access object storage resources including buckets.

4. Harvest the Bucket

In this section, you create a data catalog instance. Then you connect your data catalog instance to your bucket and harvest metadata from the bucket. Finally, you review your email notifications for the harvest jobs.

Gather Required Information

Prepare the information you need and copy them into a notepad.

Collect the following information from the Console:

Compartment ID: <compartment-ocid>
- In the Console search bar, enter catalog-compartment.
- Select catalog-compartment in the search results.
- Copy the OCID.
(OCI) Region: <region-identifier>
- From the top navigation bar, find <your-region>. Example: US West (Phoenix).
- On the top navigation bar, go to your Profile icon and select Tenancy: <your tenancy>.
- Select Manage Regions.
- In the list of regions, find the <region-identifier> for <your-region>. Example: us-phoenix-1.
URL: <object-storage-swift-URI>
- Copy the following string:
```
https://swiftobjectstorage.<region-identifier>.oraclecloud.com
```
- Replace <region-identifier> with the information you gathered for Region.
Namespace: <object-storage-namespace>
- In the top navigation bar, go to your Profile icon and select Tenancy: <your tenancy>.
- Copy the value for Object Storage Namespace.

Create a Data Catalog Instance

Open the navigation menu and select Analytics & AI. Under Data Lake, select Data Catalog.
Select Create Data Catalog.
Fill in the following information:
- Create in Compartment: catalog-compartment
- Name: catalog-instance-1
- Leave the default value for other fields.
Select Create Data Catalog.

You have successfully created a data catalog instance.

Create a Data Asset

In the Data Catalog service, in the list of data catalog instances, select catalog-instance-1.
In the Quick Actions tile, select Create Data Asset.
Fill in the following information:
- Name: object-storage-data-asset
For Type, select Oracle Object Storage
Use the information from the Gather Required Information section for the following fields:
- URL: https://swiftobjectstorage.<region-identifier>.oraclecloud.com
- Namespace: <object-storage-namespace>
Select Create.

You have successfully created a data asset in your data catalog instance.

Connect to Bucket

Connect catalog-bucket to catalog-instance-1.

In the Home page of catalog-instance-1, go to the Recently Updated tile, and select object-storage-data-asset.
Under Connections, select Add Connection.
Fill in the following information:
- Name: catalog-bucket-connection
For Type, select S2S Principal
Use the information from the Gather Required Information section for the following fields:
- OCI Region: <region-identifier>
- Compartment OCID: <compartment-ocid>
Select Make this the default connection for the data asset.
Select Test Connection.
Select Add.

You have successfully connected a bucket to your data catalog instance.

Harvest the Bucket

In the Home page of catalog-instance-1, go to the Recently Updated tile, and select object-storage-data-asset.
Select Harvest.
For the Select a Connection step, select catalog-bucket-connection, and then select Next.
In the Select Data Entities step, do the following:
- View the content of the bucket: Select catalog-bucket.
- Add the metadata for all the images to the catalog: Select Add All.
- Select Next.
Fill in or select the following fields:
- Job Name: harvest-catalog-bucket
- Include Unrecognized Files: selected
- Leave the default value for other fields.
Select Create Job.
Select the harvest-catalog-bucket job.
Confirm the following summary:
- Status: Succeeded
- Data Entities Harvested: 2
- Unrecognized Files Harvested: 2 (Images are unrecognized files.)

You have successfully created a harvest job for your bucket, in your data catalog instance.

Review your Email

Check your email account for an email with the following subject:

OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.begin

Review the content of the email.

From: noreply@notification.<region-identifier>.oci.oraclecloud.com

Example email body:

{
  "eventType" : "com.oraclecloud.datacatalog.harvestjob.end",
  "cloudEventsVersion" : "0.1",
  "eventTypeVersion" : "2.0",
  "source" : "DataCatalog",
  "eventTime" : "<time-harvest-job-began>",
  "contentType" : "application/json",
  "data" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx",
    "compartmentName" : "catalog-compartment",
    "resourceName" : "",
    "resourceId" : "ocid1.datacatalog.oc1.iad.aaaaxxxx",
    "additionalDetails" : {
      "harvestStatus" : "IN_PROGRESS",
      "dataAssetKey" : "<data-asset-key>",
      "dataAssetName" : "object-storage-data-asset",
      "connectionKey" : "<connection-key",
      "jobKey" : "<job-key>",
      "jobUrl" : "",
      "jobDefinitionKey" : "<job-definition-key>",
      "connectionName" : "catalog-bucket-connection",
      "jobDefinitionName" : "harvest-catalog-bucket",
      "jobExecutionKey" : "<job-execution-key>"
    }
  },
  "eventID" : "<event-ocid>",
  "extensions" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx"
  }
}

Find the value for "harvestStatus".
```
"harvestStatus" : "IN_PROGRESS"
```
Check your email account for an email with the subject:
OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.end