Receive Notifications for Data Catalog Harvest Events

In this tutorial, you set up the Oracle Cloud Infrastructure Events service to send you an email whenever a Data Catalog harvest job starts, succeeds, or fails in a specific compartment. Key tasks include how to:

Create topics and subscriptions.
Write rules to receive email notifications for events.
Create buckets with data.
Harvest data in Data Catalog.

Before You Begin

To successfully perform this tutorial, you must have:

An Oracle Cloud Infrastructure account. See Request and Manage Free Oracle Cloud Promotions.
Two images ready to add to your bucket later:
- <image-01>
- <image-02>

If you have administrative rights to your account, skip the rest of this section. Otherwise, have your administrator add the following policy to your account:

allow group <the-group-your-username-belongs> to manage all-resources in compartment catalog-compartment

See Common Policies for more examples.

Note

In the next section, you make a compartment for your data catalog instances, called catalog-compartment.

1. Set Up a Notification

In this section, you set up the Notifications service to register your email for a custom subscription. Later, you use this subscription, to receive notifications in your email account.

Create a Compartment

Create a compartment for data catalog instances, events, and notifications.

Open the navigation menu and click Identity & Security. Under Identity, click Compartments.
Click Create Compartment.
Fill in the following information:
- Name: catalog-compartment
- Description: Compartment for data catalogs.
- Parent Compartment: <your-tenancy>(root)
Click Create Compartment.

You have successfully created a compartment for your data catalog resources.

Create a Topic

Open the navigation menu and click Developer Services. Under Application Integration, click Notifications.
In the left panel, for Compartment, select catalog-compartment.
Refresh the left navigation if the Compartment section doesn't list catalog-compartment.
Note

Ensure you select this compartment for all the resources that you create in this tutorial.
Click Create Topic.
Fill in the following information:
- Name: catalog-topic
- Description: Topic for data catalog harvest events.
Click Create.

You have successfully created a topic for your notifications.

Create a Subscription

In the Notifications service, in the list of topics, click catalog-topic.
Click Create Subscription.
Fill in the following information:
- Protocol: Email
- Email: <your-email>
Click Create.
Check your email account for an email with the subject, noreply@notification.<your-region>.oci.oraclecloud.com
Click the Confirm Subscription link in the email.
You get a page in the browser with the message:
You have subscribed <your-email> to the topic: catalog-topic.
In the Console, wait for the status of the subscription to change from pending to active.

You have successfully created an email subscription for your topic.

2. Set Up an Event Rule

In this section, you set up a rule in the Events service to send you an email whenever a harvest job starts, succeeds or fails in your compartment.

Create a Rule

Open the navigation menu and click Observability & Management. Under Events Service, click Rules.
Click Create Rule.
Fill in the following information:
- Display Name: catalog-harvest-rule
- Description: Receive an email when a harvest job starts, succeeds or fails in the catalog-compartment.
- Rule Conditions:
  - Condition: Event Type
  - Service Name: Data Catalog
  - Event Type: Harvest - Begin
  - Event Type: Harvest - End
  Note
  
  Ensure you select both Harvest-Begin and Harvest-End.
Click + Another Condition.

Fill in the following information:

Rule Conditions:
- Condition: Attribute
- Attribute Name: CompartmentName
- Attribute Values: catalog-compartment

The Rule Logic is automatically populated:

MATCH event WHERE (
eventType EQUALS ANY OF (
com.oraclecloud.datacatalog.harvestjob.end,
com.oraclecloud.datacatalog.harvestjob.begin
)
AND (
compartmentName MATCHES ANY OF (
laleh-catalog-compartment
)
)
)

In the Actions section, fill in the following information:
- Action Type: Notifications
- Notifications Compartment: catalog-compartment
- Topic: catalog-topic
Click Create Rule.

You have successfully created an event rule for your data catalog harvest jobs.

3. Create a Bucket with Data

In this section, you create a bucket in the Object Storage service. Then you add data to the bucket.

Create a Bucket

Create a bucket in the Object Storage service.

Open the navigation menu and click Storage. Under Object Storage & Archive Storage, click Buckets.
Click Create Bucket.
Fill in the following information:
- Name: catalog-bucket
- Leave the default value for other fields.
Click Create Bucket.

You have successfully created a bucket for your data.

Add Data to Bucket

In the Object Storage service, in the list of buckets, click catalog-bucket.
Click Upload.
Select the two images that you prepared in the Before You Begin section.
Click Upload and then Close.
The two images appear in the list of Objects.

You have successfully added data to your bucket.

Give Data Catalog Access to Bucket

In this section, you give the Data Catalog service access to Object Storage resources in the catalog-compartment. This policy allows your data catalog instances to download metadata from the catalog-bucket.

Open the navigation menu and click Identity & Security. Under Identity, click Policies.
Click Create Policy.
Fill in the following information:
- Name: catalog-policy
- Description: Allow Data Catalog to access Object Storage resources.
- In the Policy Builder section, click Customize (Advanced).

Paste the following policy in the policy builder:

allow service datacatalog to read object-family in compartment catalog-compartment

Click Create.

You have successfully created a policy for data catalog to access object storage resources including buckets.

4. Harvest the Bucket

In this section, you create a data catalog instance. Then you connect your data catalog instance to your bucket and harvest metadata from the bucket. Finally, you review your email notifications for the harvest jobs.

Gather Required Information

Prepare the information you need and copy them into a notepad.

Collect the following information from the Console:

Compartment ID: <compartment-ocid>
- In the Console search bar, enter catalog-compartment.
- Click catalog-compartment in the search results.
- Copy the OCID.
(OCI) Region: <region-identifier>
- From the top navigation bar, find <your-region>. Example: US West (Phoenix).
- On the top navigation bar, go to your Profile icon and click Tenancy: <your tenancy>.
- Click Manage Regions.
- In the list of regions, find the <region-identifier> for <your-region>. Example: us-phoenix-1.
URL: <object-storage-swift-URI>
- Copy the following string:
```
https://swiftobjectstorage.<region-identifier>.oraclecloud.com
```
- Replace <region-identifier> with the information you gathered for Region.
Namespace: <object-storage-namespace>
- In the top navigation bar, go to your Profile icon and click Tenancy: <your tenancy>.
- Copy the value for Object Storage Namespace.

Create a Data Catalog Instance

Open the navigation menu and click Analytics & AI. Under Data Lake, click Data Catalog.
Click Create Data Catalog.
Fill in the following information:
- Create in Compartment: catalog-compartment
- Name: catalog-instance-1
- Leave the default value for other fields.
Click Create Data Catalog.

You have successfully created a data catalog instance.

Create a Data Asset

In the Data Catalog service, in the list of data catalog instances, click catalog-instance-1.
In the Quick Actions tile, select Create Data Asset.
Fill in the following information:
- Name: object-storage-data-asset
For Type, select Oracle Object Storage
Use the information from the Gather Required Information section for the following fields:
- URL: https://swiftobjectstorage.<region-identifier>.oraclecloud.com
- Namespace: <object-storage-namespace>
Click Create.

You have successfully created a data asset in your data catalog instance.

Connect to Bucket

Connect catalog-bucket to catalog-instance-1.

In the Home page of catalog-instance-1, go to the Recently Updated tile, and select object-storage-data-asset.
Under Connections, click Add Connection.
Fill in the following information:
- Name: catalog-bucket-connection
For Type, select S2S Principal
Use the information from the Gather Required Information section for the following fields:
- OCI Region: <region-identifier>
- Compartment OCID: <compartment-ocid>
Select Make this the default connection for the data asset.
Click Test Connection.
Click Add.

You have successfully connected a bucket to your data catalog instance.

Harvest the Bucket

In the Home page of catalog-instance-1, go to the Recently Updated tile, and select object-storage-data-asset.
Click Harvest.
For the Select a Connection step, select catalog-bucket-connection, and then click Next.
In the Select Data Entities step, do the following:
- View the content of the bucket: Click catalog-bucket.
- Add the metadata for all the images to the catalog: Click Add All.
- Click Next.
Fill in or select the following fields:
- Job Name: harvest-catalog-bucket
- Include Unrecognized Files: selected
- Leave the default value for other fields.
Click Create Job.
Click the harvest-catalog-bucket job.
Confirm the following summary:
- Status: Succeeded
- Data Entities Harvested: 2
- Unrecognized Files Harvested: 2 (Images are unrecognized files.)

You have successfully created a harvest job for your bucket, in your data catalog instance.

Review your Email

Check your email account for an email with the following subject:

OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.begin

Review the content of the email.

From: noreply@notification.<region-identifier>.oci.oraclecloud.com

Example email body:

{
  "eventType" : "com.oraclecloud.datacatalog.harvestjob.end",
  "cloudEventsVersion" : "0.1",
  "eventTypeVersion" : "2.0",
  "source" : "DataCatalog",
  "eventTime" : "<time-harvest-job-began>",
  "contentType" : "application/json",
  "data" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx",
    "compartmentName" : "catalog-compartment",
    "resourceName" : "",
    "resourceId" : "ocid1.datacatalog.oc1.iad.aaaaxxxx",
    "additionalDetails" : {
      "harvestStatus" : "IN_PROGRESS",
      "dataAssetKey" : "<data-asset-key>",
      "dataAssetName" : "object-storage-data-asset",
      "connectionKey" : "<connection-key",
      "jobKey" : "<job-key>",
      "jobUrl" : "",
      "jobDefinitionKey" : "<job-definition-key>",
      "connectionName" : "catalog-bucket-connection",
      "jobDefinitionName" : "harvest-catalog-bucket",
      "jobExecutionKey" : "<job-execution-key>"
    }
  },
  "eventID" : "<event-ocid>",
  "extensions" : {
    "compartmentId" : "ocid1.compartment.oc1..aaaaxxxx"
  }
}

Find the value for "harvestStatus".
```
"harvestStatus" : "IN_PROGRESS"
```
Check your email account for an email with the subject:
OCI Event Notification :com.oraclecloud.datacatalog.harvestjob.end