Replicate data from Autonomous
Transaction Processing to Confluent Kafka
This quickstart demonstrates how to replicate data from Autonomous
Transaction Processing to Confluent Kafka using OCI GoldenGate.
Before you begin 🔗
To successfully complete this quickstart, you must have the following:
A source Autonomous Transaction Processing instance.
A Confluent Platform instance.
Environment set up: Autonomous Transaction Processing 🔗
If you don't already have a source database set up for replication, you
can follow these steps to load a sample schema to use for this quickstart. This
quickstart uses Autonomous Transaction Processing for the source database.
In the Oracle Cloud console, select your Autonomous Transaction Processing (ATP) instance from
the Autonomous Databases page to view its details and access Database
Actions.
Unlock the GGADMIN user:
Click Database actions, then click Database
Users.
Locate GGADMIN and then click its ellipsis menu (three dots) and select
Edit.
In the Edit User panel, enter the GGADMIN password, confirm the
password, and then deselect Account is Locked.
Click Apply Changes.
Load the source sample schema and data:
From the Database actions menu, under Development, select
SQL.
Copy and paste the script from
OCIGGLL_OCIGGS_SETUP_USERS_ATP.sql into the SQL
worksheet.
Click Run Script. The Script Output tab displays confirmation
messages.
Clear the SQL worksheet and then copy and paste the SQL script from
OCIGGLL_OCIGGS_SRC_USER_SEED_DATA.sql.
Tip:
You may need to run each statement separately for the
SQL tool to execute the scripts successfully.
To verify that the tables were created successfully, close the SQL
window and reopen it again. In the Navigator tab, look for the
SRC_OCIGGLL schema and then select tables from
their respective dropdowns.
Enable supplemental logging:
Clear the SQL Worksheet.
Enter the following statement, and then click Run
Statement:
ALTER PLUGGABLE DATABASE ADD SUPPLEMENTAL LOG DATA;
Environment set up: Confluent
Cloud 🔗
If you don't already have a target Confluent Cloud cluster, topic and
schema, you can do the following:
On the Replicat Information page,
under Replicat type, select Classic Replicat, enter a Process Name,
and then click Next.
On the Replication Options page:
For Name, enter the name of the Trail from
Task 2.
For Domain, select a domain.
For Alias, select the Confluent Kafka
connection created in Task 1.
For Checkpoint Table, select the checkpoint
table you created for the target deployment.
On the Replicat Parameters page,
replace MAP *.*, TARGET *.*;
with:
MAP SRC_OCIGGLL.SRC_CUSTOMER, Table SRC.CUSTOMER;
Click Create and Run.
You return to the Overview page, where you can review the Replicat details.
Task 5: Verify the
replication 🔗
Perform some updates to the source Autonomous Transaction Processing
instance to verify replication to Confluent Kafka Cloud.
In the Oracle Cloud console, open the navigation menu, select Oracle
Database, and then select Autonomous Transaction
Processing.
In the list of Autonomous Transaction Processing instances, select your source instance
to view its details.
On the database details page, click Database actions.
Note
You should be automatically logged
in. If not, log in with the database credentials.
On the Database actions home page, select SQL.
Enter the following into the worksheet and click Run
Script.
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1000,'Houston',20,743113);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1001,'Dallas',20,822416);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1002,'San Francisco',21,157574);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1003,'Los Angeles',21,743878);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1004,'San Diego',21,840689);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1005,'Chicago',23,616472);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1006,'Memphis',23,580075);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1007,'New York City',22,124434);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1008,'Boston',22,275581);
Insert into SRC_OCIGGLL.SRC_CITY (CITY_ID,CITY,REGION_ID,POPULATION) values (1009,'Washington D.C.',22,688002);
In the source ATP deployment console, select the Extract name, and then click
Statistics. Verify that SRC_OCIGGLL.SRC_CITY has 10
inserts.
In the target Big Data OCI GoldenGate deployment console, select the Replicat name, and then click Statistics. Verify
that SRC_OCIGGLL.SRC_CUSTOMER has 7 inserts.
Task 6: (Optional) Verify the
replication in Confluent CLI 🔗