Replicate data from Autonomous Transaction Process to Azure Synapse

This quickstart demonstrates how to set up a data replication from Autonomous Transaction Processing to Azure Synapse Database using OCI GoldenGate.

Before you begin

To successfully complete this quickstart, you must have the following:

  • Azure Synapse Workspace
  • Azure Storage Container associated to Azure Synapse Workspace
  • Azure Synapse Database Scoped Credential to give permissions to SQL pool to access Storage Account
  • Target table existence: The target tables should exist on the Synapse database before replication. As GoldenGate uses Merge SQL Statement, the target table must be a hash distributed table.
  • OCI GoldenGate Azure Data Lake Storage Connection assigned to deployment. If it doesn't exist, create an Azure Data Lake Storage connection and assign to deployment.

Environment set up: Autonomous Transaction Processing

If you don't already have a source database set up for replication, you can follow these steps to load a sample schema to use for this quickstart. This quickstart uses Autonomous Transaction Processing for the source database.

To set up the source Autonomous Database:

  1. Download and unzip the sample database schema.
  2. In the Oracle Cloud console, select your Autonomous Transaction Processing (ATP) instance from the Autonomous Databases page to view its details and access Database Actions.
  3. Unlock the GGADMIN user:
    1. Click Database actions, then click Database Users.
    2. Locate GGADMIN and then click its ellipsis menu (three dots) and select Edit.
    3. In the Edit User panel, enter the GGADMIN password, confirm the password, and then deselect Account is Locked.
    4. Click Apply Changes.
  4. Load the source sample schema and data:
    1. From the Database actions menu, under Development, select SQL.
    2. Copy and paste the script from OCIGGLL_OCIGGS_SETUP_USERS_ATP.sql into the SQL worksheet.
    3. Click Run Script. The Script Output tab displays confirmation messages.
    4. Clear the SQL worksheet and then copy and paste the SQL script from OCIGGLL_OCIGGS_SRC_USER_SEED_DATA.sql.

      Tip:

      You may need to run each statement separately for the SQL tool to execute the scripts successfully.
    5. To verify that the tables were created successfully, close the SQL window and reopen it again. In the Navigator tab, look for the SRC_OCIGGLL schema and then select tables from their respective dropdowns.
  5. Enable supplemental logging:
    1. Clear the SQL Worksheet.
    2. Enter the following statement, and then click Run Statement:
      ALTER PLUGGABLE DATABASE ADD SUPPLEMENTAL LOG DATA;

Environment setup: Azure Synapse

  1. Open Azure Synapse Workspace .
  2. Click New, and then SQL Script.
  3. Connect to your sql pool and select your database.
  4. Enter the following command to create a Master Key Encryption:
    CREATE MASTER KEY ENCRYPTION BY PASSWORD='<password>';
  5. Use the following command to create Database Scoped Credential:
    Ensure that you replace the <storage-account-name> and <storage-account-access-key> placeholders with their actual values.
    CREATE DATABASE SCOPED CREDENTIAL <credential name>
    WITH
    IDENTITY='<storage-account-name>',
    SECRET='<storage-account-access-key>';
  6. Run the create table script:
    CREATE TABLE [dbo].[SRC_CUSTOMER] (
       CUSTID         NUMERIC(10) NOTNULL,
       DEAR           NUMERIC(1),
       LAST_NAME      VARCHAR(50),
       FIRST_NAME     VARCHAR(50),
       ADDRESS        VARCHAR(100),
       CITY_ID        NUMERIC(10),
       PHONE          VARCHAR(50),
       AGE            NUMERIC(3),
       SALES_PERS_ID  NUMERIC(10),
       constraint PK_SRC_CUSTOMER primarykeyNONCLUSTERED (CUSTID) NOT ENFORCED
    )
    WITH(DISTRIBUTION=HASH ([CUSTID]), HEAP ) GO
  7. Verify that the dbo.SRC_Customer table is created.

Task 1: Create the OCI GoldenGate resources

Task 2: Add the Extract

  1. On the Deployments page, select the source Autonomous Transaction Processing deployment.
  2. On the deployment details page, click Launch Console.
  3. Log in with the source deployment's administrator username and password.
  4. Add transaction information.
  5. Add an Extract.

Task 3: Add and run the Distribution Path

  1. If using GoldenGate credential store, create a user for the Distribution Path in the target Big Data deployment, otherwise skip to Step 3.
  2. In the source ATP GoldenGate deployment console, add a Path Connection for the user created in Step 1.
    1. In the source ATP GoldenGate deployment console, click Path Connections in the left navigation.
    2. Click Add Path Connection (plus icon), and then complete the following:
      1. For Credential Alias, enter GGSNetwork.
      2. For User ID, enter the name of the user created in Step 1.
      3. Enter the user's password twice for verification.
    3. Click Submit.

      The path connection appears in the Path Connections list.

  3. In the source ATP deployment console, add a Distribution Path with the following values:
    1. On the Source Options page:
      • For Source Extract, select the Extract created in Task 2.
      • For Trail Name, enter a two-character name, such as E1.
    2. On the Target Options page:
      • For Target Host, enter the host domain of the target deployment.
      • For Port Number, enter 443.
      • For Trail Name, enter a two-character name, such as E1.
      • For Alias, enter the Credential Alias created in Step 2.
  4. In the target Big Data deployment console, review the Receiver Path created as a result of the Distribution Path.
    1. In the target Big Data deployment console, click Receiver Service.
    2. Review the path details. This path was created as a result of the Distribution Path created in the previous step.

Task 4: Add and run the Replicat

  1. In the target Big Data deployment console, click Administrator Service, and then click Add Replicat (plus icon).
  2. Add a Replicat with the following values:
    1. On the Replicat Information page, under Replicat type, select Classic Replicat, enter a Process Name, and then click Next.
    2. On the Replication Options page:
      • For Name, enter the name of the Trail from Task 2.
      • For Domain, select a domain.
      • For Alias, select the Azure Synapse Analytics connection created in Task 1.
      • For Checkpoint Table, select the checkpoint table you created for the target deployment.
    3. On the Replicat Parameters page, replace MAP *.*, TARGET *.*; with:
      MAP SRC_OCIGGLL.SRC_CUSTOMER, TARGET dbo.SRC_CUSTOMER;
    4. Click Create and Run.
You're returned to the Overview page, where you can review the Replicat details.

Task 5: Verify the replication

Perform some updates to the source Autonomous Transaction Processing instance to verify replication to Azure Data Lake Storage.
  1. In the Oracle Cloud console, open the navigation menu, select Oracle Database, and then select Autonomous Transaction Processing.
  2. In the list of Autonomous Transaction Processing instances, select your source instance to view its details.
  3. On the database details page, click Database Actions.
    Note

    You should be automatically logged in. If not, log in with the database credentials.
  4. On the Database Actions home page, select SQL.
  5. Enter the following into the worksheet and click Run Script.
  6. In the source GoldenGate OCI GoldenGate deployment console, select the Extract name, and then click Statistics. Verify that SRC_OCIGGLL.SRC_CUSTOMER has 7 inserts.
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1001,0,'Brendt','Paul','10 Jasper Blvd.',107,'(212) 555 2146',19,10);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1002,0,'McCarthy','Robin','27 Pasadena Drive',11,'(214) 555 3075',29,11);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1003,0,'Travis','Peter','7835 Hartford Drive',12,'(510) 555 4448',34,12);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1004,0,'Larson','Joe','87 Carmel Blvd.',13,'(213) 555 5095',45,13);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1005,0,'Goldschmidt','Tony','91 Torre drive',14,'(619) 555 6529',55,20);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1006,0,'Baker','William','2890 Grant Avenue',15,'(312) 555 7040',64,21);
    Insert into SRC_OCIGGLL.SRC_CUSTOMER (CUSTID,DEAR,LAST_NAME,FIRST_NAME,ADDRESS,CITY_ID,PHONE,AGE,SALES_PERS_ID) values (1007,0,'Swenson','Jack','64 Imagination Drive',19,'(202) 555 8125',74,22)
  7. In the target Big Data OCI GoldenGate deployment console, select the Replicat name, and then click Statistics. Verify that SRC_OCIGGLL.SRC_CUSTOMER has 7 inserts.
  8. In Azure console, navigate to Azure Synapse workspace Console. Run Select * from dbo.SRC_CUSTOMER and verify that SRC_OCIGGLL.SRC_CUSTOMER has 7 inserts.