Connect to OCI Streaming

Learn to create a connection to OCI Streaming to use as a source or target with OCI GoldenGate data replication and Stream Analytics.

Before you begin

  1. Ensure that you review how OCI GoldenGate connects to your source and targets.
  2. Obtain the Stream Pool username:
    1. From the Oracle Cloud console navigation menu, select Streaming, and then Stream Pools.
    2. On the Stream Pools page, select your pool to view its details.
    3. On the Stream Pool details page, under Resources, click Kafka Connection Settings.
    4. Copy the username for SASL Connection Strings.

      Note:

      If you're using Oracle Identity Cloud service, you must add OracleIdentityCloudService to your user name. For example, <tenancy-name>/OracleIdentityCloudService/<username>/<streampool-ocid>
  3. Create an Auth token:
    1. In the Oracle Cloud console global header, click Profile, and then select User settings.
    2. On the User Details page, under Resources, click Auth Tokens, and then click Generate Token.
    3. In the Generate Token dialog, enter a description, and then click Generate Token.
    4. Copy the auth token from the dialog to a secure location from where you can retrieve it later, and then click Close.

The Stream Pool username and Auth token will be entered for the Stream connection's username and password.

Create the source connection

To create a source OCI Streaming connection:
  1. From the OCI GoldenGate Overview page, click Connections.
    You can also click Create Connection under the Get started section and skip to step 3.
  2. On the Connections page, click Create Connection.
  3. On the Create Connection page, complete the fields as follows:
    1. For Name, enter a name for the connection.
    2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
    3. For Compartment, select the compartment in which to create the connection.
    4. From the Type dropdown, select OCI Streaming.
    5. For Stream pool details, you can choose:
      • Select a stream pool
        1. Select a stream pool from the dropdown, or click Change compartment to select one in a different compartment.
        2. For Network connectivity, select a Traffic routing method:
          • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
          • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

            Note:

            • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
            • Learn more about Oracle GoldenGate connectivity.
      • Enter stream pool information to manually enter the stream pool information.
        1. Under Bootstrap servers, select a Traffic routing method:
          • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
          • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

            Note:

            • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
            • Learn more about Oracle GoldenGate connectivity.
        2. Enter the Bootstrap server's Host and Port.

          Note:

          If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.
    6. For Username, enter the Stream Pool username copied from the SASL Connection Settings in the prerequisite steps above.
    7. For Password, enter the OCI Streaming password.

      Note:

      To use password secrets, ensure that you select Use secrets in vault in the Settings tab under Advanced Options, located at the bottom of this form.
    8. Expand Show advanced options. You can configure the following options:
      • Security
        • Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
        • Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
      • Settings
        • Select Use vault secrets to use a password secret for your connection.

        To capture from OCI Streaming, create a consumer.properties file with one of the following deserializers or converters, and then drag and drop the file to Consumer properties:

        • Kafka Consumer properties for JSON deserializer:
          key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
          value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
        • Kafka Consumer properties for JSON converter:
          key.converter=org.apache.kafka.connect.json.JsonConverter
          value.converter=org.apache.kafka.connect.json.JsonConverter
        • Kafka Consumer properties for Avro converter:
          key.converter=io.confluent.connect.avro.AvroConverter
          value.converter=io.confluent.connect.avro.AvroConverter
      • Tags: Add tags to organize your resources.
  4. Click Create.
After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Create the target connection

To create a target OCI Streaming connection:
  1. From the OCI GoldenGate Overview page, click Connections.
    You can also click Create Connection under the Get started section and skip to step 3.
  2. On the Connections page, click Create Connection.
  3. On the Create Connection page, complete the fields as follows:
    1. For Name, enter a name for the connection.
    2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
    3. For Compartment, select the compartment in which to create the connection.
    4. From the Type dropdown, select OCI Streaming.
    5. For Stream pool details, you can choose:
      • Select a stream pool
        1. Select a stream pool from the dropdown, or click Change compartment to select one in a different compartment.
        2. For Network connectivity, select a Traffic routing method:
          • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
          • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

            Note:

            • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
            • Learn more about Oracle GoldenGate connectivity.
      • Enter stream pool information to manually enter the stream pool information.
        1. Under Bootstrap servers, select a Traffic routing method:
          • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
          • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

            Note:

            • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
            • Learn more about Oracle GoldenGate connectivity.
        2. Enter the Bootstrap server's Host and Port.

          Note:

          If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.
    6. For Username, enter the Stream Pool username copied from the SASL Connection Settings in the prerequisite steps above.
    7. For Password, enter the OCI Streaming password.

      Note:

      To use password secrets, ensure that you select Use secrets in vault in the Settings tab under Advanced Options, located at the bottom of this form.
    8. Expand Show advanced options. You can configure the following options:
      • Security
        • Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
        • Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
      • Settings
        • Select Use vault secrets to use a password secret for your connection.

        To use Snappy compression in Kafka replication, drag and drop or select Producer properties, and change replication settings as discussed in Using Compression OCI GoldenGate (Confluent) Kafka Replication.

      • Tags: Add tags to organize your resources.
  4. Click Create.
After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.