Scenario: Sending Queue Messages to a Function

Integrate Queue with Functions using Connector Hub.

This scenario involves creating a function and then referencing that function as a target in a connector (Connector Hub)  to receive and process messages from queues.

The payload received by the function contains only messages from the queue.

For a related tutorial, see Use OCI Functions and OCI Queue to Authorize User Capabilities without Exposing Admin Privilege to Approvers.

For help with troubleshooting, see Troubleshooting Connectors.

• Console
• CLI
• API

• 1. On the Connectors list page, select Create connector. If you need help finding the list page, see Listing Connectors.
  2. On the Create connector page, enter a user-friendly name for the new connector and an optional description. Avoid entering confidential information.
  3. Select the compartment where you want to store the new connector.
  4. Under Configure connector, for Source, select Queue.
  5. For Target, select Functions.
  6. Under Configure source connection, select the queue that contains the messages that you want:
     • Compartment: Select the compartment that contains the queue that you want.
     • Queue: Select the queue that contains the messages that you want.

     • Channel Filter (under Message filtering) (optional): To filter messages from channels in the queue, enter a value.

       For example, to filter messages by channel ID, enter the channel ID.

       For supported values, see channelFilter at GetMessages (Queue API).

     Note
     
     A message that has been transferred to the connector's target is considered "consumed." To meet requirements of the Queue service, the connector deletes transferred messages from the source queue. For more information, see Consuming Messages.
  7. Under Configure target, configure the function to send the messages from the queue to.
     • Compartment: Select the compartment that contains the function that you want.
     • Function application: Select the name of the function application that contains the function that you want.
     • Function: Select the name of the function that you want to send the data to.
     • Show additional options: Select this link and specify limits for each batch of data sent to the function. To use manual settings, provide values for batch size limit (either KBs or number of messages) and batch time limit (seconds).

       For example, limit batch size by selecting either 5,000 kilobytes or 10 messages. An example batch time limit is 5 seconds.

     Considerations:

     • The connector flushes source data as a JSON list in batches. Maximum batch, or payload, size is 6 MB.
     • Functions are invoked synchronously with 6 MB of data per invocation. If data exceeds 6 MB, then the connector invokes the function again to move the data that's over the limit. Such invocations are handled sequentially.
     • Functions can execute for up to five minutes. See Delivery Details.
     • Don't return data from Functions targets to connectors. Connector Hub doesn't read data returned from Functions targets. If the function returns successfully without an error, the connector treats the messages in the batch as "consumed" and deletes those messages from the queue.
  8. Accept default policies: Select the Create link provided for each default policy.
  9. Select Create.

  The creation process begins, and its progress is displayed. On completion, the connector's details page opens.
• 1. Define the Queue source for the connector: Save a file named source.json.

     Example JSON file contents for the Queue source:

     {
         "kind": "plugin",
         "pluginName": "QueueSource",
         "configMap": {
             "queueId": "<queue_OCID>"
         }
     }

  2. Define the Functions target for the connector: Save a file named target.json.

     Example JSON file contents for the Functions target:

     {
         "kind": "functions",
         "functionId": "<function_OCID>",
         "batchSizeInKbs": "5000",
         "batchSizeInNum": "10",
         "batchTimeInSec": "60"
       }

  3. Create the connector using the oci sch service-connector create command.

     Example command, with lines added for scannability:

     oci sch service-connector create
       --compartment-id <compartment_OCID>
       --display-name "Queues to Functions"
       --source file://source.json
       --target file://target.json

• Run the CreateServiceConnector operation to create the connector.

  Example request:

  POST /20200909/serviceConnectors
  Host: service-connector-hub.us-phoenix-1.oraclecloud.com
  <authorization and other headers>
  {
    "displayName": "My Queue to Function Connector",
    "compartmentId": "<compartment_OCID>",
    "description": "My connector description",
    "source": {
      "kind": "plugin",
      "pluginName": "QueueSource",
      "configMap": {
        "queueId": "<queue_OCID>"
      }
    },
    "target": {
      "kind": "functions",
      "functionId": "<function_OCID>",
      "batchSizeInKbs": "5000",
      "batchSizeInNum": "10",
      "batchTimeInSec": "60"
    }
  }