Working with REST Tasks
A REST task lets you call a REST API endpoint to perform an operation.
To use the Oracle Cloud Infrastructure Resource Principal to authenticate an API endpoint in a REST task, see Authentication.
As described in Parts of a REST Task, create a REST task in Data Integration by defining the REST request (HTTP method, URL, and request header) and the completion criteria (success condition) that must be satisfied for a successful invocation of the REST task.
After a REST request is sent, by default Data Integration waits for the call to return the response before proceeding to other tasks (synchronous API mode). The REST invocation completes when the response is received for the REST request. Currently, the supported content type for the API response is JSON.
If the REST request invokes an operation that's long running, specify a polling condition and an optional termination configuration to define the completion of the REST invocation. When polling is configured, Data Integration does not wait for the REST call to return the response before proceeding to other tasks (asynchronous API mode). For more information, see Polling in Long-Running Operations and Termination in Long-Running Operations.
The maximum size supported for a response returned from a REST task is 512K. Larger responses are truncated.
The following pages describe how you can create, edit, and delete REST tasks:
- Creating a REST Task
- Providing Authentication in the REST Task
- Viewing and Managing Parameters in a REST Task
- Validating a REST Task
- Editing a REST Task
- Deleting a REST Task
The following pages describe other management tasks that can be performed after a REST task is created:
Parts of a REST Task
To invoke a REST API endpoint, provide the information for executing and completing the REST request in a REST task.
Creating a REST task involves the following main steps:
Define execution details: Define the execution details for the REST request, including the HTTP method, URL, and request header.
The URL is the REST server to contact and the relative path to the resource that you want to access in the request. For example:
http://myserver.com:8081/workspaces/finance1234/dataAssets
For a POST or PUT method, you also provide the request body. Currently, only JSON is the supported format for a request body.
Defining execution details in Data Integration is the same for short-running API operations and long-running API operations.
Specify completion criteria: Specify the success condition criteria that must be satisfied to consider the REST task a successful completion.
Write the condition that evaluates to true or false using the response from the REST call in the execution step. If the condition returns true, the outcome of the REST task is successful. If false, the outcome is a failed REST task.
Data Integration provides a default success condition that you can use or edit.
Specifying completion criteria for a long-running API operation requires adding a polling condition and other polling values. Adding a termination request is optional.
See also Parameters in a REST task, and what you need to set up before you use the Oracle Cloud Infrastructure Resource Principal to authenticate an API endpoint in a REST task.
Polling in Long-Running Operations
For a long-running REST API operation, in addition to a success condition, you must specify a polling URL and condition, and polling interval and timeout values to identify the completion of the REST invocation.
The polling configuration is used to periodically poll the status of the REST call that's invoked at the execution step.
In polling, the success condition is an expression that's written on the response of the polling request. The polling call evaluates the polling condition to determine whether polling stops or continues. Data Integration issues a polling call repeatedly at the specified polling interval until the value for the specified polling timeout is reached, or until the polling condition returns false, whichever occurs first.
The polling interval is the length of time to wait before sending the next polling request. The polling timeout is the maximum length of time that's allowed for repeated polling to occur at the specified interval rate. The interval value must be less than the timeout value.
Suppose that the polling timeout is 24 hours, and the polling interval is 1 hour. After the REST request at the execution step is invoked, the polling REST request is sent at 1-hour intervals, but only for up to 24 hours, or until the polling condition returns false.
If the criteria specified in the success condition is satisfied, the REST call is completed successfully, and the REST task run is considered a success. If the success condition returns false, the outcome of the REST task run is considered as failed.
For example, suppose the polling configuration polls the progress of a task run by evaluating the status in the response:
cast(json_path(SYS.RESPONSE_PAYLOAD_JSON, '$.status') as String) != 'SUCCESS' or cast(json_path(SYS.RESPONSE_PAYLOAD_JSON, '$.status') as String) != 'ERROR'
Polling stops when the success condition is met.
cast(json_path(SYS.RESPONSE_PAYLOAD_JSON, '$.status') as String) == 'SUCCESS'
Termination in Long-Running Operations
Configuring the HTTP method and REST URL to terminate the REST call is optional for a long-running API operation.
Defining the details for the REST termination request in Data Integration is similar to defining the REST request in the execution step. You can, however, include expressions that use the API response from the REST execution step in the termination URL.
If you terminate a REST task run, and a termination configuration is not provided in that REST task, Data Integration:
- Stops the REST task run
- Does not stop the underlying API operation invoked by the REST task
If a termination configuration is provided, then Data Integration stops the REST task run and the underlying API operation.
Authentication
Provide the authentication for invoking the REST API endpoint in the REST task.
Currently, you can use Oracle Cloud Infrastructure resource principal as the method to authenticate Oracle Cloud Infrastructure API endpoints. You can use workspace or application resource principal.
Resource Principal Authentication
The Oracle Cloud Infrastructure resource principal can be specified as the authentication method for executing an Oracle Cloud Infrastructure API endpoint in a REST task.
A resource principal enables resources to be authorized principal actors that can perform actions on service resources. The Oracle Cloud Infrastructure resource principal authentication method uses a temporary resource provider session token (RPST) for authentication, and authorization is established through policies in Oracle Cloud Infrastructure Identity and Access Management (IAM) with Identity Domains.
IAM enables you to write policies for resource principals which are part of dynamic groups. Before you use the OCI resource principal to authenticate OCI API endpoints in REST tasks, create a dynamic group and grant permission to the Data Integration workspace or application, and network resources.
The following are examples of policies that you can use.
-
To define the resources that belong to the dynamic group, create the following matching rule in the dynamic group:
ALL{resource.type = 'disworkspace', resource.compartment.id = <compartment-ocid>}
-
Add the following policy statement that permits the dynamic group access to network resources.
allow dynamic-group <dynamic-group-name> to use virtual-network-family in tenancy
-
To use the Oracle Cloud Infrastructure application resource principal to authenticate an API endpoint in a REST task:
allow any-user to use ai-service-language-family in tenancy where ALL {request.principal.type = 'disapplication', request.principal.id = '<disapplication-ocid>'}
-
The policy statements provided in this topic are examples only. Ensure that you write policies that meet your own requirements.
-
After you add IAM components (for example, dynamic groups and policy statements), don't try to perform the associated tasks immediately. New IAM policies require about five to 10 minutes to take effect.
Parameters
You can include parameters in a REST task, so that you can reuse the task in different runtime situations.
You can use parameters for the following parts of the REST request in a REST task:
- One or more sections of the REST execution URL, polling URL, or termination URL. In Data Integration, the parameterized variable sections of the URL are known as URL parameters. See Configuring the HTTP method, URL, and URL Parameters.
- The value of a header in the request. See Parameterizing a Header Value.
- The entire request body in a POST or PUT request. See Parameterizing the Request Body.
- The success condition that determines the criteria for a successful completion of the REST call. See Parameterizing the Success Condition.
- The polling condition (written on the response that's returned from the polling request) that determines whether polling stops or continues. See Parameterizing the Polling Condition.