Generic REST Data Asset
Currently, you have to upload a manifest file when you create a REST data asset. A manifest file describes the entire API of your base URL, including paths, operations, parameters, request body and response structures, and object properties. The file must not exceed 3 MB.
Before you create a REST data asset, ensure that you understand what is supported, and any current limitations.
Before You Begin
Data Integration supports OpenAPI 3.0.0 (and higher) and only JSON or YML/YAML manifest files for your REST data source.
To create a REST data asset, you have to provide the base URL of your API endpoints, and upload the manifest file.
A complete API endpoint consists of a base URL, resource path, and a server URL that is defined in the manifest file. Consider the endpoint with the path as /users/findByStatus:
https://myapi.example.com/v1/users/findByStatus
If the manifest file has the server URL as /v1, then the base URL is:
https://myapi.example.com
If the server URL is not defined in the manifest file, or the manifest file has the server URL as /, then the base URL is:
https://myapi.example.com/v1
Consider the following supported capabilities and current limitations when creating and using a REST data asset as a source in Data Integration:
- The manifest file for uploading must not exceed 3 MB.
- JSON and YML/YAML manifest files that are based on a version less than 3.0.0 are not supported.
- Only Basic authentication or no authentication is supported.
- Only
GETcalls are supported. - Header and cookies in a request body or response are not supported.
- A paginated REST response is not supported.
- A JSON response as File is not supported.
- Only JSON
content-typeis supported. - The keywords
OneOf,AnyOf,AllOf, andAdditionalPropertiesare not supported in a manifest file. - Parameter contents should be described using the keyword
schema; the content keyword is not supported. - A default value for a parameter is not supported. In the case of an optional parameter with default, a default value must be supplied explicitly.
- Only Integer, Long, Float, Double, Number and String data types are supported.
- Data formats Date, Password, Email, and Binary are ignored and treated like String only.
- Only primitive types for path parameters are supported.
- Primitive and Array of primitives are supported for query parameters.
- Array of Array in a JSON response is not supported.
- Style and explode attributes are ignored, and a query parameter is treated as style=form, explode=true while a path parameter is treated as style=simple, explode=false.
- Reference (
#ref) should be local only; it cannot point to some URL or another file. #refforschemais supported.responses,headersandparametersshould not use#ref.- Currently, the Test connection option for a REST data asset is not implemented; it always returns true (successful).
- When viewing the data asset details, data profiling or previewing of hierarchical data is not supported.
- In a response, only root level attributes are mapped to column names.
- If an endpoint returns an array of primitives, there is only one column such as "
<name>_index", whereindexis>= 0
REST Data Asset Properties
Ensure that you have the base URL and manifest file before you create your REST data asset.
To use a secret for the password with basic authentication, see OCI Vault Secrets and Oracle Wallets.
To create a data asset that connects to a REST source, complete the following properties and connection details. See Sample Manifest File.
| Field | Description |
|---|---|
| Properties | |
| Name |
Enter a name for the data asset. |
| Identifier |
The identifier is a system-generated value based on the name you specify. You can change the identifier value, but after you create and save the data asset, you cannot update the identifier. |
| Description |
Add a description. This field is optional. |
| Type | Displays the data asset type based on the icon you select in the Select data asset type panel. For more information, see Creating a Data Asset. |
| Base URL |
Enter the base URL of your API endpoints. For example: Or,
|
| Upload Manifest file |
Drop the YML/YAML or JSON file to upload the manifest file that can pass through the base URL. Alternatively, you can click the Select file link to browse to the location that has the file. If you don't provide a manifest file and the message |
| Default connection information | |
| Name |
Enter a name for the default connection. |
| Identifier |
The identifier is a system-generated value based on the name you specify. You can change the identifier value, but after you create and save the connection, you cannot update the identifier. |
| Description |
Add a description. This field is optional. |
| Authentication |
Select from:
The authentication type cannot be changed once it's created. |
| Username | Enter a user name. |
| Password or Vault Secret OCID for database password |
Select Use password or Use vault secret OCID, depending on whether you use a secret in Oracle Cloud Infrastructure Vault to store the password for the user. Then enter the password, or copy and paste the OCID of the secret for the password associated with the user. |
| (Optional) Test connection | |
| Test connection |
After you complete all the required fields, you can click Test connection to ensure you have entered the data asset details correctly. A success or failure message displays, indicating whether the test was successful or not. If the test fails, review your connection settings and try again. |
Sample Manifest File
{
"openapi": "3.0.1",
"servers": [
{
"url": "/v1",
"description": "Generated server url"
}
],
"paths": {
"/orders": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrders",
"responses": {
"200": {
"$ref": "#/components/responses/orderSuccess"
}
}
}
},
"/paginatedOrders": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getPaginatedOrdersRowOffset",
"parameters": [
{
"name": "maximumPageSize",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "firstRowNumber",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "rowOffsetParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageSizeParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PaginatedOrders"
}
}
}
}
}
}
},
"/paginatedOrdersPageNum": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getPaginatedOrdersPageNumberBased",
"parameters": [
{
"name": "maximumPageSize",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "firstPageNumber",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageSizeParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageNumberParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PaginatedOrders"
}
}
}
}
}
}
},
"/orders/{orderNo}": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByOrderID",
"parameters": [
{
"$ref": "#/components/parameters/orderNum"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
},
"delete": {
"tags": [
"rest-connector-controller"
],
"operationId": "deleteOrder",
"parameters": [
{
"name": "orderNo",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/orders/noAuth/{orderNo}": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByOrderIDNoAuth",
"parameters": [
{
"name": "orderNo",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
},
"/orders/headers/validation": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersAfterHeadersValidation",
"parameters": [
{
"name": "UserId",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "Password",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "Email",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SalesOrders"
}
}
}
}
}
}
},
"/orders/filterbystore": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByStore",
"parameters": [
{
"$ref": "#/components/parameters/storeId"
},
{
"name": "customerName",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/orders/byName": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByCustomerName",
"parameters": [
{
"name": "customerName",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/orders/byEmail": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByCustomerEmail",
"parameters": [
{
"name": "customerEmail",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/customers/getCustomersName": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomersName",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
},
"/customers/getCustomer": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomer",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Customer"
}
}
}
}
}
}
},
"/customers/getCustomerList": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomersList",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Customer"
}
}
}
}
}
}
}
},
"/customers/getCustomerEmail": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomerInfoSingleField",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"customerEmail": {
"type": "string"
}
}
}
}
}
}
}
}
},
"/headOffice": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getHeadOffice",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HeadOffice"
}
}
}
}
}
}
},
"/namedPrimitiveArray": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getNamedPrimitiveArray",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
}
}
},
"/primitive": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getSinglePrimitive",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "string"
}
}
}
}
}
}
},
"/reservedColumns": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getReservedColumns",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Positions"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"Address": {
"type": "object",
"properties": {
"street": {
"type": "string"
},
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"zipcode": {
"type": "string"
}
}
},
"Fullfilment": {
"type": "object",
"properties": {
"storeNo": {
"type": "string"
},
"lineItems": {
"type": "array",
"items": {
"$ref": "#/components/schemas/LineItems"
}
}
}
},
"LineItems": {
"type": "object",
"properties": {
"itemName": {
"type": "string"
},
"itemId": {
"type": "string"
},
"qty": {
"type": "string"
}
}
},
"Order": {
"type": "object",
"properties": {
"customerName": {
"type": "string"
},
"orderNo": {
"type": "string"
},
"phoneNo": {
"type": "string"
},
"email": {
"type": "string"
},
"address": {
"$ref": "#/components/schemas/Address"
},
"fullfilments": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Fullfilment"
}
}
}
},
"PaginatedOrders": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
},
"SalesOrders": {
"type": "object",
"properties": {
"orders": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
},
"Customer": {
"type": "object",
"properties": {
"customerEmail": {
"type": "string"
},
"type": {
"type": "string"
},
"customerName": {
"type": "string"
},
"phoneno": {
"type": "string"
},
"address": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
},
"HeadOffice": {
"type": "object",
"properties": {
"longitude": {
"type": "number"
},
"latitude": {
"type": "number",
"format": "double"
},
"turnover": {
"type": "number",
"format": "float"
},
"milliSeconds": {
"type": "integer",
"format": "int64"
},
"storeNo": {
"type": "integer",
"format": "int32"
},
"noOfEmployees": {
"type": "integer"
}
}
},
"Positions": {
"type": "object",
"properties": {
"position": {
"type": "array",
"items": {
"type": "object",
"properties": {
"Position_1": {
"type": "integer"
},
"Position": {
"type": "string"
},
"Position_3": {
"type": "number",
"format": "float"
},
"Position_4": {
"type": "number",
"format": "double"
},
"Position_2": {
"type": "integer",
"format": "int64"
},
"Position_11": {
"type": "string"
}
}
}
}
}
}
},
"parameters": {
"orderNum": {
"name": "orderNo",
"in": "path",
"required": true,
"content": {
"application/json": {
"schema": {
"type": "string"
}
}
}
},
"storeId": {
"name": "storeId",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
},
"responses": {
"orderSuccess": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SalesOrders"
}
}
}
}
}
}
}