Generic REST Data Asset
Currently, you have to upload a manifest file when you create a REST data asset. A manifest file describes the entire API of your base URL, including paths, operations, parameters, request body and response structures, and object properties. The file must not exceed 3 MB.
Before you create a REST data asset, ensure that you understand what is supported, and any current limitations.
Before You Begin
Data Integration supports OpenAPI 3.0.0 (and higher) and only JSON or YML/YAML manifest files for your REST data source.
To create a REST data asset, you have to provide the base URL of your API endpoints, and upload the manifest file.
A complete API endpoint consists of a base URL, resource path, and a server URL that is defined in the manifest file. Consider the endpoint with the path as /users/findByStatus:
https://myapi.example.com/v1/users/findByStatus
If the manifest file has the server URL as /v1, then the base URL is:
https://myapi.example.com
If the server URL is not defined in the manifest file, or the manifest file has the server URL as /, then the base URL is:
https://myapi.example.com/v1
Consider the following supported capabilities and current limitations when creating and using a REST data asset as a source in Data Integration:
- The manifest file for uploading must not exceed 3 MB.
- JSON and YML/YAML manifest files that are based on a version less than 3.0.0 are not supported.
- OAuth 2.0, Basic, and no authentication are supported.
- Only
GETcalls are supported. - Header and cookies in a request body or response are not supported.
- A JSON response as File is not supported.
- Only JSON
content-typeis supported. - The keywords
OneOf,AnyOf,AllOf, andAdditionalPropertiesare not supported in a manifest file. - Parameter contents should be described using the keyword
schema; the content keyword is not supported. - A default value for a parameter is not supported. In the case of an optional parameter with default, a default value must be supplied explicitly.
- Only Integer, Long, Float, Double, Number and String data types are supported.
- Data formats Date, Password, Email, and Binary are ignored and treated like String only.
- Only primitive types for path parameters are supported.
- Primitive and Array of primitives are supported for query parameters.
- Array of Array in a JSON response is not supported.
- Style and explode attributes are ignored, and a query parameter is treated as style=form, explode=true while a path parameter is treated as style=simple, explode=false.
- Reference (
#ref) should be local only; it cannot point to some URL or another file. #refforschemais supported.responses,headersandparametersshould not use#ref.- Currently, the Test connection option for a REST data asset is not implemented; it always returns true (successful).
- When viewing the data asset details, data profiling or previewing of hierarchical data is not supported.
- In a response, only root level attributes are mapped to column names.
- If an endpoint returns an array of primitives, there is only one column such as "
<name>_index", whereindexis>= 0
REST Data Asset Properties
Ensure that you have the base URL and manifest file before you create your REST data asset.
To use a secret for the password with basic authentication, see OCI Vault Secrets and Oracle Wallets.
To create a data asset that connects to a REST source, complete the following properties and connection details. See Sample Manifest File.
| Field | Description |
|---|---|
| Properties | |
| Name | Enter a name for the data asset. |
| Identifier | The identifier is a system-generated value based on the name. You can change the identifier value, but after you create and save, you cannot update the identifier. |
| Description | (Optional) Add a description. |
| Type | The field shows the data source type that you selected in the Select data asset type panel (as described in Creating a Data Asset). Data Integration displays the data asset property fields and default connection fields based on this type. |
| Base URL |
Enter the base URL of your API endpoints. For example: Or,
|
| Upload Manifest file |
Drop the YML/YAML or JSON file to upload the manifest file that can pass through the base URL. Alternatively, you can click the Select file link to browse to the location that has the file. If you don't provide a manifest file and the message |
| Default connection information | |
| Name | Enter a name for the default connection. |
| Identifier | The identifier is a system-generated value based on the name. You can change the identifier value, but after you create and save, you cannot update the identifier. |
| Description | (Optional) Add a description. |
| Authentication |
Select one of the following authentication methods:
The authentication type for a connection cannot be changed after it's created. |
| (Optional) Test connection | |
| Test connection | After you complete all the required fields, click Test connection to ensure that the data asset details have been entered correctly and the connection can be established. A success message appears if the test is successful. If the test fails with an error message, review the connection settings and try again. |
Sample Manifest File
{
"openapi": "3.0.1",
"servers": [
{
"url": "/v1",
"description": "Generated server url"
}
],
"paths": {
"/orders": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrders",
"responses": {
"200": {
"$ref": "#/components/responses/orderSuccess"
}
}
}
},
"/paginatedOrders": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getPaginatedOrdersRowOffset",
"parameters": [
{
"name": "maximumPageSize",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "firstRowNumber",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "rowOffsetParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageSizeParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PaginatedOrders"
}
}
}
}
}
}
},
"/paginatedOrdersPageNum": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getPaginatedOrdersPageNumberBased",
"parameters": [
{
"name": "maximumPageSize",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "firstPageNumber",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageSizeParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
},
{
"name": "pageNumberParameter",
"in": "query",
"required": false,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PaginatedOrders"
}
}
}
}
}
}
},
"/orders/{orderNo}": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByOrderID",
"parameters": [
{
"$ref": "#/components/parameters/orderNum"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
},
"delete": {
"tags": [
"rest-connector-controller"
],
"operationId": "deleteOrder",
"parameters": [
{
"name": "orderNo",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/orders/noAuth/{orderNo}": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByOrderIDNoAuth",
"parameters": [
{
"name": "orderNo",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
},
"/orders/headers/validation": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersAfterHeadersValidation",
"parameters": [
{
"name": "UserId",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "Password",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
},
{
"name": "Email",
"in": "header",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SalesOrders"
}
}
}
}
}
}
},
"/orders/filterbystore": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByStore",
"parameters": [
{
"$ref": "#/components/parameters/storeId"
},
{
"name": "customerName",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/orders/byName": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByCustomerName",
"parameters": [
{
"name": "customerName",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/orders/byEmail": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getOrdersByCustomerEmail",
"parameters": [
{
"name": "customerEmail",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
}
}
}
}
},
"/customers/getCustomersName": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomersName",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
},
"/customers/getCustomer": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomer",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Customer"
}
}
}
}
}
}
},
"/customers/getCustomerList": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomersList",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Customer"
}
}
}
}
}
}
}
},
"/customers/getCustomerEmail": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getCustomerInfoSingleField",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"customerEmail": {
"type": "string"
}
}
}
}
}
}
}
}
},
"/headOffice": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getHeadOffice",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HeadOffice"
}
}
}
}
}
}
},
"/namedPrimitiveArray": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getNamedPrimitiveArray",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
}
}
},
"/primitive": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getSinglePrimitive",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "string"
}
}
}
}
}
}
},
"/reservedColumns": {
"get": {
"tags": [
"rest-connector-controller"
],
"operationId": "getReservedColumns",
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Positions"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"Address": {
"type": "object",
"properties": {
"street": {
"type": "string"
},
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"zipcode": {
"type": "string"
}
}
},
"Fullfilment": {
"type": "object",
"properties": {
"storeNo": {
"type": "string"
},
"lineItems": {
"type": "array",
"items": {
"$ref": "#/components/schemas/LineItems"
}
}
}
},
"LineItems": {
"type": "object",
"properties": {
"itemName": {
"type": "string"
},
"itemId": {
"type": "string"
},
"qty": {
"type": "string"
}
}
},
"Order": {
"type": "object",
"properties": {
"customerName": {
"type": "string"
},
"orderNo": {
"type": "string"
},
"phoneNo": {
"type": "string"
},
"email": {
"type": "string"
},
"address": {
"$ref": "#/components/schemas/Address"
},
"fullfilments": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Fullfilment"
}
}
}
},
"PaginatedOrders": {
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
},
"SalesOrders": {
"type": "object",
"properties": {
"orders": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Order"
}
}
}
},
"Customer": {
"type": "object",
"properties": {
"customerEmail": {
"type": "string"
},
"type": {
"type": "string"
},
"customerName": {
"type": "string"
},
"phoneno": {
"type": "string"
},
"address": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
},
"HeadOffice": {
"type": "object",
"properties": {
"longitude": {
"type": "number"
},
"latitude": {
"type": "number",
"format": "double"
},
"turnover": {
"type": "number",
"format": "float"
},
"milliSeconds": {
"type": "integer",
"format": "int64"
},
"storeNo": {
"type": "integer",
"format": "int32"
},
"noOfEmployees": {
"type": "integer"
}
}
},
"Positions": {
"type": "object",
"properties": {
"position": {
"type": "array",
"items": {
"type": "object",
"properties": {
"Position_1": {
"type": "integer"
},
"Position": {
"type": "string"
},
"Position_3": {
"type": "number",
"format": "float"
},
"Position_4": {
"type": "number",
"format": "double"
},
"Position_2": {
"type": "integer",
"format": "int64"
},
"Position_11": {
"type": "string"
}
}
}
}
}
}
},
"parameters": {
"orderNum": {
"name": "orderNo",
"in": "path",
"required": true,
"content": {
"application/json": {
"schema": {
"type": "string"
}
}
}
},
"storeId": {
"name": "storeId",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
},
"responses": {
"orderSuccess": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SalesOrders"
}
}
}
}
}
}
}