Scenario: Sending Log Data to an Autonomous Database
Send log data from Logging to an Autonomous Database using a function (Functions service).
This scenario involves creating a function and then referencing that function in a connector (Connector Hub) to process and move log data from Logging to an Autonomous Database.
Required IAM Policy
If you're a member of the Administrators group, you already have the required access to execute this scenario. Otherwise, you need access to Functions.
The workflow for creating the connector includes a default policy when needed to provide permission for writing to the target service. If you're new to policies, see Getting Started with Policies and Common Policies.
Setting Up This Scenario
This scenario involves creating a connector (Connector Hub) to send log data from Logging to an Autonomous Database (JSON).
Before you can create the connector, you must set up an Autonomous JSON Database that you want to receive the log data and set up the function to copy that log data.
Autonomous JSON Database setup details:
- Provision Autonomous Database
- Duplicate the browser tab.
- In the duplicated tab, copy the ORDS base URL: From the details page for the Autonomous Database, select Service Console, select Development, and then select Copy under RESTful Services and SODA.'
- Create a "logs" collection to store the log data that will be moved by the function and connector:
-
Go back to the details page for the Autonomous Database (the original browser tab).
-
Select Database Actions.
-
Log in with the
adminuser and the password you set when you created the database.The Database Actions | Launchpad window is displayed.
-
Select SQL.
The Database Actions | SQL window is displayed.
-
Enter the following command:
soda create logs -
Select
(Run Statement).
To query documents in the collection after the connector copies log data, enter the following command:
soda get logs -f {} -
Function setup details:
- Create the function
- Deploy the function
- Set the configuration values as follows:
ordsbaseurlUse the ORDS base URL you copied during setup of the database.
dbschemadbuserdbpwdcollection
For more information, see Passing Custom Configuration Parameters to Functions.
- (Optional) Invoke the function
Once the database and function are set up, you're ready to create the connector. Creating the connector is easy in the Console. Alternatively, you can use the Oracle Cloud Infrastructure CLI or API, which lets you execute the individual operations yourself.
The following code sample is for a function to send log data from the Logging service to an Autonomous Database. For instructions on creating and deploying functions, see Creating and Deploying Functions.
The following code sample isn't meant for production workloads. Update it for your production environment.
import io
import json
import logging
import requests
from fdk import response
# soda_insert uses the Autonomous Database REST API to insert JSON documents
def soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries):
auth=(dbuser, dbpwd)
sodaurl = ordsbaseurl + dbschema + '/soda/latest/'
bulkinserturl = sodaurl + 'custom-actions/insert/' + collection + "/"
headers = {'Content-Type': 'application/json'}
resp = requests.post(bulkinserturl, auth=auth, headers=headers, data=json.dumps(logentries))
return resp.json()
def handler(ctx, data: io.BytesIO=None):
logger = logging.getLogger()
logger.info("function start")
# Retrieving the Function configuration values
try:
cfg = dict(ctx.Config())
ordsbaseurl = cfg["ordsbaseurl"]
dbschema = cfg["dbschema"]
dbuser = cfg["dbuser"]
dbpwd = cfg["dbpwd"]
collection = cfg["collection"]
except:
logger.error('Missing configuration keys: ordsbaseurl, dbschema, dbuser, dbpwd and collection')
raise
# Retrieving the log entries from Connector Hub as part of the Function payload
try:
logentries = json.loads(data.getvalue())
if not isinstance(logentries, list):
raise ValueError
except:
logger.error('Invalid payload')
raise
# The log entries are in a list of dictionaries. We can iterate over the the list of entries and process them.
# For example, we are going to put the Id of the log entries in the function execution log
logger.info("Processing the following LogIds:")
for logentry in logentries:
logger.info(logentry["oracle"]["logid"])
# Now, we are inserting the log entries in the JSON Database
resp = soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries)
logger.info(resp)
if "items" in resp:
logger.info("Logs are successfully inserted")
logger.info(json.dumps(resp))
else:
raise Exception("Error while inserting logs into the database: " + json.dumps(resp))
# The function is done. Return empty response.
logger.info("function end")
return response.Response(
ctx,
response_data="",
headers={"Content-Type": "application/json"}
)Using the Console
This section walks through creating a connector using the Console. Your function must be deployed.
For help with troubleshooting, see Troubleshooting Connectors.
This example walks through using the Console to create a connector. In this example, the connector moves log data from Logging to an Autonomous Database using the function you created using the function code sample.
- Open the navigation menu and select Analytics & AI. Under Messaging, select Connector Hub.
- Choose the Compartment where you want to create the service connector.
- Select Create connector.
-
On the Create connector page:
- Type a Connector name such as "Send Logs to My Autonomous Database." Avoid entering confidential information.
- Select the Resource compartment where you want to store the new connector.
- Under Configure connector, select your source and target services to move log data to a metric:
- Source: Logging
- Target: Functions
- Under Configure source connection, select a Compartment name, Log group, and Log.
- Under Configure target connection, select the Function application and Function corresponding to the function you created using the function code sample.
- If prompted to create a policy (required for access to create or update a service connector), select Create.
- Select Create.
Using the CLI
This section walks through creating a connector using the CLI that moves log data to your function (which then moves the data to an Autonomous Database).
For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.
-
Create a connector: Open a command prompt and run the
oci sch service-connector createcommand:oci sch service-connector create --display-name "<display_name>" --compartment-id <compartment_OCID> --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]
For help with troubleshooting, see Troubleshooting Connectors.
Using the API
This section walks through creating the connector using the API.
For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.
Use the following operations:
-
CreateServiceConnector: Create a connector that moves log data to your function (which then moves the data to an Autonomous Database).
Example CreateServiceConnector requestPOST /20200909/serviceConnectors Host: service-connector-hub.us-phoenix-1.oraclecloud.com <authorization and other headers> { "compartmentId": "<compartment_OCID>", "description": "My connector description", "displayName": "My Connector", "source": { "kind": "logging", "logSources": [ { "compartmentId": "<compartment_OCID>", "logGroupId": "<log_group_OCID>", "logId": "<log_OCID>" } ] }, "target": { "compartmentId": "<compartment_OCID>", "kind": "functions", "functionId": "<function_OCID>" }, "tasks": [] } }
For help with troubleshooting, see Troubleshooting Connectors.