Troubleshooting Connectors
Use troubleshooting information to identify and address common issues that can occur while working with Connector Hub.
See also Known Issues for Connector Hub.
No Data is Being Moved
Troubleshoot issues with moving data through connectors.
The connector isn't moving data. For example connectors, see Connector Hub Scenarios.
Cause: Data doesn't exist
The source doesn't contain data matching the query in the connector's source configuration.
To find out if data exists at the source, do one of the following:
- Get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates a successful connector run, including the amount of data moved:
Service connector run succeeded - <number> messages (<number> bytes) written to target
- For source logs, search the logs using the query from the connector's source configuration.
- For source metrics, query the metrics.
- For source streams, query the stream's default producer metrics.
Remedy: Update the connector
Cause: Service-level error
An error at the source service, target service, or Connector Hub service is preventing movement of data.
To find out if a service-level error exists, do one of the following:
- Get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates a service-level error:
Connector run failed due to <type> error, Error Code : XXX <exception>
- View the following metric charts for the connector:
- Errors at source
- Errors at target
- Connector Hub errors
Remedy: Update the connector
Remedy: Contact support
5xx
or -1
in Connector Hub errors, then contact support. See Getting Help and Contacting Support.Cause: Missing authorization
You don't have authorization to write to the target service.
To find out if authorization is missing, get service logs for the connector. (If needed, enable logs first.) Following is an example log message that indicates missing authorization:
Connector run failed due to <type> error, Error Code : 404 NotAuthorizedOrNotFound
Remedy: Get authorization
Your accepted default policies might take a few minutes to propagate to regions that aren't your home region. The connector doesn't move data until the policies are propagated.
Deactivation for Unknown Reasons
Troubleshoot a deactivated connector.
A connector's status is Deactivated
and you didn't deactivate it.
Someone Else Deactivated the Connector
The connector was deactivated by someone else:
- Another user at your organization
- Oracle Cloud Infrastructure
For certain failure conditions, a connector that continuously fails is automatically deactivated by the service team at Oracle Cloud Infrastructure. Such a long-term continuous failure can indicate invalid configuration of the connector's source or target.
Reactivate the Connector
- Update the connector to ensure valid configuration of its source and target.
- (Optional) To get details on the data flow from a connector's source to its target, enable logs for the connector.
- Reactivate the connector.
- Confirm that the connector successfully moves data by checking for expected results at the target service.
Announcement: Connector Deactivated
A console announcement states "Connector Deactivated."
Cause: For certain failure conditions, a connector that continuously fails is automatically deactivated by the service team at Oracle Cloud Infrastructure. Such a long-term continuous failure can indicate invalid configuration of the connector's source or target.
Reactivate the Connector
- Update the connector to ensure valid configuration of its source and target.
- (Optional) To get details on the data flow from a connector's source to its target, enable logs for the connector.
- Reactivate the connector.
- Confirm that the connector successfully moves data by checking for expected results at the target service.
Announcement: Action Required to Prevent Connector Deactivation
A console announcement states "Action Required to Prevent Connector Deactivation."
Cause: The connector is consistently failing.
Can't View Query in Basic Mode
Troubleshoot issues viewing the query in Basic mode in the Console.
The Console doesn't allow selection of Basic mode after you edit the query in Advanced mode.
Cause: The query is too complex
The query includes one or more of the following items not supported in Basic mode:
OR
operator (except with type-based filters when you only use Audit logs)- Functions (for example:
isNull()
) select
summarize
Remedy: Update the query
Update the query so that it only includes elements supported in Basic mode:
- Audit logs only: Type-based filters can use the
OR
operator. Other filters must use theAND
operator.Example:((type = value1 OR type = value2) AND field = value3 AND field1 = value4)
- Any combination of logs (Service logs, custom logs, and Audit logs): Filters joined with the
AND
operator.Example:(field = value AND field1 != value1)
How Do I Know When Issues Occur?
Identify when issues occur with connectors.
Look for the following indicators of issues with connectors.
- Data freshness for a single connector: Look for unexpected lapses of time between data movement.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Connector Hub.
- Choose a Compartment.
- Select the name of the connector that you want.
- Under Resources, select Metrics.
- Review the Data freshness metric chart.
- Data freshness across connectors: Look for unexpected lapses of time between data movement.
- Open the navigation menu and click Observability & Management. Under Monitoring, click Service Metrics.
- Choose the Compartment that contains the connectors you want to view data freshness for.
-
For Metric namespace, select oci_service_connector_hub.
- Review the following metric charts:
- Data freshness
-
Logging source: If the connector retrieves data from a log, then it might be attempting more than the maximum amount of hourly retrieval of data per connector (1 GB). Log data at the target isn't delivered if this issue continues to occur past 24 hours (the maximum duration for catching missed data in previous transmissions by the connector). To determine if this issue is occurring, create alarms to monitor the following indicators.
Indicator (Metric) Alarm query in MQL, with comments Data older than 12 hours (Data freshness) DataFreshness[1h].mean() > 43200000
Comments:
- The value
43200000
is the number of milliseconds in 12 hours. - Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
Error at source (any error) (Errors at source) ErrorsAtSource[15m].groupby(errorCode,connectorId).min() > 0
Comments:
- Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
- Results are grouped by error code and connector.
Internal errors at source that don't resolve after 15 minutes (5xx) (Errors at source) ErrorsAtSource[15m]{errorCode =~ "5*"}.groupby(connectorId).sum() > 0 && ErrorsAtSource[15m].groupby(connectorId).min() > 0
Comments:
- Internal errors might indicate an issue at the source, which could delay delivery of data.
- To trigger the alarm at shorter intervals, change the interval ([15m]).
- Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
Throttling errors at source (429) (Errors at source) ErrorsAtSource[15m]{errorCode = "429"}.groupby(connectorId).sum() >0 && ErrorsAtSource[15m].groupby(connectorId).min() > 0
Comments:
- For more information on throttling errors, see documented limits for the relevant service.
- For example, for throttling errors related to the Streaming source, see Limits on Streaming Resources. Throttling at the Streaming source occurs when a connector attempts to read a stream from a partition, other calls to the same partition are also occurring, and the number of calls exceeds service limits.
- Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
Service communication errors at source (-1) (Errors at source) ErrorsAtSource[15m]{errorCode = "-1"}.groupby(connectorId).sum() >0 && ErrorsAtSource[15m].groupby(connectorId).min() > 0
404 error at source (Errors at source) ErrorsAtSource[15m]{errorCode = "404"}.groupby(connectorId).sum() >0
Comments:
- Either the connector can't access the log or it doesn't exist.
- Ensure that policies exist for accessing the specified log. See Access to Source, Task, and Target Services.
- Confirm that the specified log exists by searching for it in Logging.
- Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
Zero (0) bytes read (when data is expected) (Bytes read from source) BytesReadFromSource[15m].groupby(connectorId).sum() == 0
Comments:
- If errors aren't occurring at source, target, or task, then the log might not exist. Confirm that the specified log exists by searching for it in Logging.
- Ignore occasional failures. We recommend setting the alarm trigger delay to 30 minutes or more. With this configuration, the alarm only alerts you when multiple consecutive failures occur over the specified time period.
- The value