Oracle Cloud Infrastructure Documentation

About Data Refresh Performance

Oracle strives constantly to improve performance for data loading in pipelines.

The performance of loading data for your instance will vary. The time to complete data processing, both full warehouse loads and incremental data loads, depends on various factors. A data pipeline load includes the following:

  • Extracting data from the Oracle Fusion Cloud Applications.
  • Loading the data into Oracle Autonomous Data Warehouse.
  • Transforming the data into the prebuilt schema.
The time to complete each of these steps is determined by various factors including:
  • The availability of the source system.
  • The size and complexity of the source data.
  • The activated functional areas.
  • Custom SQL queries that impact Oracle Autonomous Data Warehouse.
  • Your queries running concurrently.
  • Customizations made on the source system objects (which require a full load for those objects).
  • The patching of source and target systems.
For daily refresh of data and pipeline execution to update all the resulting key metrics and dashboards, the execution time depends on many factors such as the amount of data getting incrementally refreshed. In order to plan your downstream workflow needs, ensure to perform these actions:
  • Turn on the pipeline execution estimation using the Data Refresh Estimate preview feature on the Enable Features page. The estimated refresh completion details are visible on the Pipeline Settings page.
  • Trigger any subsequent operations based on the estimated refresh completion time.
  • Reach out to Oracle Support if the refresh isn't completed by the estimated completion time.
  • Subscribe to Event Notifications, so you can receive an email notification as soon as there is any issue with the pipeline execution. See Enable Event Notifications (Preview).

Data refresh is typically completed daily unless the data pipeline jobs are turned off or stuck. You can observe data loading times for your specific source and warehouse configuration to estimate the time it takes for an incremental daily refresh and for a full warehouse load. This information can help you plan for the optimal time in the day to start your daily data refresh. You may want to schedule the data load to run during off-peak hours, for example, run initial full warehouse loads during weekends and incremental loads during weeknights to ensure that users aren't impacted.

You can view the estimated refresh completion time for daily pipelines on the Pipeline Settings page in the Estimated Refresh Completion field as a Preview feature. This enables you to plan your tasks in the application.
Estimated Refresh Completion details on the Pipeline Parameters page

The incremental data refresh process refreshes objects as base and derived datasets. The base datasets are objects that are part of the core pipeline and sourced directly from Oracle Fusion Cloud Applications and needed for core content (for example, transactions and dimensions). The derived datasets are based on data that require additional processing (for example, datasets used in prebuilt machine learning use cases). The incremental refresh process refreshes the base datasets first to ensure essential data is made available quickly. The system predicts and displays the incremental load completion time but doesn't do the same for derived datasets. If you haven't activated content that has derived datasets, then you won't see this information on the Pipeline Settings page in the Data Pipeline section.