Data Flow now Supports Structured Spark Streaming

  • Services: Data Flow
  • Release Date: Jan. 26, 2022

Streaming applications require continuous execution for a long period of time that often extends beyond 24 hours, and might be as long as weeks or even months.

Data Flow:

  • Implements a graceful shutdown of the Spark clusters for Spark structured streaming to allow for regular maintenance.
  • Uses Oracle Cloud Infrastructure Streaming.
  • Uses Spark structured streaming check-pointing to restart from the point of failure without producing incorrect computational results.
  • Provides access to the Spark UI and Spark History Server.

For more information see the Spark Streaming chapter in the Data Flow documentation.