Spark Dynamic Allocation is now Available with Data Flow

  • Services: Data Flow
  • Release Date: June 30, 2022

To help you save resources and reduce time on management, Spark Dynamic Allocation is now enabled in Data Flow.. You can define a Data Flow cluster based on a range of executors, instead of just a fixed number of executors. Spark provides a mechanism to dynamically adjust the resources application occupies based on the workload.
For more information, see the documentation on Spark Dynamic Allocation with Data Flow.