Oracle Cloud Infrastructure Documentation
Try Free Tier

Set Up Identity and Access Management Policies

Data Flow requires common policies to be set up in Identity and Access Management (IAM) to manage and run Spark applications.

You can use the policy templates in IAM or manually create the policies. For more information on how IAM policies work, see the Identity and Access Management without Identity Domains documentation or the Identity and Access Management with Identity Domains documentation. Create the following four policies:
  • dataflow-service-level-policy
  • dataflow-admins-policy
  • dataflow-data-engineers-policy
  • dataflow-sql-users-policy

Data Flow Policy Templates

Data Flow has four Common Policy Templates. They are listed in the order in which you need to create the policies.

Let Data Flow admins manage all Applications and Runs
For administration-like users (or super users) of the service who can take any action on the service, including managing applications owned by other users and runs started by any user within their tenancy subject to the policies assigned to the group
Let Data Flow users manage their own Applications and Runs.
All other users who are only authorized to create and delete their own applications. But they can run any application within their tenancy, and have no other administrative rights such as deleting applications owned by other users or canceling runs started by other users.
Allow Data Flow service to perform actions on behalf of the user or group on objects within the tenancy.
The Data Flow service needs permission to perform actions on behalf of the user or group on objects within the tenancy.
(Optional) Allow Data Flow users to create, edit, or change private endpoints.
This policy template allows use of the virtual-network-family, allows access to more specific resources, allows access to specific operations, and allows changing of the network configuration.

Creating Policies Using IAM Policy Builder Templates

Use the IAM Policy Builder templates to create your policies for Data Flow.

Using templates in the Policy Builder in IAM without Identity Domains or with Identity Domains, follow the steps to use the Console to create a policy:
  1. From the navigation menu, select Identity & Security.
  2. Under Identity select Policies.

Manually Create Policies

Rather than using the templates in IAM to create the policies for Data Flow, you can create them yourself in IAM Policy Builder.

Following the steps in Managing Policies in IAM with Identity Domains or without Identity Domains to manually create the following policies:

Setting Up a Policy for Spark Streaming

To use Spark Streaming with Data Flow, you need more than the common policies.

You must have created the common policies either using the IAM Policy Builder templates or manually.

You can use the IAM Policy Builder to manage access to the sources and sinks your streaming applications consume from or produce to. For example, the specific stream pool or the specific Object Storage bucket which are at a location you pick. Or you can follow these steps to create a policy manually:

  1. Create a policy called dataflow-streaming-policy at the root of your tenancy.
  2. Add the following statements to allow Data Flow Runs from the dataflow-compartment compartment to consume or produce from a specific stream pool. The stream pool has an ID of stream-pool-ocid1 and an Object Storage bucket named stream-bucket-1. 
    ALLOW ANY-USER TO {STREAM_INSPECT, STREAM_READ, STREAM_CONSUME, STREAM_PRODUCE} IN TENANCY WHERE ALL
{request.principal.type='dataflowrun', request.resource.compartment.id = '<compartment_id>', target.streampool.id = 'stream-pool-ocid1'}
ALLOW ANY-USER TO MANAGE OBJECTS IN TENANCY WHERE ALL 
{request.principal.type='dataflowrun', request.resource.compartment.id = '<compartment_id>', target.bucket.name = '<bucket_name>'}

Was this article helpful?