Data Lineage

Use the Data Lineage feature to enable the capture of end-to-end transformation lifecycle of data pipelines in Data Flow.

When enabled, Data Flow records the corresponding Spark job's lineage metadata and uploads it in to Data Catalog, where it can be anaylzed further. For information on enabling data lineage collection in a Data Flow application, see the Create Applications section.

For more information, see the Data Lineage overview in the the Data Catalog documentation.