mlm_insights.core.readers.interfaces package

Submodules

mlm_insights.core.readers.interfaces.data_reader module

class mlm_insights.core.readers.interfaces.data_reader.DataReader(name: str, engine: str, **kwargs: Any)

Bases: ABC

Abstract Base Class for all Insights readers. The child class must implement read and make_schema methods.

Note: Do not derive from this class directly. Instead, use Execution Engine specific reader base classes.

classmethod create(config: Dict[str, Any]) → DataReader

get_engine() → str

Returns the name of engine which was set during initialization.

Returns

engine_name: str

Name of engine, eg ‘dask’, ‘spark’

get_name() → str

Returns the name of reader which was set during initialization.

Returns

reader_name: str

Name of the reader

abstract read(schema_provider: SchemaProvider, **kwargs: Any) → Any

Read the data from the data-source and return engine specific data frame.

Parameters

schema_providerSchemaProvider

Engine specific schema provider. Use it to call get_engine_schema() to get engine specific schema to pass to the respective read methods

kwargs:

Key-value pair for dynamic arguments

Returns

dataframe: Any

Engine specific DataFrame