Using the Marini Integration Platform, ETL pipelines of any complexity can be built.
The DataEngine as a component of the Marini Integration Platform has numerous transformation functions:
- Data Validation
- Data Segmentation
- Data Harmonization
- Data Consolidation
- Data Cleaning
- Data Enrichment
- Data Routing
Basic ETL Pipeline
You can manage processes and make all relevant data available in your systems by synchronizing all relevant systems with each other. You get higher transparency through information in the right place. Your data is transformed and further synchronized in the ETL instance. Thus you improve your business processes.
Technical implementation
For the technical implementation of ETL pipelines, the DataEngine is required in addition to the HubEngine. It fulfills the purpose of an ETL layer. In this use case, data from system A is synchronized via the HubEngine to the DataEngine. There the data is transformed and then further synchronized into the target system B. The complete transformation logic takes place in the DataEngine. This is the simplest case of an ETL pipeline. The two systems are connected to the DataEngine via the HubEngine with one plan each. There, the data is stored and transformed in source modules.
Advanced ETL Pipeline
Your systems can be seamlessly connected via the distributed platform. Data is all routed via the distributed platform and transformed there. You can set up complex transformation workflows such as relation resolution, data harmonization, data standardization or calculations. This enables you to seamlessly integrate your systems and make the right data – in the right format and the necessary quality – available exactly where you need it. You can also control your processes via the distributed platform.
Technical implementation
The Marini Integration Platform with its components, HubEngine and DataEngine, enables ETL pipelines of any complexity. The pipeline can include any number of systems and their modules. Here, for example, four modules are input into the pipeline from one system and connected to two modules. The transform workflows in the DataEngine control all transformations as well as data aggregations and disaggregations. The number of systems and modules that can be integrated into an ETL pipeline is unlimited.
Summary
Simple to complex ETL pipelines can be modeled using the Marini Integration Platform. The DataEngine serves as the ETL layer. The HubEngine provides the connection of the source and target systems to and from the DataEngine. The ETL pipelines can also be designed non-linearly so that data can also be routed. This means that the ETL pipeline can also be split into two or more independent branches.
More resources on ETL pipelines can be found here: