Reading Time: < 1 mins

Workflows or How I Power the DataEngine

Authored by
Learn more about the Marini Integration Platform in an online demo and discover the benefits.
2021-03-18-DataEngine-Usage-Monitoring

While monitoring the DataEngine instances, we noticed unusually high disk and CPU activity. Our investigation then revealed that improperly configured workflows were the cause.

Here is the initial situation we found:

2021-03-18-DataEngine-Usage-Monitoring

The upper diagram shows a CPU utilization of around 40%, combined with high read activities to the hard disks (middle and lower diagram – green line).

This workflow was responsible for this:

2021-03-18-DataEngine-Usage-View

What we see here is a workflow that runs not only in the scheduler (i.e. every minute), but also when saving. (1) Furthermore, we have not set any conditions, so that this workflow is applied to all records in the CDP Contact module every minute. (2) What is not visible here is that the action updates the record itself, which in turn triggers an “on-save” event and thus this workflow again. (3)

So a recursive loop was built, which only exits when there is a saving error. Changing the same record, without a corresponding condition and calling the same workflow at the same time is deadly. Here, the instance was barely usable during execution.

Conclusion:

If you want to repeatedly update the same records via a workflow, you have to set appropriate conditions that prevent you from setting up recursive loops here. If several actions are executed one after the other (with a change of the calling record), the condition for the execution of the workflow should be set right at the beginning. Then a new call is prevented.

Learn more about the Marini Integration Platform in an online demo and discover the benefits.
and
Topics:
DataEngineKristian WereliusProcess Automation

Related Articles

Customer Intelligence: Strategien und Use Cases zur Nutzung von Kundendaten

A Customer Intelligence book that combines strategic depth with true real-world impact

Reading Time: 3 mins

Discover how 33 experts from leading companies turn customer data into measurable value using frameworks like CIEF, the Golden Record, and Customer Data Platforms.

Realistisches Puzzle-Portrait einer zufriedenen Kundin in mattschwarzen Puzzleteilen – Symbol für die MARINI Golden Record CDP, die fragmentierte Kundendaten zu einem einheitlichen „Single Source of Truth“ vereint.

From Data Silos to Gold: How Golden Records Lay the Foundation for Your Customer Intelligence

Reading Time: 3 mins

Golden Records in your Customer Data Platform (CDP) consolidate fragmented data into a reliable customer profile (Customer 360). Increase efficiency and maximize AI potential for your AI agents.

Unlocking HubSpot-SAP Integration: A Scalable, No-Code Solution

Reading Time: 3 mins

Learn how to perform a successful SAP HubSpot integration to seamlessly connect data models – fast and flexible.