How Do I Make My Data Smart?

When people talk about data quality these days, the term “smart data” quickly comes up. In this article, we show how data quality is related to smart data.

Smart data describes high-quality data on which informed business decisions can be made. The data is often extracted using intelligent algorithms. Smart data can be used to derive recommendations for actions and to make better decisions. In order to obtain smart data, two main challenges of data must be addressed today:

1. scattered data due to data silos
2. insufficient data quality

Smart data is complete, accurate, consolidated, merged, cleaned and relevant. Smart data filters the relevant information and ensures high data quality.

So how does my data become smart?

Data quality can be divided into 7 dimensions. We explain each dimension in detail below and then link them to smart data and the challenges of smart data.

Completeness describes whether the necessary data is complete and accessible. Completeness can be reduced by two states. On the one hand, additional data may not be accessible, e.g. in other systems. On the other hand, it may simply be unknown, such as the address of a contact.

Consistency describes whether your data is the same in different systems. The degree of consistency is measured by the difference between two data sets that refer to the same object (e.g. person or company) in the real world.

Accuracy describes how correct and precise data is. This can vary gradually. It does not always have to be a binary decision whether data is incorrect or correct. For example, a data record could actually be assigned to two departments. Let’s assume, however, that it is assigned to only one department in the corresponding system. On the one hand, this would make the data record incomplete. On the other hand, the data record would be neither completely wrong nor correct, but inaccurate. This would be an example of the degree of accuracy.

Timeliness measures the time span between when data is expected versus when it is actually available. For example, in a complex database query, results might be expected immediately but not be available for several minutes due to the complexity and size of the database.

Validity defines whether the data is in the expected or specified format. For example, there could be two fields for the street and house number. If the street and house number of a record were in one field, the content of the field would not be valid.

Currency describes the actuality of the data. For example, a customer’s address may no longer be up-to-date because the customer has moved.

Integrity measures the validity and correctness of relations and relationships between data. Missing or incorrect relations can undermine or reduce integrity.

First, you need to resolve your data silos. An iPaas solution such as as the HubEngine is suitable for this. Keep your data in sync or collect, transform and route the relevant data in one place like the DataEngine.

Resolving data silos isn’t just a challenge because of integration. Likewise, data often exists in multiple systems. If there is no common identifier – in Master Data Management this is called a Persistent Identifier (PID) – matching is often difficult. Machine learning methods for duplicate detection (“record linkage”) are used to solve this problem. This way you don’t create duplicates and your data is both more complete and more correct.

Integration not only resolves data silos, but also improves your data quality.

Through a clean integration – e.g. with an enterprise data platform – your data primarily becomes more complete (Completeness) and consistent (Consistency). The other dimensions are also positively influenced secondarily. For example, data can be available more quickly (timeliness) when you merge it. It is no longer necessary to pull together different data sources. Furthermore, data can be more valid (Validity), e.g. by defining formats in the DataEngine or validating data in the DataEngine via Dun & Bradstreet.

In addition to resolving data silos through a clean iPaas solution such as our enterprise data platform, it is also of enormous importance to increase data quality. For this, the various dimensions of data quality should be considered. Each dimension can be increased through a variety of technical and non-technical measures. Here we list for each dimension an example of how you can increase your data quality with the HubEngine or the DataEngine through automation.

• Completeness: Integration of your systems with the HubEngine.
• Consistency: Integration of your systems with the HubEngine
• Validity: Use workflows in the DataEngine to transform the data or actions in the HubEngine to force formats.
• Accuracy: Integrate your systems with the HubEngine or validate with Dun & Bradstreet
• Integrity: Conveniently map all relevant relations and relationships in the DataEngine by default.
• Timeliness: Link and transform the relevant data in the DataEngine so that you can quickly retrieve it there (or synchronized in your other systems)
• Currency: Update the data of your company contacts automatically with Dun & Bradstreet

The goal of Smart Data is to enable you to better understand the needs of your customers and thus serve their demand in a more targeted manner. You make better marketing and sales decisions and thus increase your economic success.

How exactly you use the data depends on your business processes. However, you now have the basis to design and control your processes in a well-founded manner and to derive meaningful recommendations for action from your data.

Supplementary note:

In order to obtain smart data, it is not always necessary to have big data or countless data sources. Even a few incomplete and/or low-quality data from a few sources can become Smart Data.

You can validate data with external services: Data Marketplace.