Although some challenges are part of any data integration effort, the need for mastering the best practices for data integration remains essential for any organization
Many IT professionals ask if data integration is worth the effort and wonder about its usefulness in different situations. Another common question is what is more useful: centralized, or decentralized information? Why should any enterprise invest money in data integration? In the era of microservices, federation, virtualization, meshes and auto-provisioning everything tends to move to a more decentralized model. Those concepts look like a great approach to provide more agility and self-service to areas that are related to business and even to IT itself without always going through the pains that data integration imposes. But is this always true or are any hidden issues within these approaches?
What’s Data Integration?
Data integration is a process. It requires combining data from different sources into a single and unified view while going through ingestion, cleansing, and transformation tasks. This doesn’t mean that all data must reside in a single technology or a single platform. There must be a way of “joining” all the relevant data by using shared keys or any similar method.
Effective data integration needs a strategy as it is a thoughtful process, and it may be desirable to spend time on the data that will provide value instead of wasting money by integrating all existing data without knowing if it will bring any useful results. Besides that, not integrating data may result in incorrect conclusions and the lack of decent quality will ensure that much money is wasted; “Garbage-in still means garbage-out – and the cheapest data are the data that we re-use” says Martin Wilcox, a recognized data integration expert.
What happens when data isn’t integrated?
The lack of data integration affects enterprises can be shown by a real-life example. A person decided to move from one house to another one while continuing with their current ISP contract. The user logs in to his account and fills the form to declare the move. After realizing there was no confirmation and waiting a couple of days for them to call, he decides to fill the form again, in case there might been some issues with the form. Because again there was no confirmation, the user calls the ISP and talks to a service rep. The representative confirms there was no request, so she enters all the information to close the moving operation. At the user’s new home, the tech rep shows on time the day initially given and makes everything proceed quickly (the former owners were also customers for the same ISP so connection work was minimal). After a few days, a sales rep calls the user to talk about his plan to move and asks when to make the switch. Not without surprise, the user tells the rep that the operation was already done so the request should be closed. One day after that, another sales rep calls the user to plan the move. With some disappointment the user asks to close the requirement as the move was already done. Two weeks later another tech rep appears at the user’s house willing to cut the service, although payments were up to date. After some conversation, the tech rep explains that Mr. John Doe (the former owner of the house) finished his contract with the company so there shouldn’t be any service running on that address; the tech rep did not have any information about the “new” service. A very disappointing situation for the actual user who doesn’t deserve this kind of service – all based on a lack of data integration at the ISP’s end.
This is a true story and can happen in many companies in many industries. One can easily spot the problem about the lack of data integration and the way siloed repositories are working separately – and with no understanding of the business value of properly integrated data.
The power of data integration
What happened to this company is that they’ve totally lost their way and created a mess with their data by focusing on technology solutions only instead of setting a true data strategy. A modernization journey to the cloud combined with an agile mindset to support better customer interactions is very important but it could do nothing without a strong data foundation. It is easy to get lost with words like “cloud journey”, “agility” and such, so it’s more than important to pay attention to how good data enable better decisions and better customer interactions.
An organization is like a pyramid. No matter which management technique is applied, having many business areas, separate business units, independent branches, or any other organizational structure, everything is part of the same business. To have a healthy business all the parts should work together, seamlessly, and integrated. As it’s impossible to imagine driving a car just by buying all the pieces and parts separately and trying to make them work together, many enterprises think that a business can be run having data spread all over, in many separate siloes, without any relation between domains and entities. Applying “brute force” to bring everything together may be possible but it’s for sure a waste of money and time, which may lead to unpleasant results like financial decreases, longer sales cycles, and damaging customer satisfaction (as the example shows).
After all, organizations have the need for integrated data as everything integrates to a one and only reporting and process structure. A good suggestion should be to integrate the data sooner than later.