Approaches to Improving Data Quality

Informal data quality improvements are often at odds with what the organization actually needs.  Focus on organizational data quality and not on individual efforts.

Often an organization may informally delegate its responsibility for high-quality data to some people who rely on their individual judgment to develop a separate version of quality data.  This version may vary according to their specific needs and context – and may not be what the organization would view as high-quality data. 

This approach to data quality using ad-hoc initiatives, unstructured and outside of any framework or strategy can be termed Laptop Data Quality.  It occurs every time a data user feels in some way responsible for the quality of the data they need.  Often, these users spend a considerable percentage of their working time focused on improving the quality of the data they are using, so they are diverted from the tasks they are hired to do and should be focusing on.

Laptop Data Quality is usually executed by ad-hoc processes or in Excel files / Access databases, where data is corrected according to specific needs, specific criteria, a specific context, and the data user’s discretion –rarely are any of these commons between two different users.   The practices of an enterprise approach to data management are usually not found in a Laptop Data Quality culture.

The lack of a structured approach to data quality can have a negative influence on every organization’s financial performance, impairing the decision processes, and preventing additional gains.  Unfortunately, this means the organization has valuable resources redirecting their time and skills to get data into minimal quality levels for their specific needs, often unaligned with the global business and data strategy.

Although some of the effects are easier to quantify than others, there are some questions to be answered about this approach:

• How many hours are being spent across the organization in ad-hoc tasks related with data quality, and what is the cost of those hours?
• What is the effective cost, or value not being generated, due to the hours that are diverted to these tasks?

These questions do not have an easy answer because both the costs and benefits remain hidden in the original work of the analyst, developer, data scientist or any other data user.

This approach not only affects performance, but it can also change the results since it depends mostly on the user’s context, needs, and knowledge; the same data will be corrected differently by different people.

Challenges to a Structured Approach to Data Quality

A more structured approach to data quality might not be easier to implement.  Typically, these are expensive initiatives; they are time and resource consuming and span multiple time periods.

They can also be deeply intrusive and disruptive, creating a natural resistance to change within the organization, creating a difficult environment in which to work.

Another important characteristic is that a data quality program can take years to break even and deliver ROI, making it hard, even with a strong sponsorship, to keep the necessary traction to complete all the required changes.

These are the most frequent causes that are identified on “project post-mortem” reports, noting reasons that these projects are rarely a priority.  Many organizations decide not to accept such projects where the results are hard to evaluate, and the effort will only show return on the long term.  As a result, organizational data quality suffers, and the culture continues to live with poor-quality data.

Effects of Bad Data

The effects of having bad data in the business processes are easy to identify and affect every business and all processes.

These results may be in the form of:

1. Lost revenue, sales, or business opportunities, including:

• Lost sales opportunities.
• Failure to execute product cross-selling.
• Inability to properly identify customer needs.
• Failed marketing campaigns due to lack of accurate perspectives.
• Invoicing problems, either resulting in an inability to properly bill the customers or by incurring additional costs in the billing process.
• Missed B2B opportunities or inefficient procurement due to inaccurately analysing the market.

2. Customer dissatisfaction and service costs, including:

• Loss of customers.  Besides losing the direct revenue related with the customer lifetime value there are costs associated with new customer acquisition, as well as additional indirect costs since the now-dissatisfied customer can work as a market influencer who can affect prospects.

3. Operational inefficiencies, including:

• Poor resource planning since there may not be sufficient staff due to focus on data cleansing and not on assigned responsibilities.
• Increased operational costs, either on system workloads or work hours spent on data quality related issues.

4. Regulatory compliance, including:

• Inability to comply with regulatory measures appropriately. In some industries where regulatory compliance is essential, poor data quality has a significant impact on the capability to comply with the regulatory obligations, resulting in heavy monetary penalties or even civil or criminal proceedings.

5. Poor decision making, including:

• Inability to make correct long-term decisions due to a lack of a trusted data foundation.
• Incorrect forecasts due to inaccurate or missing data.
• Inaccurate customer profiling and segmentation, leading to decreased sales and lower customer retention.

Strategy to Improved Data Quality

To avoid the traps of entering Laptop Data Quality mode, or to support a major data quality program, the best strategy addresses specific problems with measurable results.  Design data quality initiatives that:

• Have a reasonable funding model; do not need to purchase every data quality application, but some funding for resources and tools is essential.
• Are targeted to address enterprise challenges with critical data
• Have focused effort to correct data quality challenges
• Can be completed in reasonable time periods – months, not years
• Increase internal engagement on continued data quality efforts
• Deliver targeted returns when promised – shorter time periods

In large organizations, even those without a strong data culture, the opportunities to start these initiatives are quite abundant. Across all the business areas there are pain points related to the quality of data and identifying them is not a challenge.

The first step is to identify a business stakeholder who can passionately and effectively articulate the effects of poor data quality in their processes; their help can pinpoint the source of these problems.

Remember, most of the time it is not about identifying the actions that can reach the best ROI but identifying who has a problem that needs to be targeted, assessed, and mitigated quickly and with available resources.

It is easier to help someone who asks for help than to persuade someone that they need help.

A sequence of these targeted initiatives has the benign effect of increasing the awareness of the importance and impact of data quality across the organization, increasing the overall internal engagement, turning critics into evangelists and paving the way to a more structured and strategic data quality approach across the enterprise.

Improving Data Quality

Launching a successful data quality program may be a daunting task, although not an impossible one depending on the approach.  Follow these steps based on the characteristics mentioned above:

1. Start with business areas than can clearly identify and measure the business impact of bad data on their processes. In every organization the opportunities to identify these cases are abundant. Across all the business areas there are pain points related to the quality of data and identifying them is not a challenge.
2. Build a business case with those willing to defend it.  Once you have identified a critical pain point, you will have the business stakeholder who can passionately and effectively articulate the effects of poor data quality in the processes, and who will be eager to defend the project.
3. Focus on turning insights into action.  Having the business stakeholder actively involved will accelerate the process of quickly moving from the findings to specific actions.
4. Establish data quality targets based on data that is critical for business.  A deep understanding of the impacts of bad quality data on the business processes enables a more accurate prioritization of the critical data, making easier to identify clear targets at the start.

Conclusion

Making the option for data quality initiatives that are more focused and efficient creates and increases the awareness across the enterprise and acts as the motor from within the organization for a full data quality program. Avoid Laptop Data Quality efforts through a strategic yet tactically sound program for improving organizational data quality.