The DBAs pursuit of Data Quality

Introduction

Organizations gather, store, and create data at an exponential rate. To add credence to that fact, most businesses can easily say their current year’s data is larger than the previous year’s. What’s more, they predict that the amount of data they’ll manage next year will be more than what they manage today.

That’s not a shocking revelation because analytics-based projects and big data is a result of the Information Age in which we live. To handle the large influx of data, machine learning and artificial intelligence (AI) are leading the way to new technological normality featuring processing in a cognitive way.

How to recognize quality data

A well-known data quality expert said that data is high quality if it’s suitable for what they are meant to be used in; operations, analytics, planning, or decision making. In other words, the data has quality if it is easily understood, has the required details in the right amount needed, is free of irregularities, is applicable, and complete.

DBAs pursuit data quality remains challenging

Data quality errors proclaimed by businesses surveyed

Data quality may not be as accurate as we think it is. Error rates for credit records have been reported to be as huge as 30 percent; billing records between 2-7 percent; and changes to payroll records at 1 percent.

The bullwhip effect intensifies issues affecting data quality

Incorrect reporting from a retailer in a business’s supply chain can lead to the bullwhip effect. For example, say one of your suppliers, a retailer in your distribution channel, gives a consumer demand figure but the report is inaccurate by 10 percent. If the true demand was 20 units, and the retailer wrongly reported 22 units (because 20 plus 10 percent equals 22), the 22 units are expected to cover the demand.

But wait. In reality, 22 units were reported wrongly. Here’s the bullwhip effect. The orders show an incorrect supply and demand and consequently, there are overages and shortages in inventory. The storyline could worsen. If the distribution center were serving 25 retail shops, it means there are 25 extra units ordered and each shop gets one extra unit, unnecessarily.

Money is tied up in inventory that the business doesn’t need. On a global scale, if there were 25 distribution centers and each ordered 25 extra units … you get the picture. The bullwhip effect is like imagining a person cracking a bullwhip and it ripples up and down just like highs and lows in inventory.

Data quality influences other areas of a business

Data quality affects other things in business too. Incomplete and inaccurate product data brings down customer satisfaction levels. Customers shop for products priced competitively. If products aren’t priced correctly, it’s a losing situation for all involved. This trickles to your website or brick-and-mortar store if the customer is dissatisfied. They won’t come back.

Low-quality data raises costs as well. Management decisions based on bad data costs money in fixing mistakes.

Bad data quality minimizes profits when denied transactions occur based only on incorrect information. Therefore, the cost of sales that should have been is a side effect of bad quality data.

The Health Insurance Portability and Accountability Act (HIPAA), the Payment Card Industry Data Security Standard (PCI-DSS), the General Data Protection Regulation (GDPR), and other governmental regulatory compliance agencies put in place data quality standards, which if not followed, could result in fines and other disciplinary actions.

The first steps to take to improve data quality

Change starts with the top level decision makers. Because without their approval, DBAs are limited in what they can do for their organization. First, it’s important to admit that there is indeed a data quality problem. Large or small improvements made now may curtail an impending catastrophe. Realize that information is best perceived as valuable. It’s an asset to the business. Just as other corporate assets including office buildings, office equipment and intellectual property, data is a precious commodity.

The next step is to accurately define and inventory the elements of your data that are crucial. You must be able to locate every bit of meaningful data on the staff’s desktops right down to the important statistics duplicated in their Excel spreadsheets.

Combat bad quality data with constraints and triggers

DBAs can do much to bring data quality up to standards. As a DBA, you can analyze your tables and build constraints into all databases. Doing this will automatically improve data quality.

Two types define constraints: Referential – Ensures that foreign keys conform to their parent primary key. Unique – Safeguards keys to prevent the entering of duplicate keys.

At least when these constraints are implemented, there’s a major improvement in the quality of data.

Conclusion

The DBAs pursuit data quality can result in more confident decision making, more accurate customer transactions including fewer denials due to bad data. In addition, more consistently higher quality data is possible and likely.

As a DBA, begin the change by checking how your organization manages its data. Look into the procedures, policies, and processes as well as which people are responsible for specific actions. Know how your company monitors government compliance.

If this all seems overwhelming at first, look into a tool for data profiling to determine the current quality of your data. Taking action today might be the start that leads to major changes later on down the line.