'How to Improve Data Quality in 2022' by thuwarakesh data datascience
Data quality assessment is the continuous scientific process of evaluating if your data meets the standards. These standards may be tied to your business or the project goals. The need for ensuring data quality has increased as the many different ways to acquire data are multi-folded.
Handling a single data source alone can be challenging at times. Say, for example, a customer survey. It's often difficult to normalize each respondent's information, even with online survey tools. Now imagine integrating and standardizing data from ERPs, CRMs, HR systems, and not to mention the many different sensors we use these days. Without data quality assessments, these are a problem for a lifetime. But there is good news! We've evolved along the complexities surrounding data acquisition and management. Data quality assessments play a crucial role in data governance. They help us identify incorrect data issues at various levels in a data pipeline. They also help us quantify the business impact and take corrective measures as soon as possible.Take, for instance, a data quality issue in the healthcare industry. Suppose the data entry person duplicated a patient's record; the patient would receive two doses of the drug instead of one. The consequences can be disastrous.such as the above can have terrible effects regardless of the industry. But duplication is only one kind of data quality issue. There is a spectrum of other quality problems we need to worry about. Let's imagine you're working on an inventory optimization problem. The stock is monitored through an automated system. What happens if one of your sensors sends values twice the original frequency? Unreliable data will lead you to stock up on items already in the warehouse yet missing out on all the high-demand stuff.Data quality has six dimensions: Accuracy, Completeness, Consistency, Timeliness, Validity, and Uniqueness. We can also think of data accountability and orderliness as other critical characteristics. The different dimensions discussed here are the scales against which we evaluate our data quality. Maintaining 100% quality in a vast data lake is nearly impossible. Data quality tolerance is a strategic decision we must make as early as possible. But that's for a future post.Data accuracy is a prevalent quality aspect everyone is battling to get right. But what does data accuracy means anyway? Data accuracy is to what extent the data at hand captures the reality. The apparent cause is the data entry --- typos in the name and wrong values for age.. Lockheed Martin, a team of English engineering team, was working with NASA to run the program. Different measurement units used by the two groups caused the communication blackout with the spacecraft.Data completeness refers to your datasets having all the required information on every record. The requirements depend on the application and business needs. For instance, phone numbers have only a little use for a machine learning model, whereas it's critical for a delivery system. Form validations and database constraints help a lot in reducing completeness errors. Yet, planning mistakes often make huge impacts on the quality of data. Data completeness is a tradeoff. The more strict you are on the fields, the less you get on records. This tradeoff is valid for both manual and automatic data acquisition. If you make all the fields mandatory in a survey, you don't get as many responses as you intend. On the automated side, let's say you put a constraint on GPS coordinates for a data stream coming from a remote camera. You install a set of new devices that may not support GPS and send data that won't get accepted to your data lake. It is a challenging dimension to score well. The complexity grows high as you acquire data from more sources.Data consistency is having no contradiction in the data received from different sources. Because each data source may have a unique way of measuring the information, they sometimes don't match others. Say you want to find out the daily sales volume of a particular product. Your inventory management tracks sales based on the remaining items. Your POS tracks the same based on the items sold. Items returned may sneak into the inventory system without having a record in the POS.In the ideal world, both systems should account for returns. But it's rarely the case, given the complexities of large-scale organizations.Data should be available at the time it's required in the system. Suppose you generate a report every Friday, and not all your data have arrived yet; it'll seriously alter your organization's decisions and directions.There are network issues. Read about edge computing if you think cities have decent internet connections and nothing to worry about. The whole concept is built to reduce network latency. There can be operational issues. The product returns and daily sales calculations are good examples of a lack of timeliness. We have problems arising at the point of data collection. They could be wrong data entry, malfunctioning sensors, etc.I finished my high-schools about 12 years ago. But I still receive brochures from institutions that are targeting school children. It's an excellent example of having invalid data. Invalid data are records that don't have a meaning anymore. They fill up space with no use. Also, when they are used can be dangerous too. Invalid data costs a lot, yet the invalidate rules are blurry in some cases. For example, how do we know if a patient has fully recovered from disease unless you're the doctor or the patient yourself? Some decease may have an average time range for recovery. But not all. In such cases, you keep invalid data in your data store and make painful decisions based on them.Uniqueness in data means no replication of the same information twice or more. They appear in two forms; duplicate records and information duplication in multiple places. Duplicate records are often easy to pick. They appear more than once in the same datasets and are relatively straightforward to remove automatically. A good practice is to use a key column to impose a uniqueness constraint rather than the whole record. That is because specific repetitive entries may contain some fields that aren't unique anymore. Most transactional entries have a timestamp which is a perfect example. They don't appear as duplicates if we don't use one or a combination of a few fields for de-dupe. Information duplication is storing the same information in different places. For example, a patient's age may be on the admissions table and the surgery table. It's not just a good design. Duplicated information is the gateway to other quality issues. Failing to update all the records will create inconsistencies. At least one of them is, anyway, inaccurate. Another not so apparent duplication is derived information. Take age and date of birth. One is enough to find out the other. But storing both creates ambiguity.We need to perform data quality assessments for every critical area in our data store. The most granular you can go is to the field level. But you can also check up to a database level. Data quality assessment is an iterative process to verify if your data meets the required standards. Each iteration will have the following six phases.In the 'define' phase, we translate the business goals into data quality targets and decide on what is acceptable quality. This matrix should be measured against each of the six data quality dimensions. Reaching 100% is unlikely in large-scale applications. But if you're working with smaller datasets, you can be more strict about them. If you are a healthcare app that sends subsequent dosage alerts, you need to maintain a log of every dose the patient took. The timestamp field in every record is a crucial piece of information for the next dosage. Hence it should have a threshold of near 100% against all six dimensions. But in case you own a cake shop and want to send a birthday card every year, your rules can be far more flexible. The address or the phone number field should have a high threshold for accuracy. Yet, they can have an average target for uniqueness because people sometimes give their alternative phone numbers when they buy. These thresholds also depend on the domain. As seen in the last two examples, the cost of a mistake is minuscule in the second case compared to healthcare. These rules can be on multiple granularities. For instance, the address column can have a unique threshold. But we can also impose a completeness threshold as each record should have the phone number or the mailing address.In the 'assessment' phase, we evaluate our datasets against the rules we defined on the six data quality dimensions. Each will end up with an acceptable score. The acceptance score is the percentage of records that satisfy the conditions. On a small dataset or database, it's pretty easy to conduct these experiments manually. However, in a vast data warehouse, you need some automation to verify the data quality.Data quality assessment does not end with the assessment phase. A data quality assessment aims to identify the business impact as early as possible and implement corrective measures. Estimating the business impact is the goal of this phase. It's a complex exercise, and it's not domain agnostic. The way one organization summarizes the assessment scores differ from others. But the goal of the phase is obvious. We are finding the most significant holes where the quality of data leaks and will fix them.In the 'brainstorm' phase, we collaboratively developed ideas that could fix the gaps we found. It is best to have a unit that includes members from every team so that the plans areDesirability, feasibility, and viability can make or break an idea.Let's suppose we identified from the analysis phase that patients don't fill up some critical information at the front desk survey. Your technical team may suggest making the survey electronic. Electronic surveys can impose validation checks that are hard to bypass. A doctor or a nurse in the group may say this solution is undesirable as people rush to get into the treatment. The front desk staff may say they are in a hurry even when checking out. Hence, your final solution may be a primary form fill-up at admission and a detailed electronic survey after their treatment. You may have noticed the solutions are not always technical. Data quality can not be improved only by fixing the data pipeline. It may need, sometimes, unconventional strategies that aren't apparent.Automate data collection.
Please follow us on Google to support us United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
US School and Mass Shootings Reaching All-Time Highs in 2022: Data and MapsUS gun violence — including school shootings, mass shootings and accidental gun deaths — reached all-time highs after the pandemic hit and show no sign of slowing down in 2022. We explored the factors causing the surge and break down America’s gun problem in graphics.
Read more »
How Big Data is Keeping Employees Engaged in the Age of WFH | HackerNoonBig data is beginning to emerge as a key tool for businesses to successfully operate on a WFH basis.
Read more »
Who is Life360 Selling Your Data to? | HackerNoonLife360, a safety app, sold data to Cuebiq and X-Mode, two location-data brokers.
Read more »
Despite stock market volatility, many Americans are still becoming 401(k) millionairesAccording to Fidelity Investments’ 2022 first quarter data of its participants, the savings rate for 401(k) plans reached a record level of 14%.
Read more »
Moby-Dick; or The Whale: Chapter 15 - Chowder | HackerNoonMoby-Dick; or The Whale, Chapter 15: Chowder by Herman Melville is part of HackerNoon’s Book Blog Post series.
Read more »
Top 5 Interview Questions For Junior React Developers in 2022. | HackerNoonJunior React developers trying to get a job are likely to come across these five interview questions when trying to get a job in the industry as a grad.
Read more »