Where Does Dirty Data Originate?

Size: px

Start display at page:

Download "Where Does Dirty Data Originate?"

Roberta Ray
5 years ago
Views:

1 Social123.com

The importance of data quality cannot be overstated. For marketers, it starts with educating ourselves and our team as to what dirty data looks like and where it originates.

While buying email lists is tempting, the long-term consequences aren t worth the short-term fix. One bad apple is all it takes to contaminate your list.

2 The importance of data quality cannot be overstated. For marketers, it starts with educating ourselves and our team as to what dirty data looks like and where it originates. Here s a close look at where to direct your lens. Bad List Buys Looking to boost your marketing list in a snap? Buying those contacts from a list-purchasing company might appear too good to be true. That s because it is. SiriusDecisions reports that, on average, purchased contact lists are 15 months old at the point of sale, at best - and they continue to deteriorate over time at the rate of about 25% per year. While buying lists is tempting, the long-term consequences aren t worth the short-term fix. One bad apple is all it takes to contaminate your list. As Hubspot blogger Corey Eridon writes, One customer s ill-gotten address list can poison the deliverability of the other customers on that shared IP address. On average, purchased contact lists are 15 months old at the point of sale, at best - and they continue to deteriorate over time at the rate of about 25% per year. Something else to consider when looking to buy from a list broker: the source. Data that comes straight from the original source itself, the actual contact, is ideal, but very few data providers offer self-reported contact data and contact attributes. These list companies almost always compile data - they take one source, butt it up against another source, then another, then another and at some point decide which data points are most accurate and then merge it all together. If it isn t compiled data, its probably crowd sourced, meaning that the source of the data is other individuals who claim that a record is accurate or inaccurate based off of information they have that may or may not be closer to the original source (the contact). Typically, there are rewards and incentives put in place to encourage customers to correct records when they are wrong. The accuracy of data is NOT a democracy - just because someone says that a record is accurate does not mean it is.

Natural Decay: Expired Records In a workforce that is more transient than ever, a database will have a natural decay of contacts because people change jobs, companies go out of business, and mergers

3 Natural Decay: Expired Records In a workforce that is more transient than ever, a database will have a natural decay of contacts because people change jobs, companies go out of business, and mergers happen. To compound the issue, its happening now more than ever with the average position held less than 2 years. In 2015, it is totally acceptable to change jobs after 3 years, but think about the numbers - if everyone changes jobs within 2-5 years in a database of 100k+, then it is expiring fast...almost hourly. Keeping tabs on data before it goes bad is paramount to success. The problem of data decay is that it s faster than it s ever been, says Sam Zales, president of ZoomInfo, adding that 71 percent of business cards you collect have at least one change within 12 months. The bottom line is that there is no avoiding it- it s inevitable that databases everywhere are expiring every day. The best solution though, is to first accept it and then understand why so that cleansing the data can be more comprehensive and more frequent. Individuals in the U.S. hold an average of 11.7 jobs from ages Bureau of Labor Statistics, U.S. Department of Labor (2015)

4 Single Source & Multi Source Problems Dirty data can occur within a single set of records or between multiple sets of data which rely on each other and need to be merged. These two major data quality problems are referred to as single source problems and multi source problems and can be restored by cleansing the data. Data Quality Problems Single-Source Problems Multi-Source Problems Schema Level Instance Level Schema Level Instance Level Lack of integrity constraints, poor schema design Data entry errors Heterogeneous data models and schema designs Overlapping, contradicting and inconsistent data Uniqueness Referential integrity Misspellings Redundancy/ duplicates Contradictory values Naming conflicts Structural conflicts Inconsistent aggregating Inconsistent timing Rahm, Erhard: Data Cleaning: Problems and Current Approaches. University of Leipzig, Germany.

5 Single Source Problems Where Does Dirty Data Originate? Schema Level Problems: Problems that occur within the theory or standardization of database organization; schema level problems will be reflected in instance level problems. Typically these problems occur when a database is not properly engineered or has poor integrity constraints. Examples: Illegal values - values are outside of the domain range Uniqueness violations - duplication of unique fields, like a Social Security number Referential integrity issue - referenced field is not defined Violated attribute dependencies - one field depending on another field incorrectly, like city and zipcode *The 2 different contacts, have the same exact phone number including the extension, which violates uniqueness rules and creates structural problems.

6 Instance Level Problems: Errors, inconsistencies and inaccuracies in the actual contents of each record; these problems are not visible at the schema level. Errors that occur at the instance level encompass a wider rage of inconsistencies that do not reflect the database structure but data conflicts themselves. Examples: Misspellings - typos or phonetic errors Missing values - unavailable values during data entry Embedded values - multiple values entered into one attribute field, like including City and State in the City field Misfielded values - value is in the incorrect field, ex. city = Virginia Violated attribute dependencies - one field depending on another field incorrectly, like city and zipcode Duplicated records - same contact shown twice due to data entry or merging errors Contradicting records - duplicate contact with different values *These are duplicate contacts, with contradicting information, missing values, and embedded data

Multi Source Problems Where Does Dirty Data Originate? When multiple data sources need to be merged in a data warehouse, the need for data cleansing increases significantly.

7 Multi Source Problems Where Does Dirty Data Originate? When multiple data sources need to be merged in a data warehouse, the need for data cleansing increases significantly. This is because the sources often contain the same data in different representations or they overlap or contradict each other. Schema Level & Instance Level When there are both Instance and Schema issues with a database, it means that not only are there inaccuracies within some of the record instances, but there are also structural issues with the database because of schema problems. Schema problems almost always create instance problems, so if both are occurring it means that the structure of the database is flawed, creating inaccuracies in certain instances. But it also means that there are instance inaccuracies caused not by schema issues, rather they are caused by data entry errors, naturally expired data, bad list buys, etc.

8 Data Cleaning Resolution Where Does Dirty Data Originate? Ignoring data quality issues can risk the success of routine operations and lead your company down the wrong path. Identifying from where dirty data originates is a step in the right direction. Resolve schema conflicts to ensure successful data cleaning, then implement these five best data cleansing practices for continued good health. 5 Best Practices for Data Cleaning Before implementing data cleaning, it s important look at the big picture. What are your goals and expectations? How do you plan to execute it successfully? As you implement data cleaning, keep the following tips in mind. 1. Develop a Data Quality Plan Knowing where most data quality errors occur and identifying incorrect data will help your team better assess the root problem and develop a project plan. A comprehensive data quality plan will impact many departments so keep communication open and emphasize that better intelligence will save everyone time and money. 2. Standardize Contact Data at the Point of Entry Create a stringent process for entering data and check important data at the point of entry. This ensures that all information is standardized when it enters your database and will make it easier to catch duplicates. 3. Validate the Accuracy of Your Data Validate the accuracy of your data in real-time or by cleaning your existing database. Research and invest in data management tools that can identify old/inaccurate records and replace them with accurate records. It is also important to find a data provider with the ability to validate addresses and phone numbers. 4. Identify Duplicates Save your team time and incorporate data management tools that effectively identify and remove duplicates. The less manual work, the better. 5. Append Data After your data has been standardized, validated, and scrubbed for duplicates, find a good data provider to append more valuable information to each record. Reliable third-party sources like Social123 can capture information directly from the source, then clean and compile the data to provide more complete information for business intelligence and analytics. This will help you strengthen segmentation and send more targeted information to customers and prospects.

About We are a social sourced B2B data provider with the

appends, we offer the most reliable data solution on the

Search for new leads using our platform or for existing

with 35+ fields of self-reported attributes like groups,

9 About We are a social sourced B2B data provider with the largest and most accurate global contact database that is always refreshing itself in real-time. From origination of new targeted leads to full service appends, we offer the most reliable data solution on the market. Search for new leads using our platform or for existing databases, we will identify inaccurate records and append them with 35+ fields of self-reported attributes like groups, skills, title, and even a validated address, tested with our proprietary validation software Powers Ferry Road, Ste. 650 Marietta, GA Social123.com

Validation & Verification Unlocking the Mystery

Email Validation & Verification Unlocking the Mystery Synthio.com 1.888.530.6723 Sales@Synthio.com @Synthio_ Email Validation: Unlocking the Mystery When It Misses the Mark While many businesses offer