22576 Chap.3 Data Quality Assessment and Cleaning
Data Quality Assessment and Cleaning
This chapter covers the cleaning half of Week 2 and its tutorials. It defines five dimensions of data quality, accuracy, completeness, consistency, validity and uniqueness, with typical examples and fixes for each, and sets out three treatments for missing values with the conditions under which each is appropriate.
It closes on duplicates, which must be resolved before keys can be trusted, and on documenting every decision, which the group report's first rubric criterion rewards.
The Week 3 tutorial applies the five dimensions to a file of share-market trading data, where symbols repeat, units are mixed, combined fields hold two values and some columns contradict each other.
Working through a file like that, one dimension at a time, is the best preparation for the cleaning section of the group report and for applied quiz questions that ask for a cleaned summary value.
What this chapter covers
- 01
Why poor-quality data misleads decisions
- 02
Accuracy, completeness, consistency, validity and uniqueness
- 03
Deleting rows, dropping variables or imputing
- 04
Mean, median or most frequent value as a replacement
- 05
Consistency fixes for dates, units and names
- 06
Duplicates and the primary key
Worked example · free
Choose a missing-value treatment for three columns
- 1Fax number: 3,100 of 5,000 is 62% missing, more than half, so drop the variable unless it matters to the question.
- 1Age: 120 of 5,000 is 2.4% missing and the column is symmetric, so impute the mean or delete those rows.
- 1Annual spend: 90 blanks in a right-skewed column, so impute the median, which extreme values do not distort.
Key terms
- Completeness
- The data quality dimension concerned with missing values in variables.
- Validity
- The dimension concerned with whether values make sense, such as an age that is not negative.
- Imputation
- Replacing a missing value with an estimate, such as the mean, the median or the most frequent category.
- Accuracy
- The quality dimension concerned with whether values are free of error and reflect what really happened.
- Uniqueness
- The quality dimension concerned with duplicated records.
Data Quality Assessment and Cleaning FAQ
When should I delete rows with missing values?
When only a few values are missing, roughly under 10 to 20% of a column, and the sample is large enough to lose them. Deleting is a poor choice when the sample is already small, because every lost row weakens later tests and models, and imputation may serve better.
Should missing values be filled with the mean or the median?
Use the mean for a roughly symmetric numerical column and the median for a skewed one, because a few extreme values drag the mean away from the typical case. For a categorical column, such as marital status, the tutorials use the most frequent category instead.
How should duplicate records be removed?
Confirm that every variable is duplicated, not just the identifier, before deleting a record; two rows can share an ID for legitimate reasons, such as repeat purchases. Remove true duplicates before merging, because a key that is not unique multiplies rows in the joined table.
What is the difference between a consistency and a validity problem?
A consistency problem is the same value written in different ways, such as two spellings of a state name or prices in dollars and thousands of dollars. A validity problem is a value that cannot be right at all, such as a negative age, a word in a numeric field or an end date before a start date.
How do I document cleaning decisions for the group report?
For each issue, state the variable, the quality dimension it breaches, the evidence you used, such as a summary statistic or a chart, the treatment you chose and why. The rubric's top band asks for treatments documented with visual and non-visual evidence and the logic behind them made clear.
Why split a combined field before cleaning it?
A field holding two facts, such as a 52-week range with both the high and the low price, cannot be checked or summarised properly until each fact has its own column. The workshop recommends splitting first, then fixing formats and types column by column, as the Week 3 trading-data tutorial does.
Assessment move
Take one messy tutorial file and write a cleaning log with three columns: the issue, the quality dimension it breaches and the treatment with its reason. That log is the backbone of the group report's first section, so practise writing it in full sentences.
Before each quiz, practise one complete sequence in KNIME: Missing Value node, Numeric Outliers node and a statistics check, reporting one cleaned value to two decimal places.
Working through Data Quality Assessment and Cleaning in 22576? Sia is AskSia’s AI Business Analytics tutor — ask any 22576 Data Quality Assessment and Cleaning question and get a clear, step-by-step explanation grounded in how 22576 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
