Foundations of Data Quality Management (Paperback)

Data quality is one of the most important problems in data management. A database system typically aims to support the creation, maintenance, and use of large amount of data, focusing on the quantity of data. However, real-life data are often dirty: inconsistent, duplicated, inaccurate, incomplete, or stale. Dirty data in a database routinely generate misleading or biased analytical results and decisions, and lead to loss of revenues, credibility and customers. With this comes the need for data quality management. In contrast to traditional data management tasks, data quality management enables the detection and correction of errors in the data, syntactic or semantic, in order to improve the quality of the data and hence, add value to business processes. While data quality has been a longstanding problem for decades, the prevalent use of the Web has increased the risks, on an unprecedented scale, of creating and propagating dirty data. This monograph gives an overview of fundamental issues underlying central aspects of data quality, namely, data consistency, data deduplication, data accuracy, data currency, and information completeness. We promote a uniform logical framework for dealing with these issues, based on data quality rules.

The text is organized into seven chapters, focusing on relational data. Chapter One introduces data quality issues. A conditional dependency theory is developed in Chapter Two, for capturing data inconsistencies. It is followed by practical techniques in Chapter 2b for discovering conditional dependencies, and for detecting inconsistencies and repairing data based on conditional dependencies. Matching dependencies are introduced in Chapter Three, as matching rules for data deduplication. A theory of relative information completeness is studied in Chapter Four, revising the classical Closed World Assumption and the Open World Assumption, to characterize incomplete information in the real world. A data currency model is presented in Chapter Five, to identify the current values of entities in a database and to answer queries with the current values, in the absence of reliable timestamps. Finally, interactions between these data quality issues are explored in Chapter Six. Important theoretical results and practical algorithms are covered, but formal proofs are omitted. The bibliographical notes contain pointers to papers in which the results were presented and proven, as well as references to materials for further reading.

This text is intended for a seminar course at the graduate level. It is also to serve as a useful resource for researchers and practitioners who are interested in the study of data quality. The fundamental research on data quality draws on several areas, including mathematical logic, computational complexity and database theory. It has raised as many questions as it has answered, and is a rich source of questions and vitality.

Table of Contents: Data Quality: An Overview / Conditional Dependencies / Cleaning Data with Conditional Dependencies / Data Deduplication / Information Completeness / Data Currency / Interactions between Data Quality Issues

資料品質是資料管理中最重要的問題之一。資料庫系統通常旨在支援大量資料的創建、維護和使用，著重於資料的數量。然而，現實生活中的資料往往是骯髒的：不一致、重複、不準確、不完整或過時。資料庫中的骯髒資料經常產生誤導或有偏見的分析結果和決策，並導致收入、信譽和客戶的損失。因此，需要進行資料品質管理。與傳統的資料管理任務相比，資料品質管理能夠檢測和修正資料中的錯誤，無論是語法還是語義上的，以提高資料的品質，從而為業務流程增加價值。雖然數十年來資料品質一直是一個長期存在的問題，但網絡的普及使用增加了在前所未有的規模上創建和傳播骯髒資料的風險。本專著概述了資料品質的核心問題，包括資料一致性、資料去重、資料準確性、資料時效性和信息完整性。我們提倡一個基於資料品質規則的統一邏輯框架來處理這些問題。本文分為七章，重點關注關聯資料。第一章介紹了資料品質問題。第二章發展了一個條件依賴理論，用於捕捉資料不一致性。接著，在第2b章中介紹了實用的技術，用於發現條件依賴，以及基於條件依賴檢測不一致性和修復資料。第三章介紹了匹配依賴，作為資料去重的匹配規則。第四章研究了相對信息完整性的理論，修訂了經典的封閉世界假設和開放世界假設，以描述現實世界中的不完整信息。第五章提出了一個資料時效模型，用於識別資料庫中實體的當前值，並在沒有可靠時間戳的情況下回答帶有當前值的查詢。最後，在第六章中探討了這些資料品質問題之間的相互作用。本文涵蓋了重要的理論結果和實用算法，但省略了正式的證明。參考文獻中提供了相關結果的論文指引，以及進一步閱讀的資料。本文旨在研究生水平的研討課程中使用，也可作為對資料品質研究感興趣的研究人員和從業人員的有用資源。資料品質的基礎研究涉及數學邏輯、計算複雜性和資料庫理論等多個領域。它提出了許多問題，同時也是問題和活力的豐富來源。目錄：資料品質概述 / 條件依賴 / 使用條件依賴清理資料 / 資料去重 / 信息完整性 / 資料時效性 / 資料品質問題之間的相互作用。