Unraveling the Regimes of Synthetic Data Metrics: Expectations, Ethics, and Politics

Abstract

Synthetic data - artificially produced data used for various data science tasks - have become the subject of intense scholarly interest, engendering both hope and hype in fields like machine learning (ML) and data privacy. In this commentary, we shed light on a little-studied facet of the emerging synthetic data landscape: their evaluation through the use of different quality measures, such as privacy, utility, and fidelity metrics. While these may seem highly technical, this commentary argues that evaluation metrics are inextricably linked to the expectations, ethics and politics of synthetic data. Situating synthetic data metrics within longer histories of data measurement in big data and ML discourses, we unfold a conceptualization of synthetic data metrics as metrological regimes which highlights the multifaceted ways in which they are implicitly and explicitly political. We put this concept to use by providing a three-fold preliminary analysis of metrics for the evaluation of synthetic tabular data: first, we outline the current constitution of synthetic data’s metrological regimes around utility, privacy, and fidelity metrics; second, we highlight the performativity of these metrological regimes; that is, how they overshadow other crucial measures and enact quantifications of essentially contested concepts; and third, we emphasize the fragility of synthetic data’s metrological regimes by pointing to the eruption of specific negotiations regarding which privacy metrics (not) to use for synthetic data evaluation. By foregrounding how metrics shape the expectations, ethics, and politics of synthetic data, this commentary underlines the need for their critical study.