Title:Adaptive Data Analysis for Growing Data
Abstract:Reuse of data in adaptive workflows poses challenges regarding overfitting and the statistical validity of results. Previous work has demonstrated that interacting with data via differentially private algorithms can mitigate overfitting, achieving worst-case generalization guarantees with asymptotically optimal data requirements. However, such past work assumes data is static and cannot accommodate situations where data grows over time. In this paper we address this gap, presenting the first generalization bounds for adaptive analysis on dynamic data. We allow the analyst to adaptively schedule their queries conditioned on the current size of the data, in addition to previous quer…
Title:Adaptive Data Analysis for Growing Data
Abstract:Reuse of data in adaptive workflows poses challenges regarding overfitting and the statistical validity of results. Previous work has demonstrated that interacting with data via differentially private algorithms can mitigate overfitting, achieving worst-case generalization guarantees with asymptotically optimal data requirements. However, such past work assumes data is static and cannot accommodate situations where data grows over time. In this paper we address this gap, presenting the first generalization bounds for adaptive analysis on dynamic data. We allow the analyst to adaptively schedule their queries conditioned on the current size of the data, in addition to previous queries and responses. We also incorporate time-varying empirical accuracy bounds and mechanisms, allowing for tighter guarantees as data accumulates. In a batched query setting, the asymptotic data requirements of our bound grows with the square-root of the number of adaptive queries, matching prior works’ improvement over data splitting for the static setting. We instantiate our bound for statistical queries with the clipped Gaussian mechanism, where it empirically outperforms baselines composed from static bounds.
| Comments: | 28 pages, 6 figures, camera ready version for NeurIPS 2025. Addresses a bug in the computation of the JLNRSS and JLNRSS+ curves in Fig 2 of the previous version. Generalizes the results to non-uniform differential privacy |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2405.13375 [cs.LG] |
| (or arXiv:2405.13375v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2405.13375 arXiv-issued DOI via DataCite |
Submission history
From: Neil G. Marchant [view email] [v1] Wed, 22 May 2024 06:17:58 UTC (62 KB) [v2] Wed, 12 Nov 2025 12:47:53 UTC (126 KB)