Institute of Information Science, Academia Sinica

If a dataset is only used once, a rich theory exists for ensuring that the conclusions are valid. But what happens if the same dataset is reused for multiple analyses? Since each analysis may now depend on the outcome of previous analyses, the danger of overfitting the dataset is increased. For example, if the same dataset is used to select a model and then fit that model, the resulting model may appear to explain the data better than it should.
In this talk, I will discuss a recent line of research on adaptive data analysis. I will show that there are sophisticated techniques that enable us to ensure that adaptive analysis provides sound conclusions. I will also show that such techniques are in fact necessary and that adaptive data analysis is inherently more powerful than non-adaptive data analysis.