Prediction and Its Limits in Scientific Discovery

The desire to predict discoveries, to have some idea, in advance, of what will be discovered, by whom, when, and where, pervades nearly all aspects of modern science, from individual scientists to publishers, from funding agencies to hiring committees. The successes and failures of predicting scientific discoveries -- the creation of new knowledge -- have broad implications for understanding the nature of knowledge itself, and the likely future role that computers and artificial intelligence can play in its creation. In this talk, I'll begin with a simple conceptual framework for thinking broadly about prediction and its limits for scientific discovery. I'll then do a dive deep into two areas where accurate predictions would be highly useful for science policy: the predictability of researcher productivity over an entire career, and the predictability of when in a career scientists make their most important discoveries. To close, I will draw some broader conclusions from these results about the future of science, and I'll describe strategies and dangers for scientific discovery in the age of big data.