A many-designs study of 516 algorithms challenges the predictability of wisdom of the crowd forecast aggregation

Abstract

Accurate forecasts are central to decision-making in many domains. Although aggregating independent judgments can improve forecast accuracy, a phenomenon known as “Wisdom of the Crowd,” methods for effectively combining these forecasts remain underexplored. In this preregistered many-designs study, 129 research teams independently submitted a total of 516 algorithms to aggregate forecasts over six months, across four domains: economics, politics, climate, and sports. By testing hundreds of independently developed aggregation algorithms under identical conditions, we established an empirical benchmark for forecast aggregation. Drawing on forecasts by 1,182 people, we evaluated algorithm accuracy and variability. The share of algorithms significantly outperforming the mean and median varied by domain, with the strongest gains over simple benchmarks in politics and climate. Algorithm performance was also more persistent over time in politics and climate than in economics and sports. When asked to predict algorithm performance, the participating researchers were overconfident about the success of their own submissions and of others. We also examined potential predictors of algorithm accuracy and variability. Although researchers expected various features to predict accuracy, no algorithmic or researcher feature consistently explained why algorithms varied in performance. The main exception was the use by algorithms of previously observed realized outcomes, typically to compute past forecast error or train a model, which predicted greater accuracy in politics and climate. The null findings contrast with prior evidence that certain weighting criteria can improve forecast aggregation, and illustrate the value of preregistered many-designs studies which report the full distribution of submitted approaches, revealing when previously supported approaches fail to generalize.