MathArena publishes every graded attempt, so its AIME 2026 leaderboard can be tested directly. Resampling the 30 problems, 25 of the 31 models with full coverage have a 95 percent rank interval that includes first place, and an exact paired test separates only one of the 29 imperfect models from a perfect score. Sixteen problems were solved by every model on every attempt, and five carry 80 percent of the spread. Intervals computed over problems rather than attempts run a median 1.7 times wider, because they answer a broader question.
LLM synthetic respondents are known to produce response distributions that are too narrow. Nobody writes down what that costs you. Running a conjoint through a mixed logit and into a market simulator, a panel with correct mean price sensitivity and compressed heterogeneity reproduced the human holdout choice shares almost exactly, then set the revenue-maximizing price 26% too low and sized the premium segment at less than half its true value. Simulated in R with logitr, cross-checked in Python by Gauss-Hermite quadrature.