Human-Level on Which Questions? What a Model-Built Benchmark Measures, and What It Doesn’t?

Venkatesh Saligrama
Professor, ENG (ECE, SE, CS)
Boston University
Frontier vision-language models are increasingly evaluated by whether they approach or exceed human performance. But how much does that conclusion depend on the questions we choose to ask?
We investigate this question using two visual-reasoning evaluations, one with AI-generated questions and the other with human-generated questions. Several frontier models outperform humans on the AI-generated questions but fall below them on the human-generated questions. Although humans also find the latter somewhat harder, the models deteriorate much more sharply.
We investigate this reversal through controlled comparisons of question wording, visual evidence, component reasoning, and the effects of supplying relevant facts. Our findings suggest that question construction determines not just difficulty, but which evidence and relationships a model must identify and use. AI-generated questions more often concern information captured in image descriptions, while human-generated questions more often require selecting scene-specific evidence to make a judgment.
These findings motivate a broader view of AI evaluation: a benchmark score reflects the interaction between the demands of the test and the capabilities of the model. I will discuss how this perspective can help us understand evaluation failures and develop more informative tests of frontier AI systems.
Joint work with Kevin Miller, Arjun Chandra, Wei-Lun Chao
Venkatesh Saligrama is a Professor of Electrical and Computer Engineering, Data Science Faculty Fellow, and founding member of the Faculty of Computing and Data Science (CDS). He is also Affiliated Professor of Computer Science and Systems Engineering at Boston University. Professor Saligrama’s research interests are in machine learning and its applications, including resource efficient machine learning (budgeted ML), learning with limited supervision, AI and society, and learning and estimation on graphs/networks. Professor Saligrama is an IEEE fellow elected for contributions to detection and estimation theory. He is a Distinguished Lecturer for IEEE Signal Processing Society. He is the recipient of numerous awards including the Presidential Early Career Award, ONR Young Investigator Award, and the NSF Career Award. He has served as area chair for several machine learning conferences such as ICML, COLT and NueRIPS, and as an Associate Editor for the IEEE Transactions on Information Theory, the IEEE Transactions on Signal Processing.