Diogo Machado, who leads the data-science team at Technopolis in Brighton, UK, tested whether GPT-4o from OpenAI could assess grant proposals. He gave the AI model 139 genuine research proposals submitted in 2021 to the Austrian Science Fund and asked it to select which should receive funding.
The AI agreed with human reviewers 60% of the time. But it showed a systematic bias: it was less willing than human reviewers to fund proposals rated highly for novelty, according to the analysis published in Scientometrics in August.
Each proposal included human assessments of four criteria: novelty, scientific quality, team capacity, and feasibility. The AI model's reluctance to back novel work could skew funding toward established research and away from potential breakthroughs, Machado found.
Machado suspects the bias stems from how large language models learn. These models recognize patterns that appear frequently in their training data. Established ideas appear often; novel contributions appear rarely. This means the models have fewer patterns to assess new ideas against, making their value harder to recognize.
The study has a limitation: the AI was simply asked which proposal should be funded, rather than being given the same criteria human reviewers used. Susan Guthrie, director of science and emerging technology at RAND Europe in Cambridge, UK, notes that novelty itself is subjective and difficult to standardize. Even if AI tools were asked to assess novelty directly, there's no guarantee they could do so effectively.
UK Research and Innovation announced last month that it is exploring AI-assisted assessment through a controlled trial comparing AI-generated evaluations with past funding decisions. Wellcome in London has banned reviewers from inputting grant applications into generative AI tools due to privacy concerns, though it permits AI use for side tasks like checking whether similar research exists.
Understanding potential biases in AI assessment tools could help funders develop better safeguards, Machado says.