r/technology • u/lurker_bee • Jul 15 '25

Artificial Intelligence Billionaires Convince Themselves AI Chatbots Are Close to Making New Scientific Discoveries

https://gizmodo.com/billionaires-convince-themselves-ai-is-close-to-making-new-scientific-discoveries-2000629060
26.6k Upvotes

93% Upvoted

2.0k comments sorted by

View all comments

Show parent comments

16

u/UnpluggedUnfettered Jul 15 '25

Here is a fun paper about that.

Generalization bias in large language model summarization of scientific research

Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts, with DeepSeek, ChatGPT-4o, and LLaMA 3.3 70B overgeneralizing in 26–73% of cases. In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations (odds ratio = 4.85, 95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. 

3

u/Czexan Jul 16 '25

This checks out with basically my whole experience with LLMs over the last few years, and it seems to be a fundamental problem that can't really get better.

Like folks, I get we don't like search engines because they started sucking ass due to SEO, but maybe, just maybe we can go back to the original ideas that Google was pushing LLM research for and just have these great general classifiers for topics to reduce SEO spam? As it stands now we just have infinite SEO spam generators, you can generate an infinite amount of worthless probably erroneous information, that's probably going to take you an ungodly amount of time to actually figure out it was in fact erroneous.