This repository is for the Retracted-Science project where we probe multiple LLMs whether they are aware of the retraction status of an article based on their APA style reference.
In this experiment, we explicitly ask the LLMs on whether an article has been retracted given an APA style reference. The setups include variations with just the APA style reference and APA style reference with additional metadata of the article, where models are tested without web search, with websearch (the model decides whether to use websearch), forced websearch (the model creates a query, the query is executed, the results are injected into the prompt), forced websearch with a hand crafted query.
In this experiment, the models are given a list of APA styles of reference to search and summarize their content. The setups include variations where the prompt simple request the model to summarize each article, a prompt where the model is nugded to include a retraction notice if it exists in the summarization along with and without access to websearch.