Study Design

In this study, participants read a news article and answer questions about their perception of bias in the article. Afterward, we split them into three groups. The first group read a ChatGPT-generated explanation of the article’s bias. The second group read an explanation written by a human media-bias expert. The third group, the control group, read a summary of the article. Finally, we asked participants about their bias perception again. We were very curious to see who explains bias better—humans or machines!

Results

It turns out that machines lead to a significantly greater change in the bias perception!

BUT.

When we looked into the data, we saw that this was because of the low bias articles. When ChatGPT explained the bias content of a low bias article, the bias perception of our readers shot up significantly… Even tho this meant that they diverged from the experts more than before the explanation!

With the human explanation, we saw that the explanations shifted the perception towards the accurate bias level, in low, medium, and high bias articles alike. With ChatGPT, we saw that accurate shift only in medium and high bias articles.

What we take away

We conclude that ChatGPT can be used to explain bias when there is a significant amount of bias but should be avoided on low bias articles. In our paper we make the point of ChatGPT and generally, LLMs, acting as “plausibility machines” that will come up with a plausible and therefore often persuading answer. This means that prompt design is crucial…because as soon as you ask “pls explain bias 👉🏼👈🏼”, ChatGPT will be like “YES okay WOW. You’re so right, you’re such a genius! This center article is SO BIASED and here is a 10-point list why 💥”.

… And we should be aware of that if we’re considering integrating LLMs into our applications. Which, yes, we knew, but now we have more data supporting it 🤓