NewsUnfold[1] is a news-reading platform designed to detect and highlight linguistic media bias, and gather user feedback to improve media bias detection models. The platform can additionally integrate a news nutrition label to increase bias awareness.
Problem:
- Media bias in online news influences public opinion and decision-making.
- Current bias detection methods face challenges in reliability, accuracy, and cost, mainly because there isn’t enough data.
Human-in-the-Loop (HITL) Approach:
- Readers provided voluntary feedback on machine-generated bias highlights, improving the dataset’s quality.
Data Collection & Classifier Improvement:
- NewsUnfold collected ~2,000 annotations, creating the NewsUnfold Dataset (NUDA).
- It gathered reliable annotations, achieving 90.97% agreement with expert labels after filtering out unreliable data.
- Inter-annotator agreement (IAA) increased by 26.31% over the original classifiers’ annotations.
- Classifier performance improved by 2.49% when adding the reader feedback data.
UX Study/Reader Feedback:
- The platform was user-friendly and intuitive, with participants expressing positive feedback on the interface.
Applications and Future Potential:
- Could be applied to other digital platforms like news aggregators or social media to raise media bias awareness.
- Value for readers through bias highlights.
- Scalable and cost-effective method for creating media bias datasets.
NewsUnfold showed that HITL feedback mechanisms are a promising way to gather data and improve media bias detection.
References
- (2025): NewsUnfold: Creating a News-Reading Application That Indicates Linguistic Media Bias and Collects Feedback. In: Proceedings of the international AAAI conference on web and social media (ICWSM'25), AAAI, Copenhagen, Denmark, 2025.