Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Authors
Tarek Naous
Anagha Savit
Carlos Rafael Catalan
Geyang Guo
Jaehyeok Lee
Kyungdon Lee
Lheane Marie Dizon
Mengyu Ye
Neel Kothari
Sahajpreet Singh
Sarah Masud
Tanish Patwa
Trung Thanh Tran
Zohaib Khan
Alan Ritter
JinYeong Bak
Keisuke Sakaguchi
Tanmoy Chakraborty
Yuki Arase
Wei Xu
Abstract
As Large Language Models (LLMs) gain stronger multilingual capabilities, their ability to handle culturally diverse entities becomes crucial. Prior work has shown that LLMs often favor Western-associated entities in Arabic, raising concerns about cultural fairness. Due to the lack of multilingual benchmarks, it remains unclear if such biases also manifest in different non-Western languages. In this paper, we introduce Camellia, a benchmark for measuring entity-centric cultural biases in nine Asian languages spanning six distinct Asian cultures. Camellia includes 19,530 entities manually annotated for association with the specific Asian or Western culture, as well as 2,173 naturally occurring masked contexts for entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLM families across various tasks such as cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show a struggle by LLMs at cultural adaptation in all Asian languages, with performance differing across models developed in regions with varying access to culturally-relevant data. We further observe that different LLM families hold their distinct biases, differing in how they associate cultures with particular sentiments. Lastly, we find that LLMs struggle with context understanding in Asian languages, creating performance gaps between cultures in entity extraction.
