Bio

Simran Khanuja is a PhD student at the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University since August 2022. Her research focuses on expanding the capabilities of multimodal systems to serve a wide range of users across languages and cultures, with applications in localization, information access, conversational AI, education, and assistive technologies. Previously, she was a Pre-Doctoral Researcher at Google Research and worked at Microsoft Research. She has made contributions towards advancing under-represented languages in NLP and her work has been published at top NLP conferences like ACL and EMNLP, including best paper awards at EMNLP 2024, IEEE BigData 2024, and SLT 2022. She is also a recipient of the Waibel Presidential Fellowship for 2024-25 and has been recognized as a Rising Star in AI by the University of Michigan.

An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance

Title: An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance

Abstract: Given the rise of multimedia content, human translators increasingly focus on culturally adapting not only words but also other modalities such as images to convey the same meaning. While several applications stand to benefit from this, machine translation systems remain confined to dealing with language in speech and text. In this work, we take a first step towards translating images to make them culturally relevant. First, we build three pipelines comprising state-of-the-art generative models to do the task. Next, we build a two-part evaluation dataset: i) concept: comprising 600 images that are cross-culturally coherent, focusing on a single concept per image, and ii) application: comprising 100 images curated from real-world applications. We conduct a multi-faceted human evaluation of translated images to assess for cultural relevance and meaning preservation. We find that as of today, image-editing models fail at this task, but can be improved by leveraging LLMs and retrievers in the loop. Best pipelines can only translate 5% of images for some countries in the easier concept dataset and no translation is successful for some countries in the application dataset, highlighting the challenging nature of the task.