Dipankar Srirag

Room 217,

CSE Building (K-17),

University of NSW,

Sydney, NSW 2052

A graduate of Master of Information Technology from UNSW, with a strong background in NLP and Deep Learning. My master’s thesis involved adapting Large Language Models (LLMs) to other dialects of English.

As a research assistant, I work with Dr. Aditya Joshi on building a benchmark for sentiment and sarcasm classification for dialects of English. This project is funded by the Google Research Scholar grant.

As a casual academic, I tutor 50+ students for COMP9444: Deep Learning and Neural Networks, achieving exceptional satisfactory remarks.

news

Sep 04, 2024	Our preprint, Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models is on arXiv.
Aug 16, 2024	I make a presentation for Evaluating Dialect Robustness of Language Models via Conversation Understanding, at the Workshop for Cross Cultural Considerations in NLP, co-located at ACL 2024.
May 13, 2024	I start my role as Research Assistant at UNSW for the project: A benchmark for sentiment and sarcasm classification for dialects of English, headed by Dr. Aditya Joshi.
May 11, 2024	I join the Dept. of Computer Science and Engineering as a Casual Academic at UNSW, tutoring COMP9444: Neural Networks and Deep Learning taught by Dr. Alan Blair and Dr. Sonit Singh.
May 08, 2024	Our preprint, Evaluating Dialect Robustness of Large Language Models via Conversation Understanding is on arXiv.

selected publications

Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models

Dipankar Srirag, Aditya Joshi, and Jacob Eisenstein

2024

Abs PDF

Dialect adapters that improve the performance of LLMs for NLU tasks on certain sociolects/dialects/national varieties (’dialects’ for the sake of brevity) have been reported for encoder models. In this paper, we extend the idea of dialect adapters to decoder models in our architecture called LoRDD. Using MD-3, a publicly available dataset of word game-playing conversations between dialectal speakers, our task is Target Word Prediction (TWP) from a masked conversation. LoRDD combines task adapters and dialect adapters where the latter employ contrastive learning on pseudo-parallel conversations from MD-3. Our results for en-IN conversations on two models (Mistral and Gemma) show that LoRDD outperforms four baselines on TWP, while bridging the performance gap with en-US by 12% on word similarity and 25% on accuracy. The focused contribution of LoRDD is in its promise for dialect adaptation of decoder models.
Evaluating Dialect Robustness of Language Models via Conversation Understanding

Dipankar Srirag, Nihar Ranjan Sahoo, and Aditya Joshi

2024

Abs PDF

With an evergrowing number of LLMs reporting superlative performance for English, their ability to perform equitably for different dialects of English (i.e., dialect robustness) needs to be ascertained. Specifically, we use English language (US English or Indian English) conversations between humans who play the word-guessing game of ’taboo’. We formulate two evaluative tasks: target word prediction (TWP) (i.e., predict the masked target word in a conversation) and target word selection (TWS) (i.e., select the most likely masked target word in a conversation, from among a set of candidate words). Extending MD3, an existing dialectic dataset of taboo-playing conversations, we introduce M-MD3, a target-word-masked version of MD3 with the en-US and en-IN subsets. We create two subsets: en-MV (where en-US is transformed to include dialectal information) and en-TR (where dialectal information is removed from en-IN). We evaluate one open-source (Llama3) and two closed-source (GPT-4/3.5) LLMs. LLMs perform significantly better for US English than Indian English for both TWP and TWS tasks, for all settings, exhibiting marginalisation against the Indian dialect of English. While GPT-based models perform the best, the comparatively smaller models work more equitably after fine-tuning. Our error analysis shows that the LLMs can understand the dialect better after fine-tuning using dialectal data. Our evaluation methodology exhibits a novel way to examine attributes of language models using pre-existing dialogue datasets.