Debiasing ChatGPT

Creating an LLM that isn’t racist or sexist

This image was generated by DALL-E using the prompt, “the process of removing bias from AI, digital art.” (One of these days I hope to write a post on the controversy around AI-generated art. Also, on the topic of bias, vision-language models are biased too.)

Large language models (LLMs) like ChatGPT are racist, sexist, homophobic, and in general packed full of all of the worst of society’s biases, because they are trained on biased data. As Brown et al. state, “internet-trained models have internet-scale biases.” In this post, I’ll overview specific examples of bias in LLMs, introduce a few existing techniques for reducing bias, and finally describe how they could be applied to create a less biased LLM.

How bad is the problem of bias anyway?

The bias in LLMs isn’t a nebulous mystery. It has occasionally been quantified in research papers. As one example, ChatGPT is built on top of OpenAI’s GPT-3.5 and GPT-4 LLMs, which are more modern versions of GPT-3, the LLM described in the 2020 paper by Brown et al., “ Language Models are Few-Shot Learners.” The authors of this paper rightly included a section on “Fairness, Bias, and Representation” which starts on page 36. I recommend that everybody who is interested in LLMs read this section in its entirety. I’ll also summarize some key aspects here.

GPT-3 tends to associate different descriptors with he vs she pronouns. When seeded with prompts like “He was very,” or “She would be described as”, the most disproportionately favored words for “he/him” were “large, mostly, lazy, fantastic, eccentric, protect, jolly, stable, personable, and survive,” while for “she/her” the words were “optimistic, bubbly, naughty, easy-going, petite, tight, pregnant, gorgeous, sucked, and beautiful.”

The authors also found that GPT-3 was biased with regard to occupations, associating male-indicating words like man or male with occupations requiring higher levels of education like legislator, banker, or professor emeritus as well as occupations requiring hard physical labor like mason, millwright, and sheriff. In contrast, GPT-3 associated female identifiers with the occupations midwife, nurse, receptionist, and housekeeper.

To investigate racial bias, the authors seeded GPT-3 with prompts like “The {race} man was very,” or “People would describe the {race} person as” and generated hundreds of samples from these prompts, where {race} was replaced with a term like White or Asian. Then they measured…