Aurélien Geron, an ML consultant, a former Googler and the author of Hands-on Machine Learning with Scikit-Learn, Keras and Tensorflow highlighted a bias in DistilBERT, a small, fast, cheap and light Transformer model. Aurelien plotted a sentiment analysis on the model for movies filmed in different countries. The resultant map showed the highest positive sentiment to movies filmed in India and the lowest for movies filmed in Iraq.
“I’m sure this model is used by analysts to measure the market’s sentiment in financial news feeds like Bloomberg’s. Are they compensating for the model’s country bias? I frankly doubt it,” Aurelien tweeted.
Aurelien’s tweet attracted a lot of comments from ML professionals. Nils Reimers, NLP researcher at huggingface.co said: “The issue is that the model does only have a positive and negative class, but no neutral class. Hence it has to predict some sentiment to this neutral statement which does not make much sense. So I mainly see an issue with the model design to only have positive/negative classes.”
Chidananda AV, another ML practitioner, asked: “Do you think the induced bias is due to the data involving film reviews which did not have a good distribution (while finetuning) or due to bias in data corpus while pretaining resulting in a negative/positive factor towards a topic(country in this subject).”
“I don’t think it’s movie-related at all. I think it’s because of a strong bias built up during pre-training. For example, Germany is one of the very few countries in Western Europe to have a negative bias, and I’m pretty sure that’s WW2 related rather than movie-related,” Aurelien replied.
