Is ChatGPT a Reliable Translator?

ChatGPT performs competitively with commercial translation products

ChatGPT is a flexible tool that may be applied in a variety of ways to improve learning and productivity. ChatGPT may be your go-to resource for information, whether you're looking for quick answers to trivia questions or in-depth discussions of challenging ideas.

The AI chatbot is building a reputation as a prospective translation tool that might compete with services like Google Translate in the language services sector, which is also similar.

It's interesting to note that ChatGPT lists "translating text" as one of its possible use cases. Given the fanfare that this technology is now receiving, it should come as no surprise that language professionals have been putting it to the test to determine just how effective it is as a translation tool.

How good is ChatGPT at translating text?

Users must enter a text prompt into ChatGPT, which then creates a text answer. ChatGPT is a text-only system. This is important because a tool like the Google Translate mobile app can translate spoken language in real-time as well as text that appears in photos thanks to its Lens technology.

ChatGPT vs Google Translate

To test ChatGPT and Google Translate, we choose a tweet from a non-English account at random. The in question tweet was posted by the Spanish news outlet El PAS. We copied and pasted the text into the box provided to translate this tweet using ChatGPT:

"Translate 'Descubre todas las actuaciones musicales, horarios, estrellas que entregan galardones, favoritos y un concurso: una guía de la 37ª edición de los Goya' into English"

We received the following response from ChatGPT:

"Discover all the musical performances, schedules, stars that award prizes, favorites, and a contest: a guide to the 37th edition of the Goya Awards."

We didn't need a prompt for the Google Translate test; we just pasted the Spanish text into the translation tool. Also, this is what Google Translate produced in English:

"Discover all the musical performances, schedules, award-winning stars, favorites and a contest: a guide to the 37th edition of the Goya Awards"

The answers are very similar when you look at the two results, but there is one important difference: ChatGPT translated the verb entregar to reward and Google Translate translated it to win or get.

ChatGPT is correct than with the sanctioned restatements for entregar including: to hand over, to give, to present, and to give out.

So, in this particular illustration, the restated "award-winning stars" provided by Google Translate is not correct.

By accessing several standard test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) in high-resource European languages but lags significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical objectification or Reddit commentary but is potentially a good translator for spoken language.