Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

Abstract

This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot}\mathbf{prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator

Analyzed/compared the outputs between ChatGPT and Google Translate; both automatic and human evaluation

Cited by in corpus (15)

• ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
• Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• Towards Interpretable Mental Health Analysis with Large Language Models
• LLM for SoC Security: A Paradigm Shift
• The Perils & Promises of Fact-checking with Large Language Models
• Domain Terminology Integration into Machine Translation: Leveraging Large Language Models
• UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction
• Computational Argumentation-based Chatbots: a Survey
• LegiGPT: Party Politics and Transport Policy with Large Language Model
• Perceived Trustworthiness of Natural Language Generators
• Learning to Describe for Predicting Zero-shot Drug-Drug Interactions
• ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models
• Missing the human touch? A computational stylometry analysis of GPT-4 translations of online Chinese literature