Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries
Asked the same health questions in English, Spanish, Chinese and Hindi, GPT-3.5 gave fewer complete answers, less consistent answers and more answers that contradict the medical reference outside English. It fell furthest behind in Hindi.
The study starts from three English collections of health questions whose answers come from medical sources: HealthQA, built from articles on a patient health website; LiveQA, from frequently asked questions on U.S. National Institutes of Health sites; and MedicationQA, about medicines. Every question was machine-translated into Spanish, Chinese and Hindi, which together with English are the four most widely spoken languages in the world. GPT-3.5 answered each version. GPT-3.5 itself then compared its answer with the reference answer and gave it one of four labels: more comprehensive and appropriate (agrees with the reference and covers it more fully), less comprehensive and appropriate (agrees but covers less), neither contradictory nor similar (answers the question but gives different information), or contradictory (contradicts the reference and answers wrongly). Medical experts checked a sample of those labels.
| Label | English | Spanish | Chinese | Hindi |
|---|---|---|---|---|
| More comprehensive and appropriate | 1,013 | 891 | 878 | 575 |
| Less comprehensive and appropriate | 98 | 175 | 185 | 402 |
| Neither contradictory nor similar | 20 | 63 | 57 | 110 |
| Contradictory | 3 | 5 | 14 | 47 |
Start with the bottom row. Answering in English, GPT-3.5 contradicted the reference answer 3 times; answering the same questions in Hindi, it did so 47 times. The top row moves the other way: 1,013 English answers were judged more comprehensive and appropriate than the reference, against 575 Hindi answers. Spanish and Chinese fall in between. In the other two collections, Hindi again has the fewest answers in the top row and the most in the bottom row.

Asking the same question again shows a second gap. Some variation between repeated answers is expected, and it grows with temperature in every language. But at every temperature, and on both measures above, Hindi answers agree with one another least, while English and Spanish stay close together. The paper’s other panels, covering word overlap, answer length, sentence meaning and a second topic measure on all three collections, mostly show the same ordering. A third test reverses the model’s role: it sees a question with a candidate answer and must say whether that answer is correct. On HealthQA at temperature 1.0, its macro-F1 score, which runs from 0 to 1 and gives equal weight to the correct and the incorrect candidate answers, was 0.85 in both English and Spanish, 0.73 in Chinese and 0.65 in Hindi.
The study tested GPT-3.5 in 2023 on machine-translated questions, and its non-English answers were translated back into English before they were compared; the paper notes that translation quality may contribute to the gaps. Within those conditions the lesson is direct: health answers checked in English have not thereby been checked in Hindi, Chinese or Spanish, and each language needs its own evaluation.