WWW 2024

Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries

Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, Srijan Kumar

Asked the same health questions in English, Spanish, Chinese and Hindi, GPT-3.5 gave fewer complete answers, less consistent answers and more answers that contradict the medical reference outside English. It fell furthest behind in Hindi.

The English question 'How often do I need an eyesight test?' above its Spanish translation, with a stack labeled Spanish / Chinese / Hindi, feeding an LLM.
One benchmark question in English and Spanish; each question was also translated into Chinese and Hindi, and every version was sent to the model separately. Adapted from Figure 1 of the paper.

The study starts from three English collections of health questions whose answers come from medical sources: HealthQA, built from articles on a patient health website; LiveQA, from frequently asked questions on U.S. National Institutes of Health sites; and MedicationQA, about medicines. Every question was machine-translated into Spanish, Chinese and Hindi, which together with English are the four most widely spoken languages in the world. GPT-3.5 answered each version. GPT-3.5 itself then compared its answer with the reference answer and gave it one of four labels: more comprehensive and appropriate (agrees with the reference and covers it more fully), less comprehensive and appropriate (agrees but covers less), neither contradictory nor similar (answers the question but gives different information), or contradictory (contradicts the reference and answers wrongly). Medical experts checked a sample of those labels.

HealthQA: how GPT-3.5’s answers compare with the reference answer
LabelEnglishSpanishChineseHindi
More comprehensive and appropriate1,013891878575
Less comprehensive and appropriate98175185402
Neither contradictory nor similar206357110
Contradictory351447
Number of answers, out of 1,134 HealthQA questions per language, that received each label; English and Hindi columns shaded. Redrawn from the HealthQA columns of Table 1 of the paper.

Start with the bottom row. Answering in English, GPT-3.5 contradicted the reference answer 3 times; answering the same questions in Hindi, it did so 47 times. The top row moves the other way: 1,013 English answers were judged more comprehensive and appropriate than the reference, against 575 Hindi answers. Spanish and Chinese fall in between. In the other two collections, Hindi again has the fewest answers in the top row and the most in the bottom row.

Two line charts for HealthQA. Similarity among repeated answers falls as temperature rises; the Hindi line is lowest at every temperature in both charts, and English and Spanish are close together at the top.
How similar a question’s 10 answers are to each other (1 = identical) as randomness (τ) rises. BERTScore compares meaning; simLDA20 compares topics. Red dashed line: Hindi. Adapted from Figure 3 of the paper (two of its 18 panels).

Asking the same question again shows a second gap. Some variation between repeated answers is expected, and it grows with temperature in every language. But at every temperature, and on both measures above, Hindi answers agree with one another least, while English and Spanish stay close together. The paper’s other panels, covering word overlap, answer length, sentence meaning and a second topic measure on all three collections, mostly show the same ordering. A third test reverses the model’s role: it sees a question with a candidate answer and must say whether that answer is correct. On HealthQA at temperature 1.0, its macro-F1 score, which runs from 0 to 1 and gives equal weight to the correct and the incorrect candidate answers, was 0.85 in both English and Spanish, 0.73 in Chinese and 0.65 in Hindi.

The study tested GPT-3.5 in 2023 on machine-translated questions, and its non-English answers were translated back into English before they were compared; the paper notes that translation quality may contribute to the gaps. Within those conditions the lesson is direct: health answers checked in English have not thereby been checked in Hindi, Chinese or Spanish, and each language needs its own evaluation.