NATIONAL PHD IN ARTIFICIAL INTELLIGENCE

Thesis title: Towards Better Measurement of Progress in Machine Translation: Evaluation and Meta-Evaluation

Reliable evaluation is central to Machine Learning. Without it, progress cannot be measured, claims of improvement are hard to verify, and research is misdirected. In Machine Translation, evaluation is particularly difficult, as many perfectly valid translations exist for the same source. Both the automatic metrics used to score translations and the procedures used to judge the evaluative ability of those metrics must account for this. This thesis examines weaknesses in current practices for evaluating translations and for evaluating the evaluators, and proposes solutions that keep measurements of progress fair, robust, and informative. First, we show that standard Machine Translation meta-evaluation practices can distort metric rankings and introduce a revised protocol that corrects these effects. Second, we anchor metric rankings to human performance: by estimating human baselines and placing metrics and humans on the same scale, we make performance headroom explicit and show that claims of human parity are fragile. Third, we show that easy test sets blunt progress measurement; we introduce a methodology for identifying hard-to-translate source texts and use it to construct more difficult benchmarks that resist saturation and preserve headroom. Finally, we reframe automatic evaluation as a pairwise decision between two candidate translations, improving accuracy on high-quality outputs while remaining efficient. Taken together, these contributions keep progress in Machine Translation measurable and trustworthy: meta-evaluation that reflects genuine evaluative ability, benchmarks that remain challenging as systems improve, and automatic evaluation with resolution on high-quality outputs.