Lightnews — Scholar-powered news

Vilém Zouhar @zouharvi.bsky.social · 25d

I'd love to hear your feedback 🌿 arxiv.org/abs/2508.101...

See you in Suzhou! 🇨🇳

Estimating Machine Translation Difficulty

Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks. These high-quality outputs make it difficult to distinguish between state-of...

arxiv.org

3

Vilém Zouhar @zouharvi.bsky.social · 25d

My two biggest take-aways are:
- Standard testsets are too easy (Figure 1).
- We can make testsets that are not easy (Figure 2). 😎

1 3 17

Reposted by Vilém Zouhar

Tom Kocmi @kocmitom.bsky.social · Aug 23

We saw increased momentum in participation growth this year: 36 unique teams competing to improve the performance of MT. Furthermore, we added collected outputs of 24 popular LLMs and online systems. Reaching 50 evaluated systems in our annual benchmark.

1 1 3

Vilém Zouhar @zouharvi.bsky.social · Aug 7

It gets worse the more you look at it. Why is the height of 69.1 the same as the height of 30.8? Why are the labels rotated if there's enough space?

1 7

Vilém Zouhar @zouharvi.bsky.social · Jul 25

Organizers are happy to help with any questions. 🙂
Website with all details and contacts: www2.statmt.org/wmt25/mteval...

Shared task: Automated Translation Quality Evaluation Systems

www2.statmt.org

Vilém Zouhar @zouharvi.bsky.social · Jul 25

📐Task 3: Quality-informed segment-level error correction

Automatically post-edit machine-translated text using quality annotations to generate minimal and accurate corrections.

Description: www2.statmt.org/wmt25/mteval...

Submission platform: www.codabench.org/competitions...

QE-informed Segment-level Error Correction

www2.statmt.org

1

Vilém Zouhar @zouharvi.bsky.social · Jul 25

📐Task 2: Span-level error detection

Identify and locate translation errors within each segment (start/end indices) and classify their severity.

Description: www2.statmt.org/wmt25/mteval...

Submission platform: www.codabench.org/competitions...

Fine-grained error span detection

www2.statmt.org

1

Vilém Zouhar @zouharvi.bsky.social · Jul 25

📐Task 1: Segment-level quality score prediction

Predict a quality score for each source–target segment pair, using document-level context and either ESA or MQM annotations.

Description: www2.statmt.org/wmt25/mteval...

Submission platform: www.codabench.org/competitions...

MT Evaluation Subtask 1: Segment-Level Quality Score Prediction

www2.statmt.org

1

Vilém Zouhar @zouharvi.bsky.social · Jul 25

The 2025 MT Evaluation shared task brings together the strengths of the previous Metrics and Quality Estimation tasks under a single, unified evaluation framework.

The following tasks are now open for participants (deadline July 31st but participation has never been easier 🙂 ):

1 2 1

Vilém Zouhar @zouharvi.bsky.social · Jul 18

Faster but an extra dependency. 🤷

Vilém Zouhar @zouharvi.bsky.social · Jul 15

Not possible post-hoc but possible for the other direction! Thanks for your paper. 🙂

1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

Thank you everyone who helped. 😊

Special thanks to @mrinmaya.bsky.social and Peng Cui from @csateth.bsky.social and all my friends I bugged with proofreading. 😁

1 2

Vilém Zouhar @zouharvi.bsky.social · Jul 15

"How to Select Datapoints for Efficient Human Evaluation of NLG Models?" has now been accepted to TACL (a)! 🌿

📃 Paper (with nuances and caveats ): arxiv.org/abs/2501.182...
📦 Package: github.com/zouharvi/sub...
Feedback welcome!

How to Select Datapoints for Efficient Human Evaluation of NLG Models?

Human evaluation is the gold standard for evaluating text generation models. However, it is expensive. In order to fit budgetary constraints, a random subset of the test data is often chosen in practi...

arxiv.org

1 6

Vilém Zouhar @zouharvi.bsky.social · Jul 15

Recommendation based on translation and summarization:
1️⃣ if you have a good automatic metric, use variance/consistency
2️⃣ if not, use model output diversity
3️⃣ if outputs not available, use artificial crowd/distilled predictors
4️⃣ if those are not available, use source diversity

1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

We frame this as a 0/1 Knapsack problem: find a subset Y ⊆ X with maximum utility while staying under budget B. 🤓

maximize: ∑ zₓ · Utility(x)
subject to: ∑ zₓ · Cost(x) ≤ B
zₓ ∈ {0, 1}

The Utility(x) can be metric average, variance, diversity, etc.

1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

This works even if you don't have the model outputs yet.
1️⃣ "artificial crowd" simulate what model outputs would look like; apply the previous methods.
2️⃣ "utility predictors" estimate usefulness from the source text.
3️⃣ "source-based diversity" remove similar inputs.

1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

So what works? Selecting inputs that expose model differences:
1️⃣ high variance in metric scores
2️⃣ diversity in model outputs
3️⃣ high metric consistency with the rest of the dataset

We now need almost 30% fewer annotated examples to get the same model ranking.

1 1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

We frame this as finding the smallest subset of data (Y ⊆ X) that gives the same model ranking as on the full dataset.

Simply picking the hardest examples (lowest average metric score) is a step up but can backfire by selecting the most expensive items to annotate.

1

Vilém Zouhar @zouharvi.bsky.social · Jul 15

You have a budget to human-evaluate 100 inputs to your models, but your dataset is 10,000 inputs. Do not just pick 100 randomly!🙅

We can do better. "How to Select Datapoints for Efficient Human Evaluation of NLG Models?" shows how.🕵️
(random is still a devilishly good baseline)

2 3 33

Vilém Zouhar @zouharvi.bsky.social · Jul 9

docs.python.org/3.13/library...

statistics — Mathematical statistics functions

Source code: Lib/statistics.py This module provides functions for calculating mathematical statistics of numeric ( Real-valued) data. The module is not intended to be a competitor to third-party li...

docs.python.org

2

Vilém Zouhar @zouharvi.bsky.social · Jul 9

TIL that since python3.4 there's default `statistics` module with things like mean, mode, quantiles, variance, covariance, correlations, zscore, and more!. No more needless numpy imports!

2 5

Vilém Zouhar @zouharvi.bsky.social · Jul 7

Past iterations of the Terminology Shared Task don't come anywhere near the data quality and evaluation scrutiny of this one. In the era of LLM-as-MTs, participation has never been easier!

Kirill Semenov @kiryukhasemenov.bsky.social · Jun 6

📣Take part in 3rd Terminology shared task @WMT!📣
This year:
👉5 language pairs: EN->{ES, RU, DE, ZH},
👉2 tracks - sentence-level and doc-level translation,
👉authentic data from 2 domains: finance and IT!

www2.statmt.org/wmt25/termin...

Don't miss an opportunity - we only do it once in two years😏

Terminology Translation Task

www2.statmt.org

Vilém Zouhar @zouharvi.bsky.social · Jul 3

Thank you for your response. I will keep my score.

2 19

Vilém Zouhar @zouharvi.bsky.social · Jul 3

For the longest time I've been using Google Translate as a gateway to explain machine translation concepts to people as it's a tool that everyone knows. Now I get to contribute over the summer. 🌞

If you're near Mountain View, let's talk evaluation. 📏

1 15

Vilém Zouhar @zouharvi.bsky.social · May 31

I resigned on making it work natively and so I just typeset examples in different scripts in Typst/Inkscape, export to pdf, and include the PDF in Latex in tables. As silly as that is, still easier than making things work by changing the compiler or the babel package.

1 1