How we measure AI translation quality
You should be able to judge a translation without reading the target language, and defend it to your stakeholders. Here is how we measure AI translation quality: how we classify each issue, how we judge its severity, what the quality gate checks, and what you receive with each delivery.
Every delivery includes the quality-gate result and an issue log. At AI + Edit, sampled quality reports follow the schedule agreed for your program and can cover several small releases; at AI + Edit + Review, each delivery gets a full report.
How we classify AI translation quality issues
We classify issues with an MQM-based error typology. Multidimensional Quality Metrics (MQM) is a public industry framework for describing translation errors. We use its approach to categories and severity levels, adapted to the content we translate, so your reviewers can read our findings in terms they already know.
A finding counts only when it names a concrete defect and points to a reference: the source, your glossary, a rule, or your instructions. Preferences are logged separately and never block delivery.
| Category | What it covers | Example |
|---|---|---|
| Accuracy | The target says something the source does not: a mistranslation, an omission, an addition, or a changed number | “Non-perishable” translated as “perishable” |
| Fluency | Grammar, spelling, and punctuation | An adjective that does not agree with the noun it describes |
| Terminology | A term that does not match your glossary, or a term used inconsistently | A service name translated when your glossary keeps it in English |
| Style | Tone, register, unidiomatic phrasing that reads like a translation, and wording that does not follow your style guide | A literal phrase a native speaker would not write, or promotional wording where your style guide asks for a neutral tone |
| Locale | Conventions of the target market: units, dates, number formats, and regional vocabulary | Gallons left unconverted for readers who use liters |
| Tags and format | Tags, placeholders, variables, line breaks, punctuation pairs, and length limits | A broken placeholder or a missing closing parenthesis |
| Client requirements | Instructions you gave us for your program | Interface labels left in English when your instructions ask for the labels in your translated interface |
Critical, major, and minor
Every issue gets one of three severity levels. Severity depends on the likely impact on your readers and your business, in context, not on the size of the fix. A one-word change can be critical.
| Severity | Definition | Example |
|---|---|---|
| Critical | The issue renders the content unfit for its purpose, or creates a serious safety, security, legal, financial, or reputational risk. | A security instruction reversed, so readers are told they may share their sign-in codes with anyone claiming to be support |
| Major | The issue changes or obscures the meaning, or breaks one of your requirements in a way readers would notice. It does not carry the risk of a critical issue. | A key concept mistranslated, or an instruction you gave us ignored in a way readers would notice |
| Minor | The issue does not change the meaning or stop the reader, but the text falls short of how a native speaker would write it, or of one of your rules. | A regional word for the wrong market, or wordy phrasing |
A quality gate that holds back any file that fails
An automated quality gate runs on every delivered file, at every service level. It checks what code can check reliably: tags, placeholders, numbers, locked terms, locale rules, consistency, and the testable rules in your own rulebook. If the file fails, it does not ship: the issue goes back into the cycle to be fixed, and the gate runs again. If a delivery is at risk, we tell you before the agreed date.
Your style rules and instructions go into your rulebook. We turn the rules that can be tested reliably into automatic checks. Linguists judge meaning, tone, and anything else that needs judgment. Each client’s rules live in that client’s rulebook only.
If the gate flags something that is not an error, we fix the gate’s rule and record the case in your delivery note. We never edit your file just to make it pass.
The gate checks the mechanics, and people make the language judgments. At AI + Edit and AI + Edit + Review, a reviewer’s language quality assessment (LQA) can also be measured against an acceptance level, if your contract sets one. An acceptance level is a pass-or-fail threshold that we agree with you before the pilot. Issues are weighted by severity, so one critical issue counts for more than several minor ones, and the result is set against the number of words assessed. Your reports and your monthly summary state the method and the result, and your contract sets what happens if a delivery falls short.
What the quality report contains
The quality report is the evidence you can take to your stakeholders: what was checked, what was found, and how each issue was settled. AI + Edit reports are based on a sample and follow the schedule agreed for your program, so one report can cover several small releases. AI + Edit + Review reports cover the whole file, with every delivery. At AI + Check, each delivery comes with the quality-gate result and an issue log of the check.
A quality report contains:
- The job, the service level, the scope, and the steps that ran
- How the sample was drawn, and its size
- Issue counts by severity and category
- The quality-gate result, and the acceptance result if your contract sets an acceptance level
- An issue log: segment, source, target, category, severity, explanation, fix, and status
- Glossary changes and open queries
- Lessons we applied from your reviewers’ last round of edits
Each report carries the label “Assessment by Locstars.” Any figures in a report describe your own content. We don’t publish quality scores on this site.
An example report
To show the format, we published an example built on an original help article that we wrote for this purpose: a short guide to turning on two-step verification in a fictional product, translated from English into French at AI + Edit + Review. It shows the job summary with the quality-gate result, the issue counts, the issue log with a correction and a note for every issue, the glossary changes and open queries, and the delivered French text. The product, the client, and the instructions are invented. It illustrates the format and is not a rating of our work. To have your own AI or machine translation (MT) output assessed this way, see quality review of your AI output.
Corrections, and lessons for your next batch
If you find a language issue within the agreed scope, we correct it free of charge. Send the segment and your comment to info@locstars.com, and we log it like any other finding.
Every edit your reviewer makes is compared with what we delivered. Each confirmed lesson becomes a glossary entry, a rule or an automatic check before your next batch. When your reviewer’s edit is better than our fix, we say so.