How to Score Translation Quality

Translation is not a simple task of replacing words from one language to another. There are many factors that can affect the quality of translation, such as the context, the tone, the style, the terminology, and the accuracy of the meaning. Therefore, it is important to have a systematic way of evaluating and measuring translation quality.

One of the methods that localization project managers (LPMs) use to assess translation quality is to score it based on the number and type of errors that occur in translation. Errors are deviations from the expected or desired quality standards that can negatively impact the user experience, the brand image, or the legal compliance of the product or service.

This article will focus on the following points:

• How to score translation quality using error categories and severity levels.

• The common error categories and issue types that are used in translation quality evaluation.

• How to assign weights and penalties to different types of errors and how to calculate the final language score for a translation project.

Error Categories and Issue Types

There are many ways to classify translation errors, but one of the widely used frameworks is based on five error categories: Language, Accuracy, Terminology, Style, and Zero-Weight. Each category contains several issue types that describe the specific nature of the error. Here is a brief overview of each category and its issue types:

Language: This category covers errors related to grammar, spelling, punctuation, and inconsistency.

• Grammar errors are mistakes in syntax, morphology, or word order that affect the grammaticality of the sentence.

• Spelling errors are mistakes in orthography or diacritics that affect the readability of the text.

• Punctuation errors are mistakes in using commas, periods, quotation marks, etc. that affect the clarity and flow of the text.

• Inconsistency errors are mistakes in using different terms, formats, or styles for the same concept or element within or across documents.

Accuracy: This category covers errors related to addition, omission, and mistranslation.

• Addition errors are mistakes in adding extra information or details that are not present in the source text.

• Omission errors are mistakes in leaving out important information or details that are present in the source text.

• Mistranslation errors are mistakes in conveying a different meaning or message than what is intended by the source text.

Terminology: This category covers errors related to glossary violations and using terms industry not accepted.

• Glossary violation errors are mistakes in using terms that are different from or inconsistent with the approved glossary or terminology list for the project or client.

• Industry-not-accepted errors are mistakes in using terms that are not commonly used or accepted by the industry or domain of the product or service.

Style: This category covers errors related to style guide violation, unidiomatic expression, and register.

• Style guide violation errors are mistakes in using language elements that are different from or inconsistent with the approved style guide or voice and tone guidelines for the project or client.

• Unidiomatic expression errors are mistakes in using expressions that are unnatural, awkward, or incorrect in the target language or culture.

• Register errors are mistakes in using language levels that are inappropriate for the intended audience or purpose of the text.

Zero-Weight: This category covers issues that do not affect translation quality but may require attention or feedback from the LPMs. These include client edits and source issues.

• Client edit issues are changes made by the client after receiving the translation that do not improve or worsen translation quality but reflects their personal preference or opinion.

• Source issue issues are problems or ambiguities in the source text that affect translation quality but are beyond the control of the translator.

  
  

Severity Level

In addition to categorizing translation errors by their type, it is also important to categorize them by their severity level. Severity level indicates how serious or impactful an error is on translation quality. Different severity levels may have different penalties or consequences for translation quality evaluation.

There are four common severity levels: Critical, Major, Minor, and Neutral. Here is a brief description of each level:

• Critical: This level indicates an error that has a severe impact on translation quality and may cause serious harm or damage to the user, the client, or the product or service.

• Major: This level indicates an error that has a significant impact on translation quality and may cause confusion or dissatisfaction for the user or the client.

• Minor: This level indicates an error that has a minor impact on translation quality and may cause slight inconvenience or annoyance for the user or the client.

• Neutral: This level indicates an issue that has no impact on translation quality but may require attention or feedback from the LPMs.

  
  

Language Score Calculation

After identifying and categorizing the errors in translation, the next step is to calculate the language score for the translation project. Language score is a numerical value that represents the overall quality of translation based on the number and type of errors found.

There are many ways to calculate language score, but one of the common methods is based on the following steps:

1. Assigning weights to different issue types: Different issue types may have different consequences or implications for translation quality. Therefore, they need different weights or penalties to reflect their importance or their impact on translation. For example, a mistranslation error may have a higher weight than a punctuation error because the alternation of the meaning and message causes more harm than clarity and flow. The LPMs may also need to focus on resolving certain types of issues more than others depending on the project or client requirements. They may increase the weight or penalty of those issues to bring more awareness or attention to translators.

2. Assign weights to different severity levels: Different severity levels may also have different consequences or implications for translation quality. Therefore, they need different weights or penalties to reflect their seriousness or impact. For example, a critical error may have a higher weight than a minor error because it affects the user experience and the brand image more than the convenience and satisfaction. The LPMs may also need to adjust the weights or penalties of different severity levels depending on the project or client expectations. They may increase or decrease the weight or penalty of certain severity levels to align with the quality standards or goals.

3. Calculate error points reported: Error points reported are the total number of points deducted from the translation quality based on the number, severity levels, and type of errors found. To calculate error points reported, we multiply the number of errors by the category penalty and the error severity penalty for each issue type and severity level combination. For example, if there are 3 mistranslation errors with major severity level, and the category penalty for mistranslation is 5 and the error severity penalty for major is 3, then the error points reported for this combination are 3 x 5 x 3 = 45.

4. Assign error points allowed: Error points allowed are the maximum number of points that can be deducted from translation quality. Different types of content may have different requirements or expectations for translation quality. For example, legal documents may have a higher requirement for accuracy, while marketing materials may have a higher requirement for style. The LPMs need to assign error points allowed for each type of content and error category based on their analysis and experience. Below is an example:

5. Calculate final score: Final score is the final numerical value that represents the overall quality of translation after deducting the error points reported from 100. Final score is usually calculated based on errors per 1000 words to normalize the score across different document lengths. To calculate final score, we subtract error points reported (per 1000 words) from 100 and divide by error points allowed (per 1000 words). For example, if there are 60 error points reported in a document with 2000 words, and the error points allowed are 80 per 1000 words, then the final score is (100 - (60 / 2000) x 1000) / 80 = 93.75.

6. Set minimum pass score: Minimum pass score is the minimum final score that is required for a translation project to be accepted or approved by the LPMs or the clients. Minimum pass score may vary depending on the type of content, the project scope, or the client specifications. The LPMs need to set minimum pass score based on their judgment and negotiation with the clients. For example, they may set minimum pass score as 90 for legal documents and 80 for marketing materials.

Example

Conclusion

Translation quality evaluation is an essential part of localization that ensures that the product or service meets the needs and expectations of the target market or culture. By categorizing and quantifying translation errors based on their type and severity level, LPMs can measure and improve translation quality using language score as a metric. Language score calculation involves assigning weights and penalties to different types of errors and setting error points allowed and minimum pass score for different types of content. By using this method, LPMs can ensure that translation quality meets their standards and goals.