Comparable evaluation data for use in automatic machine translation evaluations. The evaluation set consists of 500 sentences translated separately by four different professional human translators for each of the 11 official South African languages. This creates a set with four reference translations for each of the 11 languages where each of the texts can be used as an input text as well. This ensures that the evaluation set can be used to evaluate machine translation between any two of the 11 languages.