of the AI boom, the pace of technological iteration has reached an unprecedented level. Previous obstacles now seem to have viable solutions. This article serves as an “NMT 101” guide. While introducing our project, it also walks readers step by step through the process of fine-tuning an existing translation model to support a low-resource language that is not included in mainstream multilingual models.
Background: Dongxiang as a Low-Resource Language
Dongxiang is a minority language spoken in China’s Gansu Province and is classified as vulnerable by the UNESCO Atlas of the World’s Languages in Danger. Despite being widely spoken in local communities, Dongxiang lacks the institutional and digital support enjoyed by high-resource languages. Before diving into the training pipeline, it helps to briefly understand the language itself. Dongxiang, as its name suggests, is the mother tongue of the Dongxiang people. Descended from Central Asian groups who migrated to Gansu during the Yuan dynasty, the Dongxiang community has linguistic roots closely tied to Middle Mongol. From a writing-system perspective, Dongxiang has undergone a relatively recent standardization. Since the 1990s, with governmental promotion, the language has gradually adopted an official Latin-based orthography, using the 26 letters of the English alphabet and delimiting words by whitespace.
Although it is still classified under the Mongolic language family, due to the prolonged coexistence with Mandarin-speaking communities through history, the language has a trove of lexical borrowing from Chinese (Mandarin). Dongxiang exhibits no overt tense inflection or grammatical gender, which may be an advantage to simplify our model training.
Further background on the Dongxiang language and its speakers can be found on our website, which hosts an official English-language introduction released by the Chinese government.