Revisiting Low Resource Status of Indian Languages in Machine Translation
Published in ACM India Joint International Conference on Data Science & Management of Data (CODS-COMAD) 2021, 2021
Recommended citation: Jerin Philip, Shashank Siripragada, Vinay P Namboodiri, CV Jawahar, “Revisiting Low Resource Status of Indian Languages in Machine Translation”, 8th ACM IKDD CODS and 26th COMAD (CODS COMAD 2021), January 2-4, 2021, Bangalore, India https://arxiv.org/abs/2008.04860
Indian language machine translation performance is hampered due to the lack of large scale multi-lingual sentence aligned corpora and robust benchmarks. Through this paper, we provide and analyse an automated framework to obtain such a corpus for Indian language neural machine translation (NMT) systems. Our pipeline consists of a baseline NMT system, a retrieval module, and an alignment module that is used to work with publicly available websites such as press releases by the government. The main contribution towards this effort is to obtain an incremental method that uses the above pipeline to iteratively improve the size of the corpus as well as improve each of the components of our system. Through our work, we also evaluate the design choices such as the choice of pivoting language and the effect of iterative incremental increase in corpus size. Our work in addition to providing an automated framework also results in generating a relatively larger corpus as compared to existing corpora that are available for Indian languages. This corpus helps us obtain substantially improved results on the publicly available WAT evaluation benchmark and other standard evaluation benchmarks.
Recommended citation: Jerin Philip, Shashank Siripragada, Vinay P Namboodiri, CV Jawahar, “Revisiting Low Resource Status of Indian Languages in Machine Translation”, 8th ACM IKDD CODS and 26th COMAD (CODS COMAD 2021), January 2-4, 2021, Bangalore, India