Language model pretraining has led to significant performance gains but careful comparison
between different approaches is challenging. Training is computationally expensive, often …

fairseq is an open-source sequence modeling toolkit that allows researchers and developers
to train custom models for translation, summarization, language modeling, and other text …

Consumers increasingly rate, review and research products online. Consequently, websites
containing consumer reviews are becoming targets of opinion spam. While recent work has …

Building open-domain chatbots is a challenging area for machine learning research. While
prior work has shown that scaling neural models in the number of parameters and the size of …

This paper shows that pretraining multilingual language models at scale leads to significant
performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-…

Large language models, which are often trained for hundreds of thousands of compute days,
have shown remarkable capabilities for zero- and few-shot learning. Given their …

In the field of artificial intelligence, a combination of scale in data and model capacity
enabled by unsupervised learning has led to major advances in representation learning and …

An effective method to improve neural machine translation with monolingual data is to
augment the parallel training corpus with back-translations of target language sentences. This …

This paper explores the environmental impact of the super-linear growth trends for AI from a
holistic perspective, spanning Data, Algorithms, and System Hardware. We characterize the …

Machine translation systems achieve near human-level performance on some languages, yet
their effectiveness strongly relies on the availability of large amounts of parallel sentences, …