TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained langua...
| Authors | Julien Knafou, Luc Mottin, Anaïs Mottaz et al. |
|---|---|
| Primary category | cs.CL |
| Published | 22 September 2026 |
| Last updated at source | 23 September 2026 |
Models and datasets referencing this paper
- jknafou/TransCorpus-bioDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.