Acerca de mí

Soy postdoctorante en el equipo de investigación en Data and Web Science de la Universidad de Mannheim.

Me intereso principalmente en la creación de datasets de gran tamaño para el entrenamiento de modelos de lenguaje, especialmente para idiomas de escasos recursos e idiomas históricos. Estoy interesado en tareas como Reconocimiento de entidades (NER), Análisis de dependencias y etiquetado de parte del discurso, Traducción automática y Estructuración de documentos.

Me encanta el café, las galletas y la programación. ☕🍪

Intereses
  • Modelación del lenguaje
  • Lingüística de corpus
  • Reconocimiento de entidades nombradas
  • Lingüística computacional
  • Traducción automática
Educación

  • Doctorado en Ciencias de la Computación, 2022

    Sorbonne Université

  • Pregrado MIASHS, 2018

    Université Paris 8

  • Master en Matemáticas, 2017

    Aix-Marseille Université

  • Pregrado en Matemáticas, 2016

    Universidad Nacional de Colombia

Publicaciones destacadas

A Data-driven Approach to Natural Language Processing for Contemporary and Historical French
We determine that the importance of the pre-training dataset size was largely overestimated, as we are able to repeatedly show that language models can be pre-trained with corpora of a modest size.
PDF Citar HAL
A Data-driven Approach to Natural Language Processing for Contemporary and Historical French
BERTrade: Using Contextual Embeddings to Parse Old French
We consider several neural language models, some of which trained or fine-tuned on a new corpus of raw Old and Middle French texts, and use their internal representations of words as inputs to train taggers and parsers on the SRCMF treebank.
PDF Citar Código fuente HAL
BERTrade: Using Contextual Embeddings to Parse Old French
From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French
We present the $\text{FreEM}_{\text{max}}$ corpus of Early Modern French and D’AlemBERT, a RoBERTa-based language model trained on $\text{FreEM}_{\text{max}}$.
PDF Citar DOI HAL arXiv
From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
We audit 5 multilingual corpora, finding that lower-resource corpora have systematic issues.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
PDF Citar Proyecto DOI TACL HAL arXiv
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR.
PDF Citar HAL arXiv
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus
We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data.
PDF Citar Código fuente Datos DOI CMLC-9 Website HAL
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
We explore the impact of the training corpus on contextualized word embeddings in five mid-resource languages.
PDF Citar Datos Proyecto Vídeo DOI ACL Anthology ACL 2020 HAL arXiv
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
CamemBERT: a Tasty French Language Model
We explore the impact of the training data size on a French version of RoBERTa.
PDF Citar Datos Proyecto Vídeo DOI ACL Anthology arXiv Website ACL 2020 HAL
CamemBERT: a Tasty French Language Model
Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures
We propose a new pipeline to filter, clean and classify Common Crawl by language, we publish the final corpus under the name OSCAR.
PDF Citar Código fuente Datos Proyecto Diapositivas DOI CMLC-7 Website HAL
Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures

Publicaciones recientes

A Data-driven Approach to Natural Language Processing for Contemporary and Historical French
We determine that the importance of the pre-training dataset size was largely overestimated, as we are able to repeatedly show that language models can be pre-trained with corpora of a modest size.
PDF Citar HAL
A Data-driven Approach to Natural Language Processing for Contemporary and Historical French
Le projet FREEM : ressources, outils et enjeux pour l’étude du français d’Ancien Régime
We present annotated corpora and NLP models for some downstream tasks in Early Modern French.
PDF Citar ACL Anthology
Le projet FREEM : ressources, outils et enjeux pour l’étude du français d’Ancien Régime
BERTrade: Using Contextual Embeddings to Parse Old French
We consider several neural language models, some of which trained or fine-tuned on a new corpus of raw Old and Middle French texts, and use their internal representations of words as inputs to train taggers and parsers on the SRCMF treebank.
PDF Citar Código fuente HAL
BERTrade: Using Contextual Embeddings to Parse Old French
From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French
We present the $\text{FreEM}_{\text{max}}$ corpus of Early Modern French and D’AlemBERT, a RoBERTa-based language model trained on $\text{FreEM}_{\text{max}}$.
PDF Citar DOI HAL arXiv
From FreEM to D'AlemBERT: a Large Corpus and a Language Model for Early Modern French
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
We audit 5 multilingual corpora, finding that lower-resource corpora have systematic issues.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
PDF Citar Proyecto DOI TACL HAL arXiv
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources
We developed our online catalogue as a supporting tool for gathering metadata through organized public hackathons.
PDF Citar HAL arXiv
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR.
PDF Citar HAL arXiv
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus
We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data.
PDF Citar Código fuente Datos DOI CMLC-9 Website HAL
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus
SinNer@CLEF-HIPE2020: Sinful Adaptation of SotA models for Named Entity Recognition in Historical French and German Newspapers
In this article we present the approaches developed by the Sorbonne-INRIA for NER (SinNer) team for the CLEF-HIPE 2020 challenge on Named Entity Processing on old newspapers.
PDF Citar Vídeo CEUR-WS CLEF-HIPE-2020 CLEF-2020 HAL
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
We explore the impact of the training corpus on contextualized word embeddings in five mid-resource languages.
PDF Citar Datos Proyecto Vídeo DOI ACL Anthology ACL 2020 HAL arXiv
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
CamemBERT: a Tasty French Language Model
We explore the impact of the training data size on a French version of RoBERTa.
PDF Citar Datos Proyecto Vídeo DOI ACL Anthology arXiv Website ACL 2020 HAL
CamemBERT: a Tasty French Language Model
Building a User-Generated Content North-African Arabizi Treebank: Tackling Hell
We introduce the first treebank for a romanized user-generated content variety of Algerian, a North-African Arabic dialect.
PDF Citar Vídeo DOI ACL Anthology ACL 2020
Les modèles de langue contextuels Camembert pour le Français : impact de la taille et de l'hétérogénéité des données d'entrainement
We explore the impact of the training data size and heterogeneity on French language modeling.
PDF Citar Datos Proyecto TALN 2020 HAL Website
Les modèles de langue contextuels Camembert pour le Français : impact de la taille et de l'hétérogénéité des données d'entrainement
Establishing a New State-of-the-Art for French Named Entity Recognition
We explore convert the NER annotations of the French TreeBank to a more user-friendly format and establish a new state of the art for French NER.
PDF Citar LREC 2020 HAL arXiv ACL Anthology
Establishing a New State-of-the-Art for French Named Entity Recognition
French Contextualized Word-Embeddings with a sip of CaBeRnet: a New French Balanced Reference Corpus
We investigate the impact of different types and size of training corpora on language models.
PDF Citar CMLC-8 ACL Anthology HAL
French Contextualized Word-Embeddings with a sip of CaBeRnet: a New French Balanced Reference Corpus
How OCR Performance can Impact on the Automatic Extraction of Dictionary Content Structures
We explore the impact of the OCR quality on grobid-dictionaries models.
PDF Citar Proyecto TEI 2019 HAL
Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures
We propose a new pipeline to filter, clean and classify Common Crawl by language, we publish the final corpus under the name OSCAR.
PDF Citar Código fuente Datos Proyecto Diapositivas DOI CMLC-7 Website HAL
Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures

Presentaciones recientes y próximas

Des Méthodes de TAL modernes pour l'Enrichissement de Documents
Nous présentons une pipeline pour le traitement et l’enrichissement de documents basée sur les dernières méthodes d’apprentissage neuronal.
sept. 22, 2020
Diapositivas Follow
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
We explore the impact of the training corpus on contextualized word embeddings in five mid-resource languages.
jul. 6, 2020
Diapositivas Vídeo Follow
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures
We propose a new pipeline to filter, clean and classify Common Crawl by language, we publish the final corpus under the name OSCAR.
jul. 22, 2019
PDF Código fuente Diapositivas Follow
Preparing the Dictionnaire Universel for Automatic Enrichment
A talk about automatic enrichment of dictionaries.
jun. 13, 2019
Diapositivas Follow
Reducing computation time by months by rewriting Bash scripts in Go
mar. 24, 2019
Código fuente Diapositivas Follow

Proyectos

*
All Language Modeling Corpus Linguistics Digital Humanities
abr. 27, 2016
BASNUM
BASNUM

Digitization and analysis of Basnage de Beauval’s Universal Dictionary: lexicography and scientific networks
abr. 27, 2016
CamemBERT
CamemBERT

A state-of-the-art language model for French.
abr. 27, 2016
OSCAR
OSCAR

OSCAR or Open Super-large Crawled Aggregated coRpus is a huge multilingual corpus

Contacto