Publications

2024

1. Language Model Priors and Data Augmentation Strategies for Low-resource Machine Translation: A Case Study Using Finnish to Northern Sámi

   Jonne Sälevä, and Constantine Lignos

   In Findings of the Association for Computational Linguistics: ACL 2024, Aug 2024

   Abs

   We investigate ways of using monolingual data in both the source and target languages for improving low-resource machine translation. As a case study, we experiment with translation from Finnish to Northern Sámi.Our experiments show that while conventional backtranslation remains a strong contender, using synthetic target-side data when training backtranslation models can be helpful as well.We also show that monolingual data can be used to train a language model which can act as a regularizer without any augmentation of parallel data.

2. CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English

   Andrew Rueda, Elena Alvarez-Mellado, and Constantine Lignos

   In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

   Abs

   Modern named entity recognition systems have steadily improved performance in the age of larger and more powerful neural models. However, over the past several years, the state-of-the-art has seemingly hit another plateau on the benchmark CoNLL-03 English dataset. In this paper, we perform a deep dive into the test outputs of the highest-performing NER models, conducting a fine-grained evaluation of their performance by introducing new document-level annotations on the test set. We go beyond F1 scores by categorizing errors in order to interpret the true state of the art for NER and guide future work. We review previous attempts at correcting the various flaws of the test set and introduce CoNLL#, a new corrected version of the test set that addresses its systematic and most prevalent errors, allowing for low-noise, interpretable error analysis.

3. ParaNames 1.0: Creating an Entity Name Corpus for 400+ Languages Using Wikidata

   Jonne Sälevä, and Constantine Lignos

   In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

   Abs

   We introduce ParaNames, a massively multilingual parallel name resource consisting of 140 million names spanning over 400 languages. Names are provided for 16.8 million entities, and each entity is mapped from a complex type hierarchy to a standard type (PER/LOC/ORG). Using Wikidata as a source, we create the largest resource of this type to date. We describe our approach to filtering and standardizing the data to provide the best quality possible. ParaNames is useful for multilingual language processing, both in defining tasks for name translation/transliteration and as supplementary data for tasks such as named entity recognition and linking. We demonstrate the usefulness of ParaNames on two tasks. First, we perform canonical name translation between English and 17 other languages. Second, we use it as a gazetteer for multilingual named entity recognition, obtaining performance improvements on all 10 languages evaluated.

4. QueryNER: Segmentation of E-commerce Queries

   Chester Palen-Michel, Lizzie Liang, Zhe Wu, and Constantine Lignos

   In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

   Abs

   We present QueryNER, a manually-annotated dataset and accompanying model for e-commerce query segmentation. Prior work in sequence labeling for e-commerce has largely addressed aspect-value extraction which focuses on extracting portions of a product title or query for narrowly defined aspects. Our work instead focuses on the goal of dividing a query into meaningful chunks with broadly applicable types. We report baseline tagging results and conduct experiments comparing token and entity dropping for null and low recall query recovery. Challenging test sets are created using automatic transformations and show how simple data augmentation techniques can make the models more robust to noise. We make the QueryNER dataset publicly available.

2023

1. LR-Sum: Summarization for Less-Resourced Languages

   Chester Palen-Michel, and Constantine Lignos

   In Findings of the Association for Computational Linguistics: ACL 2023, Jul 2023

   Abs

   We introduce LR-Sum, a new permissively-licensed dataset created with the goal of enabling further research in automatic summarization for less-resourced languages.LR-Sum contains human-written summaries for 40 languages, many of which are less-resourced. We describe our process for extracting and filtering the dataset from the Multilingual Open Text corpus (Palen-Michel et al., 2022).The source data is public domain newswire collected from from Voice of America websites, and LR-Sum is released under a Creative Commons license (CC BY 4.0), making it one of the most openly-licensed multilingual summarization datasets. We describe abstractive and extractive summarization experiments to establish baselines and discuss the limitations of this dataset.

2. What changes when you randomly choose BPE merge operations? Not much.

   Jonne Saleva, and Constantine Lignos

   In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, May 2023

   Abs

   We introduce two simple randomized variants of byte pair encoding (BPE) and explore whether randomizing the selection of merge operations substantially affects a downstream machine translation task. We focus on translation into morphologically rich languages, hypothesizing that this task may show sensitivity to the method of choosing subwords. Analysis using a Bayesian linear model indicates that one variant performs nearly indistinguishably compared to standard BPE while the other degrades performance less than we anticipated. We conclude that although standard BPE is widely used, there exists an interesting universe of potential variations on it worth investigating. Our code is available at: https://github.com/bltlab/random-bpe.

3. Improving NER Research Workflows with SeqScore

   Constantine Lignos, Maya Kruse, and Andrew Rueda

   In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), Dec 2023

   Abs

   We describe the features of SeqScore, an MIT-licensed Python toolkit for working with named entity recognition (NER) data.While SeqScore began as a tool for NER scoring, it has been expanded to help with the full lifecycle of working with NER data: validating annotation, providing at-a-glance and detailed summaries of the data, modifying annotation to support experiments, scoring system output, and aiding with error analysis.SeqScore is released via PyPI (https://pypi.org/project/seqscore/) and development occurs on GitHub (https://github.com/bltlab/seqscore).

2022

1. Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling

   Elena Álvarez-Mellado, and Constantine Lignos

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022

   Abs

   This work presents a new resource for borrowing identification and analyzes the performance and errors of several models on this task. We introduce a new annotated corpus of Spanish newswire rich in unassimilated lexical borrowings—words from one language that are introduced into another without orthographic adaptation—and use it to evaluate how several sequence labeling models (CRF, BiLSTM-CRF, and Transformer-based models) perform. The corpus contains 370,000 tokens and is larger, more borrowing-dense, OOV-rich, and topic-varied than previous corpora available for this task. Our results show that a BiLSTM-CRF model fed with subword embeddings along with either Transformer-based embeddings pretrained on codeswitched data or a combination of contextualized word embeddings outperforms results obtained by a multilingual BERT-based model.

2. MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition

   David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, and 20 more authors

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Dec 2022

   Abs

   African languages are spoken by over a billion people, but they are under-represented in NLP research and development. Multiple challenges exist, including the limited availability of annotated training and evaluation datasets as well as the lack of understanding of which settings, languages, and recently proposed methods like cross-lingual transfer will be effective. In this paper, we aim to move towards solutions for these challenges, focusing on the task of named entity recognition (NER). We present the creation of the largest to-date human-annotated NER dataset for 20 African languages. We study the behaviour of state-of-the-art cross-lingual transfer methods in an Africa-centric setting, empirically demonstrating that the choice of source transfer language significantly affects performance. While much previous work defaults to using English as the source language, our results show that choosing the best transfer language improves zero-shot F1 scores by an average of 14% over 20 languages as compared to using English.

3. Toward More Meaningful Resources for Lower-resourced Languages

   Constantine Lignos, Nolan Holley, Chester Palen-Michel, and Jonne Sälevä

   In Findings of the Association for Computational Linguistics: ACL 2022, May 2022

   Abs

   In this position paper, we describe our perspective on how meaningful resources for lower-resourced languages should be developed in connection with the speakers of those languages. Before advancing that position, we first examine two massively multilingual resources used in language technology development, identifying shortcomings that limit their usefulness. We explore the contents of the names stored in Wikidata for a few lower-resourced languages and find that many of them are not in fact in the languages they claim to be, requiring non-trivial effort to correct. We discuss quality issues present in WikiAnn and evaluate whether it is a useful supplement to hand-annotated data. We then discuss the importance of creating annotations for lower-resourced languages in a thoughtful and ethical way that includes the language speakers as part of the development process. We conclude with recommended guidelines for resource development.

4. Multilingual Open Text Release 1: Public Domain News in 44 Languages

   Chester Palen-Michel, June Kim, and Constantine Lignos

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022

   Abs

   We present a Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001–2022 and collected from Voice of America‘s news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.

5. Borrowing or Codeswitching? Annotating for Finer-Grained Distinctions in Language Mixing

   Elena Alvarez-Mellado, and Constantine Lignos

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022

   Abs

   We present a new corpus of Twitter data annotated for codeswitching and borrowing between Spanish and English. The corpus contains 9,500 tweets annotated at the token level with codeswitches, borrowings, and named entities. This corpus differs from prior corpora of codeswitching in that we attempt to clearly define and annotate the boundary between codeswitching and borrowing and do not treat common “internet-speak” (lol, etc.) as codeswitching when used in an otherwise monolingual context. The result is a corpus that enables the study and modeling of Spanish-English borrowing and codeswitching on Twitter in one dataset. We present baseline scores for modeling the labels of this corpus using Transformer-based language models. The annotation itself is released with a CC BY 4.0 license, while the text it applies to is distributed in compliance with the Twitter terms of service.

2021

1. TMR: Evaluating NER Recall on Tough Mentions

   Jingxuan Tu, and Constantine Lignos

   In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, Apr 2021

   Abs

   We propose the Tough Mentions Recall (TMR) metrics to supplement traditional named entity recognition (NER) evaluation by examining recall on specific subsets of ”tough” mentions: unseen mentions, those whose tokens or token/type combination were not observed in training, and type-confusable mentions, token sequences with multiple entity types in the test data. We demonstrate the usefulness of these metrics by evaluating corpora of English, Spanish, and Dutch using five recent neural architectures. We identify subtle differences between the performance of BERT and Flair on two English NER corpora and identify a weak spot in the performance of current models in Spanish. We conclude that the TMR metrics enable differentiation between otherwise similar-scoring systems and identification of patterns in performance that would go unnoticed from overall precision, recall, and F1.

2. The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation

   Jonne Saleva, and Constantine Lignos

   In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, Apr 2021

   Abs

   This paper evaluates the performance of several modern subword segmentation methods in a low-resource neural machine translation setting. We compare segmentations produced by applying BPE at the token or sentence level with morphologically-based segmentations from LMVR and MORSEL. We evaluate translation tasks between English and each of Nepali, Sinhala, and Kazakh, and predict that using morphologically-based segmentation methods would lead to better performance in this setting. However, comparing to BPE, we find that no consistent and reliable differences emerge between the segmentation methods. While morphologically-based methods outperform BPE in a few cases, what performs best tends to vary across tasks, and the performance of segmentation methods is often statistically indistinguishable.

3. SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation

   Chester Palen-Michel, Nolan Holley, and Constantine Lignos

   In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, Nov 2021

   Abs

   To address a looming crisis of unreproducible evaluation for named entity recognition, we propose guidelines and introduce SeqScore, a software package to improve reproducibility. The guidelines we propose are extremely simple and center around transparency regarding how chunks are encoded and scored. We demonstrate that despite the apparent simplicity of NER evaluation, unreported differences in the scoring procedure can result in changes to scores that are both of noticeable magnitude and statistically significant. We describe SeqScore, which addresses many of the issues that cause replication failures.

4. Macro-Average: Rare Types Are Important Too

   Thamme Gowda, Weiqiu You, Constantine Lignos, and Jonathan May

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun 2021

   Abs

   While traditional corpus-level evaluation metrics for machine translation (MT) correlate well with fluency, they struggle to reflect adequacy. Model-based MT metrics trained on segment-level human judgments have emerged as an attractive replacement due to strong correlation results. These models, however, require potentially expensive re-training for new domains and languages. Furthermore, their decisions are inherently non-transparent and appear to reflect unwelcome biases. We explore the simple type-based classifier metric, MacroF1, and study its applicability to MT evaluation. We find that MacroF1 is competitive on direct assessment, and outperforms others in indicating downstream cross-lingual information retrieval task performance. Further, we show that MacroF1 can be used to effectively compare supervised and unsupervised neural machine translation, and reveal significant qualitative differences in the methods’ outputs.

5. MasakhaNER: Named Entity Recognition for African Languages

   David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, and 46 more authors

   Transactions of the Association for Computational Linguistics, Jun 2021

   Abs

   We take a step towards addressing the under- representation of the African continent in NLP research by bringing together different stakeholders to create the first large, publicly available, high-quality dataset for named entity recognition (NER) in ten African languages. We detail the characteristics of these languages to help researchers and practitioners better understand the challenges they pose for NER tasks. We analyze our datasets and conduct an extensive empirical evaluation of state- of-the-art methods across both supervised and transfer learning settings. Finally, we release the data, code, and models to inspire future research on African NLP.1

2020

1. If You Build Your Own NER Scorer, Non-replicable Results Will Come

   Constantine Lignos, and Marjan Kamyab

   In Proceedings of the First Workshop on Insights from Negative Results in NLP, Nov 2020

   Abs

   We attempt to replicate a named entity recognition (NER) model implemented in a popular toolkit and discover that a critical barrier to doing so is the inconsistent evaluation of improper label sequences. We define these sequences and examine how two scorers differ in their handling of them, finding that one approach produces F1 scores approximately 0.5 points higher on the CoNLL 2003 English development and test sets. We propose best practices to increase the replicability of NER evaluations by increasing transparency regarding the handling of improper label sequences.

2. Effective Architectures for Low Resource Multilingual Named Entity Transliteration

   Molly Moran, and Constantine Lignos

   In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, Dec 2020

   Abs

   In this paper, we evaluate LSTM, biLSTM, GRU, and Transformer architectures for the task of name transliteration in a many-to-one multilingual paradigm, transliterating from 590 languages to English. We experiment with different encoder-decoder combinations and evaluate them using accuracy, character error rate, and an F-measure based on longest continuous subsequences. We find that using a Transformer for the encoder and decoder performs best, improving accuracy by over 4 points compared to previous work. We explore whether manipulating the source text by adding macrolanguage flag tokens or pre-romanizing source strings can improve performance and find that neither manipulation has a positive effect. Finally, we analyze performance differences between the LSTM and Transformer encoders when using a Transformer decoder and find that the Transformer encoder is better able to handle insertions and substitutions when transliterating.

2019

1. The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval

   Constantine Lignos, Daniel Cohen, Yen-Chieh Lien, Pratik Mehta, W. Bruce Croft, and Scott Miller

   In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Nov 2019

   Abs

   When performing cross-language information retrieval (CLIR) for lower-resourced languages, a common approach is to retrieve over the output of machine translation (MT). However, there is no established guidance on how to optimize the resulting MT-IR system. In this paper, we examine the relationship between the performance of MT systems and both neural and term frequency-based IR models to identify how CLIR performance can be best predicted from MT quality. We explore performance at varying amounts of MT training data, byte pair encoding (BPE) merge operations, and across two IR collections and retrieval models. We find that the choice of IR collection can substantially affect the predictive power of MT tuning decisions and evaluation, potentially introducing dissociations between MT-only and overall CLIR performance.

2. SARAL: A Low-Resource Cross-Lingual Domain-Focused Information Retrieval System for Effective Rapid Document Triage

   Elizabeth Boschee, Joel Barry, Jayadev Billa, Marjorie Freedman, Thamme Gowda, Constantine Lignos, Chester Palen-Michel, Michael Pust, Banriskhem Kayang Khonglah, Srikanth Madikeri, Jonathan May, and Scott Miller

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Jul 2019

   Abs

   With the increasing democratization of electronic media, vast information resources are available in less-frequently-taught languages such as Swahili or Somali. That information, which may be crucially important and not available elsewhere, can be difficult for monolingual English speakers to effectively access. In this paper we present an end-to-end cross-lingual information retrieval (CLIR) and summarization system for low-resource languages that 1) enables English speakers to search foreign language repositories of text and audio using English queries, 2) summarizes the retrieved documents in English with respect to a particular information need, and 3) provides complete transcriptions and translations as needed. The SARAL system achieved the top end-to-end performance in the most recent IARPA MATERIAL CLIR+summarization evaluations. Our demonstration system provides end-to-end open query retrieval and summarization capability, and presents the original source text or audio, speech transcription, and machine translation, for two low resource languages.

2018

1. Combining rule-based and statistical mechanisms for low-resource named entity recognition

   Ryan Gabbard, Jay DeYoung, Constantine Lignos, Marjorie Freedman, and Ralph Weischedel

   Machine Translation, Jul 2018

   Publisher Site

2017

1. Morphology and language acquisition

   Constantine Lignos, and Charles Yang

   In Cambridge Handbook of Morphology, Jul 2017

   Publisher Site

2015

1. Provably correct reactive control from natural language

   Autonomous Robots, Jul 2015

   Publisher Site

2014

1. Spectro-temporal correlates of lexical access during auditory lexical decision

   Jonathan Brennan, Constantine Lignos, David Embick, and Timothy P.L. Roberts

   Brain and Language, Jul 2014

   Publisher Site
2. Revisiting frequency and storage in morphological processing

   Constantine Lignos, and Kyle Gorman

   In Proceedings of the 48th Annual Meeting of the Chicago Linguistic Society, Jul 2014

   This article was submitted in November 2012, but proceedings were not officially published until 2014.

2013

1. You Can’t Get There from Here: On Interpreting Learning Experiments

   Constantine Lignos

   University of Pennsylvania Working Papers in Linguistics, Jul 2013

   Abs

   Artificial language learning experiments provide a unique opportunity to observe learning under controlled conditions. We cannot, however, observe what learning strategy participants use; we can only carefully design the language and observe the response. This poses an inference problem that I name "the poverty of the experiment." I use computational learning models to address this inference problem, using data from an artificial grammar learning study (Saffran 2001) in which the authors conclude that participants learned hierarchical structure from distributional cues. Simulations show that that learning hierarchical structure is not required to pass the tests administered in those experiments and that a heuristic learner is the best fit for the observed human performance. Artificial language learning experiments cannot in themselves provide evidence for a particular learning strategy; they must be paired with appropriate modeling work to confirm that an implementation of a proposed learning strategy actually produces the expected results.

2. Sorry Dave, I’m Afraid I Can’t Do That: Explaining Unachievable Robot Tasks Using Natural Language

   Vasumathi Raman, Constantine Lignos, Cameron Finucane, Kenton CT Lee, Mitchell P Marcus, and Hadas Kress-Gazit

   In Robotics: Science and Systems IX, Jul 2013

2012

1. Infant word segmentation: An incremental, integrated model

   Constantine Lignos

   In Proceedings of the West Coast Conference on Formal Linguistics 30, Jul 2012
2. Make it so: Continuous, flexible natural language interaction with an autonomous robot

   Daniel J Brooks, Constantine Lignos, Cameron Finucane, Mikhail S Medvedev, Ian Perera, Vasumathi Raman, Hadas Kress-Gazit, Mitch Marcus, and Holly A Yanco

   In Proceedings of the Grounding Language for Physical Systems Workshop at the Twenty-Sixth AAAI Conference on Artificial Intelligence, Jul 2012

2011

1. Modeling Infant Word Segmentation

   Constantine Lignos

   In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, Jun 2011

2010

1. Recession Segmentation: Simpler Online Word Segmentation Using Limited Resources

   Constantine Lignos, and Charles Yang

   In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, Jul 2010
2. A Rule-Based Acquisition Model Adapted for Morphological Analysis

   Constantine Lignos, Erwin Chan, Mitchell Marcus, and Charles Yang

   In Multilingual Information Access Evaluation I. Text Retrieval Experiments, Jul 2010
3. Investigating the Relationship Between Linguistic Representation and Computation through an Unsupervised Model of Human Morphology Learning

   Erwin Chan, and Constantine Lignos

   Research on Language and Computation, Jul 2010
4. Learning from Unseen Data

   Constantine Lignos

   In Proceedings of the Morpho Challenge 2010 Workshop, Sep 2010
5. Evidence for a Morphological Acquisition Model from Development Data

   Constantine Lignos, Erwin Chan, Charles Yang, and Mitchell P. Marcus

   In Proceedings of the 34th Annual Boston University Conference on Language Development, Sep 2010

2009

1. A Rule-Based Unsupervised Morphology Learning Framework

   Constantine Lignos, Erwin Chan, Mitchell P. Marcus, and Charles Yang

   In Working Notes of the 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Sep 2009

2006

1. Effects of head movement on perceptions of humanoid robot behavior

   Emily Wang, Constantine Lignos, Ashish Vatsal, and Brian Scassellati

   In HRI ’06: Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction, Salt Lake City, Utah, USA, Sep 2006