« Older Entries

MCQLL Meeting, 11/6 — Yikang Shen and Zhouhan Lin

This week at MCQLL, Yikang Shen and Zhouhan Lin will present on previous and ongoing work.

Title: Ordered Neurons and syntactically supervised structural neural language models.

Abstract: In this talk, we will first present the ONLSTM model, which is a language model that learns syntactic distances through an extra master input gate and a master forget gate. In the second part of the talk, we will present a way of incorporating supervised syntactic trees to the neural language model, also through the syntactic distances. Experimental results reveal that for neural models this way of injecting supervised tree structure helps the language model to yield better results.

The meeting is Wednesday from 14:30-16:00 in Room 117. A late lunch will be provided.

MCQLL Meeting, 10/30 — Dima Bahdanau

This week at MCQLL, Dima Bahdanau presents recent work.

Title: CLOSURE: Assessing Systematic Generalization of CLEVR Models

Abstract: The CLEVR dataset of natural-looking questions about 3D-rendered scenes has recently received much attention from the research community. A number of models have been proposed for this task, many of which achieved very high accuracies of around 97-99%. In this work, we study how systematic the generalization of such models is, that is to which extent they are capable of handling novel combinations of known linguistic constructs. To this end, we define 7 additional question families which test models’ understanding of similarity-based references (such as e.g. “the object that has the same size as …”) in novel contexts. Our experiments on the thereby constructed CLOSURE benchmark show that state-of-the-art models often do not exhibit systematicity after being trained on CLEVR. Surprisingly, we find that the explicitly compositional Neural Module Network model also generalizes badly on CLOSURE, even when it has access to the ground-truth programs at test time. We improve the NMN’s systematic generalization by developing a novel Vector-NMN module architecture with vector-shaped inputs and outputs. Lastly, we investigate the extent to which few-shot transfer learning can help models that are pretrained on CLEVR to adapt to CLOSURE. Our few-shot learning experiment contrast the adaptation behavior of the models with intermediate discrete programs with that of the end-to-end continuous models.

The meeting is Wednesday from 14:30-16:00 in Room 117.

MCQLL Meeting, 10/23 – Kushal Arora

At the meeting of MCQLL this week, Kushal Arora will present his recent work with Aishik Chakraborty.

Title: Learning Lexical Subspaces in a Distributional Vector Space

Abstract: In this paper, we propose LexSub, a novel approach towards unifying lexical and distributional semantics. We inject knowledge about lexical-semantic relations into distributional word embeddings by defining subspaces of the distributional vector space in which a lexical relation should hold. Our framework can handle symmetric attract and repel relations (e.g., synonymy and antonymy, respectively), as well as asymmetric relations (e.g., hypernymy and meronomy). In a suite of intrinsic benchmarks, we show that our model outperforms previous post-hoc approaches on relatedness tasks, and on hypernymy classification and detection while being competitive on word similarity tasks. It also outperforms previous systems on extrinsic classification tasks that benefit from exploiting lexical relational cues. We perform a series of analyses to understand the behaviors of our model.

This meeting will be held in room 117 at 14:30 on Wednesday.

MCQLL Meeting, 10/16 — Michaela Socolof

This week at MCQLL, Michaela Socolof will lead a discussion on the paper “A noisy-channel model of rational human sentence comprehension under uncertain input” by Roger Levy (abstract below). The setup will be somewhat different from the typical MCQLL presentations. Michaela will have some discussion topics prepared for the group, please give the paper a read before the meeting to make the discussion more dynamic!

As usual the meeting is from 14:30-16:00 on Wednesday. A late lunch will be provided.

Abstract: Language comprehension, as with all other cases of the extraction of meaningful structure from perceptual input, takes places under noisy conditions. If human language comprehension is a rational process in the sense of making use of all available information sources, then we might expect uncertainty at the level of word-level input to affect sentence-level comprehension. However, nearly all contemporary models of sentence comprehension assume clean input—that is, that the input to the sentence-level comprehension mechanism is a perfectly-formed, completely certain sequence of input tokens (words). This article presents a simple model of rational human sentence comprehension under noisy input, and uses the model to investigate some outstanding problems in the psycholinguistic literature for theories of rational human sentence comprehension. We argue that by explicitly accounting for input level noise in sentence processing, our model provides solutions for these outstanding problems and broadens the scope of theories of human sentence comprehension as rational probabilistic inference.

Link to the paper: https://www.mit.edu/~rplevy/papers/levy-2008-emnlp.pdf

MCQLL, 10/9 — Siva Reddy

At the meeting of MCQLL this week, Siva Reddy will discuss ongoing work on Measuring Stereotypical Bias in Pretrained Neural Network Models of Language.

A key ingredient behind the success of neural network models for language is pretrained representations: word embeddings, contextual embeddings and pretrained architecutures. Since pretrained representations are obtained from learning on massive text corpora, there is a danger that unwanted societal biases are reflected in these models. I will discuss ideas on how to assess these biases in popular pretrained language models.

This meeting will be in room 117 at 14:30 on Wednesday, October 9th.

MCQLL Meeting, 10/02 — Emi Baylor

At Next week’s MCQLL meeting, Emi will be presenting about a research project investigating morphological productivity.

Emi will discuss morphological productivity by presenting on German plural nouns, and what makes them uniquely suited to use in the testing of theories of productivity.

The meeting will be in room 117 at 14:30 on Wednesday.

MCQLL Lab Meeting, 9/25 — Jacob Hoover

At next week’s MCQLL meeting, Jacob will be presenting about a new research project on finding syntactic structures in contextual embeddings.

A recent paper by Hewitt and Manning shows that the pretrained embeddings of contextual neural networks (ELMo, BERT) encode information about dependency structure (more concretely: a learned linear transformation (“probe”) on top of pre-trained embeddings is able to reconstruct dependencies based on the Penn Treebank). Several questions arise when considering this result: Does BERT have a theory of syntax? What is does this mean? What structure/information is this probe extracting from the embeddings? Jacob will introduce the results from this paper for a general audience, and discuss some of these questions as current ideas for research.

The meeting will be in room 117 at 14:30 on Wednesday.

MCQLL, 9/11 — Tim O’Donnell

At next week’s meeting of the Montreal Computational and Quantitative Linguistics Lab, Tim will present recent work with Shijie Wu and Ryan Cotterell on measuring irregularity in morphological systems.

We meet Wednesdays at 14:30 in Room 117.

MCQLL, 4/10 – Wilfred Yau

At next week’s meeting, Wilfred will be presenting the following paper: Mao, L., & Hulden, M. (2016). How regular is Japanese loanword adaptation? A computational study. Please find the abstract below:
Abstract: The modifications that foreign loanwords undergo when adapted into Japanese have been the subject of much study in linguistics. The scholarly interest of the topic can be attributed to the fact that Japanese loanwords undergo a complex series of phonological adaptations, something which has been puzzling scholars for decades. While previous studies of Japanese loanword accommodation have focused on specific phonological phenomena of limited scope, the current study leverages computational methods to provide a more complete description of all the sound changes that occur when adopting English words into Japanese. To investigate this, we have de- veloped a parallel corpus of 250 English transcriptions and their respective Japanese equivalents. These words were then used to develop a wide-coverage finite state transducer based phonolog- ical grammar that mimics the behavior of the Japanese adaptation process, mapping e.g cream[kôi:m] to [kW.Ri:.mW]. By developing rules with the goal of accounting completely for a large number of borrowings, and analyzing forms mistakenly generated by the system, we discover an internal inconsistency within the loanword phonology of the Japanese language, something arguably underestimated by previous studies. The result of the investigation suggests that there are multiple dimensions that shape the output form of the current Japanese loanwords. These dimensions include orthography, phonetics, and historical changes. (link to paper: https://www.aclweb.org/anthology/C16-1081?fbclid=IwAR3NYKaphmkENltohIpdTJhZLZFe1Fzbtez90-P_FaIWHIkK5imlN7Qyh4I)
We will meet 5:30pm Wednesday in room 117. Food will be provided.

MCQLL, 03/27

At next week’s meeting, James will be presenting some preliminary work with Morgan Sonderegger and Jane Stuart-Smith on dialectal & speaker variability in the consonant voicing effect.

The meeting will start 5:30pm Wednesday 3/27 in room 117. Food will be provided.

MCQLL, 2/27

At next week’s MCQLL meeting, Graham will give a presentation on the following paper:
Abstract:
How do children begin to use language to say things they have never heard before? The origins of linguistic productivity have been a subject of heated debate: Whereas generativist accounts posit that children’s early language reflects the presence of syntactic abstractions, constructivist approaches instead emphasize gradual generalization derived from frequently heard forms. In the present research, we developed a Bayesian statistical model that measures the degree of abstraction implicit in children’s early use of the determiners “a” and “the.” Our work revealed that many previously used corpora are too small to allow researchers to judge between these theoretical positions. However, several data sets, including the Speechome corpus—a new ultra-dense data set for one child—showed evidence of low initial levels of productivity and higher levels later in development. These findings are consistent with the hypothesis that children lack rich grammatical knowledge at the outset of language learning but rapidly begin to generalize on the basis of structural regularities in their input.
We will be meeting Wednesday at 5:30pm in room 117.

MCQLL, 2/20 – Seara Chen

At next week’s MCQLL meeting, Seara Chen will be presenting on computational model for phonotactics. She will present a survey of two of the major types of computational models for phonotactics, which are based on a collection of papers. In addition, she will also give a short explanation of the current experiment they are working in the lab that will be used to compare different automated phonotactic scorer.

The meeting will be Wednesday 2/20 5:30pm in room 117. All are welcome!

MCQLL, 2/13 – Vanna Willerton

At next week’s MCQLL meeting, Vanna will be presenting two short papers on the topic of language acquisition. Both papers use statistical methods to deduce interesting information regarding the role of data in early language learning:

  1. How Data Drive Early Word Learning: A Cross-Linguistic Waiting Time Analysis. Mollica & Piantadosi (2017)
  2. Humans store ~1.5 megabytes during language acquisition: information theoretic boundsMollica & Piantadosi (?)

It is not required that you all read them, but they are quite short so you are welcome to read ahead of time to make a more interesting discussion. Please click the titles for papers.

We will be meeting Wednesday 5:30pm in room 117.

MCQLL, 2/6 – Jacob Hoover

At next week’s MCQLL lab meeting, Jacob will present on Non-projectivity and mild–context sensitivity. He will be presenting on Marco Kuhlmann’s 2010 book “Dependency Structures and Lexicalized Grammars”.  Word-to-word dependencies have a history in descriptive linstuistics, based on the intuition that the structure of a sentence can be captured by the relationships between the words.  Dependency structures can be sorted into different classes depending on the amount and form of crossing dependencies that are allowed.  Examining classes of non-projective dependency structures and how they relate to grammar formalisms (starting with projective dependency structures = lexicalized context-free grammars), as well as dependency corpora is a way to investigate what kind of limited context-sensitivity should be used to best deal with the long distance dependencies and free word order in natural languages.

We will meet Wednesday 2/6 5:30pm in room 117.

 

MCQLL, 1/30 – Amy Bruno

Next Wednesday, Amy will present her project on Empirical Learnability and Inference with Minimalist Grammars, which is the second part of the presentation that she did at the end of last semester.

Abstract: This is a draft presentation of some of my current PhD research, intended for a more computationally-oriented audience. It contains collaborative work done over the past year with Eva Portelance (Stanford), Daniel Harasim (EPFL), and Leon Bergen (UCSD). Minimalist Grammars are a lexicalied grammar formalism inspired by Chomsky’s (1994) Minimalist Program, and as such are well suited to formalize theories in contemporary syntactic theory. Our work formulate a learning model based on the technique of Variational Bayesian Inference and apply the model to pilot experiments. In this presentation, I focus on giving an introduction to the central issues in syntactic theory and motivating the problems we wish to address. I give an introduction to syntactic theory and formal grammars, and demonstrate why context free grammars are insufficient to adequately characterize natural language. Minimalist Grammars, a lexicalized mildly context-sensitive formalism are introduced as a more linguistically adequate formalism.
The meeting will be Wednesday 5:30pm in room 117.

MCQLL, 12/5

At next week’s meeting, Amy and Benji will both give a presentation. Amy is going to present her project “Inference and Learnability over Minimalist Grammars” (abstract below). Benji is going to present the paper Parsing as Deduction (Pereira &Warren, 1093) (paper attached).

(Working) Title: Inference and Learnability over Minimalist Grammars

Abstract: This is a draft presentation of some of my current PhD research, intended for a more computationally-oriented audience. It contains collaborative work done over the past year with Eva Portelance (Stanford), Daniel Harasim (EPFL), and Leon Bergen (UCSD). Minimalist Grammars are a lexicalied grammar formalism inspired by Chomsky’s (1994) Minimalist Program, and as such are well suited to formalize theories in contemporary syntactic theory. Our work formulate a learning model based on the technique of Variational Bayesian Inference and apply the model to pilot experiments. In this presentation, I focus on giving an introduction to the central issues in syntactic theory and motivating the problems we wish to address. I give an introduction to syntactic theory and formal grammars, and demonstrate why context free grammars are insufficient to adequately characterize natural language. Minimalist Grammars, a lexicalized mildly context-sensitive formalism are introduced as a more linguistically adequate formalism.

We will meet Wednesday at 5:30pm in room 117. Food will be provided.

MCQLL, 11/28

At next week’s meeting, Yves will be presenting the family of stochastic processes known as Dirichlet processes.

The Dirichlet distribution, a generalization of the Beta distribution, is a probabilistic distribution over a finite-dimensional categorical distribution. The Dirichlet process can be seen as an infinite-dimensional generalization of this which balances the trade-off between partitioning random observations into fewer or additional categories. I will describe this through the metaphor of the “Chinese restaurant process” and talk about its use in the fragment grammar model of morphological productivity.

We will be meeting at 5:30pm Wednesday November 28th in room 117.

MCQLL Meeting Wednesday, 11/21

At this week’s MCQLL meeting, Bing’er Jiang will present Feldman et al.’s (2013) A Role for the Developing Lexicon in Phonetic Category Acquisition. Please find the abstract below:

Infants segment words from fluent speech during the same period when they are learning phonetic categories, yet accounts of phonetic category acquisition typically ignore information about the words in which sounds appear. We use a Bayesian model to illustrate how feedback from segmented words might constrain phonetic category learning by providing information about which sounds occur together in words. Simulations demonstrate that word-level information can successfully disambiguate overlapping English vowel categories. Learning patterns in the model are shown to parallel human behavior from artificial language learning tasks. These findings point to a central role for the developing lexicon in phonetic category acquisition and provide a framework for incorporating top-down constraints into models of category learning.

 

We will be meeting Wednesday November 21 at 5:00pm in room 117. Food will be provided. See you then!

MQLL Meeting, 10/24

At next week’s meeting, Seara will present her project on the inverse relation between size of inflectional classes and word frequency. Here is the abstract:

In this project, we attempt to quantitatively demonstrate the the inverse relation between size of inflectional classes and word frequency. I will go over the background behind productivity in inflections and word frequency, the stages in quantitatively demonstrating the relationship between word frequency and size of inflectional class. Then finally the next step of the project moving forward.

The meeting will be next Wednesday from 5:30pm to 7:30pm at room 117.

MQLL Meeting, 10/17

At next week’s meeting, Wilfred will be presenting the following paper: “Learning Semantic Correspondence with Less Supervision” by Liang et al. (2009). Please find the abstract below:

A central problem in grounded language acquisition is learning the correspondences between a rich world state and a stream of text which references that world state. To deal with the high de- gree of ambiguity present in this setting, we present a generative model that simultaneously segments the text into utterances and maps each utterance to a meaning representation grounded in the world state. We show that our model generalizes across three domains of increasing difficulty—Robocup sportscasting, weather forecasts (a new domain), and NFL recaps.

Meeting will be Wednesday Oct 17 from 5:30pm to 7:30pm at room 117.

« Older Entries
Blog authors are solely responsible for the content of the blogs listed in the directory. Neither the content of these blogs, nor the links to other web sites, are screened, approved, reviewed or endorsed by McGill University. The text and other material on these blogs are the opinion of the specific author and are not statements of advice, opinion, or information of McGill.