The compositionality of neural networks has been a hot topic recently, on which no consensus exists. In this paper, we argue that this in part stems from the fact that there is little consensus on what it means for a network to be compositional, and we present a set of five tests that provide a bridge between the linguistic and philosophical knowledge about compositionality and deep neural networks. We use our tests to evaluate a recurrent model, a convolution-based model and a Transformer, and provide an in depth analysis of the results.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni
While neural network models have been successfully applied to domains that require substantial generalisation skills, recent studies have implied that they struggle when solving the task they are trained on requires inferring its underlying compositional structure. In this paper, we introduce Attentive Guidance, a mechanism to direct a sequence to sequence model equipped with attention to find more compositional solutions.
Dieuwke Hupkes, Anand Singh, Kris Korrel, German Kruszewski and Elia Bruni
Compositionality decomposed: how do neural networks generalise?
Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni • Journal of Artificial Intelligence Research
Location Attention for Extrapolation to Longer Sequences
Yann Dubois, Gautier Dagan, Dieuwke Hupkes, Elia Bruni • ACL 2020
Internal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency.
Diana Rodríguez Luna, Edoardo Maria Ponti, Dieuwke Hupkes and Elia Bruni •
Analysing neural language models: contextual decomposition reveals default reasoning in number and gender assignment
Jaap Jumelet, Willem Zuidema and Dieuwke Hupkes • CONLL 2019
Mastering emergent language: learning to guide in simulated navigation
Mathijs Mul, Diane Bouchacourt and Elia Bruni •
Assessing incrementality in sequence-to-sequence models
Dennis Ulmer, Dieuwke Hupkes and Eia Bruni • Repl4NLP, ACL 2019
Transcoding compositionally: using attention to find more generalizable solutions
Kris Korrel, Dieuwke Hupkes, Verna Dankers and Elia Bruni • BlackboxNLP, ACL 2019
On the Realization of Compositionality in Neural Networks
Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes and Elia Bruni • BlackboxNLP, ACL 2019
The Fast and the Flexible: training neural networks to learn to follow instructions from small data
Rezka Leonandya, Elia Bruni, Dieuwke Hupkes and German Kruszewski • IWCS 2019
Learning compositionally through attentive guidance
Dieuwke Hupkes, Anand Singh, Kris Korrel, German Kruszewski and Elia Bruni • CICLing 2019
Do language models understand anything? On the ability of LSTMs to understand negative polarity items
Jaap Jumelet and Dieuwke Hupkes • BlackboxNLP, EMNLP 2018