[PDF] On the impressive performance of randomly weighted encoders in

View - Download On the impressive performance of randomly weighted encoders in

Previous PDF

Next PDF

On the impressive performance of

randomly weighted encoders in summarization tasks

Jonathan Pilault

1;2;3;, Jaehong Park1;, Christopher Pal1;2;3;4

1Element AI,2Montreal Institute for Learning Algorithms,

3Ecole Polytechnique de Montreal,4Canada CIFAR AI Chair

1fjonathan.pilault, jaehong.parkg@elementai.com

Abstract

In this work, we investigate the performance

ofuntrained randomly initializedencoders in a general class of sequence to sequence mod- els and compare their performance with that of fully-trainedencoders on the task of abstrac- tive summarization. We hypothesize that ran- dom projections of an input text have enough representational power to encode the hierar- chical structure of sentences and semantics of documents. Using a trained decoder to pro- duce abstractive text summaries, we empiri- cally demonstrate that architectures with un- trained randomly initialized encoders perform competitively with respect to the equivalent ar- chitectures with fully-trained encoders. We further find that the capacity of the encoder not only improves overall model generaliza- tion but also closes the performance gap be- tween untrained randomly initialized and full- trained encoders. To our knowledge, it is the first time that general sequence to sequence models with attention are assessed for trained and randomly projected representations on ab- stractive summarization.

1 Introduction

Recent state-of-the-art Natural Language Process-

ing (NLP) models operate directly on raw in- put text, thus sidestepping typical prepossessing steps in classical NLP that use hand-crafted fea- tures (

Young et al.

2018
). It is typically assumed that such engineered features are not needed since critical parts of language are modeled directly by encoded word and sentence representations in

Deep Neural Networks (DNN). For instance, re-

searchers have attempted to evaluate the ability of recurrent neural networks (RNN) to represent lexi- cal, structural or compositional semantics (

Linzen

et al. 2016

Hupk eset al.

2017

Lak eand Ba-

roni 2017
), and study morphological learning in Equal contribution, order determined by coin flipmachine translation (Belinkov et al.,2017 ;Dalvi et al. 2017
). Various diagnostic methods have been proposed to analyze the linguistic properties that a fixed length vector can hold (

Ettinger et al.

2016

Adi et al.

2016

Kiela et al.

2017

Nevertheless, relatively little is still known

about the exact properties that can be learned and encoded in sentence or document represen- tations from training. While general linguistic structures has been shown to be important in NLP

Strubell and McCallum

2018
), knowing whether this information comes from the architectural bias or the trained weights can be meaningful in de- signing better performing models. Recently, it was demonstrated that randomly parameterized combinations of pre-trained word embeddings of- ten have comparable performance to fully-trained sentence embeddings (

Wieting and Kiela

2019

Such experiments question the gains of trained

modern sentence embeddings over random meth- ods. By showing that random encoders perform close to state-of-the-art sentence embeddings, W i- eting and Kiela 2019
) challenged the assumption that sentence embeddings are greatly improved from training an encoder.

As a follow-up to

W ietingand Kiela

2019
we generalize their approaches to more complex sequence-to-sequence (seq2seq) learning, particu- larly on abstractive text summarization. We inves- tigate various aspects of random encoders using a

Hierarchical Recurrent Encoder Decoder (HRED)

architecture that either has (1) an untrained, ran- domly initialized encoders or (2) a fully trained encoders. In this work, we seek to answer three main questions: (i) How effective are untrained randomly initialized hierarchical RNNs in captur- ing document structure and semantics? (ii) Are untrained encoders close in performance to trained encoders on a challenging task such as long-text summarization tasks? (iii) How does the capacityarXiv:2002.09084v1 [cs.CL] 21 Feb 2020 of encoder or decoder affect the quality of gener- ated summaries for both trained and untrained en- coders? Toanswersuchquestions, weanalyseper- plexity and ROUGE scores of random HRED and fully trained HREDs of various hidden sizes. We go beyond the NLP classification tasks on which random embeddings were shown to be useful ( Wi- eting and Kiela 2019
) by testing its efficacy on a conditional language generation task.

Our main contribution is to present empirical

evidence that using random projections to repre- sent text hierarchy can achieve results on par with fully trained representations. Even without power- fulpretrainedwordembeddings, weshowthatran- dom hierarchical representations of an input text perform similarly to trained hierarchical represen- tations. We also empirically demonstrate that, for general seq2seq models with attention, the gap between random encoder and trained encoder be- comes smaller with increasing size of representa- tions. We finally provide an evidence to validate that optimization and training of our networks was done properly. To the best of our knowledge, it is the first time that such analysis has been per- formed on a general class of seq2seq with atten- tion and for the challenging task of long text sum- marization.

2 Related Work

2.1 Fixed random weights in neural networks

A random neural network (

Minsky and Self-

ridge 1961
) can be defined as a neural network whose weights are initialized randomly or pseudo- randomly and are not trained or optimized for a particular task. Random neural networks have been studied since training and optimization pro- cedures were often infeasible with the computa- tional resources at the time. It was shown that, for low dimensional problems, Feed Forward Neural

Networks (FFNN) with fixed random weights can

achieve comparable accuracy and smaller stan- dard deviations compared to the same network trained with gradient backpropagation (

Schmidt

et al. 1992
). Inspired by this work, Extreme

Learning Machines (ELM) have been proposed

Huang G.-B. and C.-K.

2004
). ELM is a sin- gle layer FFNN where only the output weights are learned through simple generalized inverse oper- ations of the hidden layer output matrices. Sub- sequent theoretical studies have demonstrated that even with randomly generated hidden weights,ELM maintains the universal approximation capa- bility of the equivalent fully trained FFNN ( Huang et al. 2006
). Such works explored the effects of randomness in vision tasks with stationary mod- els. In our work, we explore randomness in NLP tasks with autoregressive models.

Similar ideas have been developed for RNNs

with Echo State Networks (ESN) (

Jaeger

2001
and more generally Reservoir Computing (RC)

Krylov and Krylov

2018
). At RC"s core, the dynamics of an input sequence are modeled by a large reservoir with random, untrained weights whose state is mapped to an output space by a trainable readout layer. ESN leverage the Marko- vian architectural bias of RNNs and are thus able to encode input history using recurrent random projections. ESN has comparable generalization to ELM and is generally known to be more robust for non-linear time series prediction problems ( Li et al. 2011
). In such research, randomness was used in autoregressive models but not within the context of encoder-decoder architectures in NLP.

2.2 Random encoders in deep architectures

various types of input data. In computer vision, it was shown that random kernels in Convolutional

Neural Networks (CNN) perform reasonably well

on object recognition tasks (

Jarrett et al.

2009

Other works highlighted the importance of setting

random weights in CNN and found that perfor- mance of a network could be explained mostly by the choice of a CNN architecture, instead of its op- timized weights (

Saxe et al.

2011
quotesdbs_dbs17.pdfusesText_23

[PDF] idesign data solutions private limited

[PDF] idfc bank forex rates

[PDF] idioms and phrases with meaning and sentences pdf

[PDF] idioms speech therapy goals

[PDF] idubbbz twitter account

[PDF] ied paris 8

[PDF] ieee 1588

[PDF] ieee abstract format

[PDF] ieee abstract size

[PDF] ieee citation

[PDF] ieee citation example

[PDF] ieee conference paper structure

[PDF] ieee font download

[PDF] ieee heading font

[PDF] ieee paper format