[PDF] Iterative Recursive Attention Model for Interpretable Sequence

View - Download Iterative Recursive Attention Model for Interpretable Sequence

Previous PDF

Next PDF

255(a) Simple unipolar sentence

(b) Sentence with a negation (c) Contrastive multipolar sentence Figure 6: Visualization of attention across sentence words (horizontal) andT=3 time steps (vertical).

The lastT-1 columns contain the attention weights

over the result of the previous attentive query.

4.5 Visualizing Attention

To gain an intuition about the working of IRAM,

we visually analyzed its attention mechanism on a number of sentences from our dataset. We limit ourselves to examples from the test set of the SST dataset as the length of examples is manageable for visualization. We isolate three specific cases where the attention mechanism demonstrates interesting results: (1)simpleunipolarsentences, (2)sentences with negations, and (3) multipolar sentences. The least interesting case is the unipolar, as the attention mechanism often does not need multiple iterations. Fig. 6a sho wsthe attention mechanism simply propagating information, since sentiment classification is straightforward and does not re- quire multiple attention steps. This can be seen from most of the attention weight in the second and third steps being on the columns corresponding to the summaries.

The more interesting cases are sentences involv-

ing negations and modifiers. Fig. 6b sho wsthe handling of negation: attention is initially on all words except on the negator. In the second step, the mechanism combines the output of the first step with the negation. We interpret this as flipping the sentiment - the model cannot rely solely on recog- nizing a negative word, and has to account for what that word negates through a functional dependence.

These examples highlight one of the drawbacks of

recurrent networks which we aim to alleviate. In case a standard attention mechanism is applied to a sentence containing a negator, the hidden repre- sentation of the negator has to scale or negate the intensity of an expression. Our model has the ca- pacity to process such sequences iteratively, first constructing the representation of an expression, which is then adjusted by the nonlinear transforma- tion and simpler to combine with the negator in the next step.

Lastly, Fig.

6c sho wsa contrasti vemultipolar sentence, where the model in the first step focuses on positive words, and then combines the negative words (tortured,unsettling) with the results of the first step. In such cases, the model succeeds to isolate the contrasting aspects of the sentence and attends to them in different iterations of the model, alleviating the burden of simultaneously represent- ing the positive and negative aspects. After both contrastive representations have been formed, the model has the capacity toweighthem one against other and compute the final representation.

5 Conclusion

The proposed iterative recursive attention model

(IRAM) has the capacity to construct representa- tions of the input sequence in a recursive fashion, making inference more interpretable. We demon- strated that the model can learn to focus on various task-relevant parts of the input, and can propagate the information in a meaningful way to handle the more difficult cases. On the sentiment analysis task, the model performs comparable to the state of the art. Our next goals will be to try to use the iterative attention mechanism to extract tree-like sentence structures akin to constituency parse trees, evalu- ate the model on more complex datasets as well as extend the model to support an adaptive number of iterative steps.

Acknowledgment

This research has been supported by the Euro-

pean Regional Development Fund under the grant

KK.01.1.1.01.0009 (DATACROSS).

256References

Leila Arras, Gr

´egoire Montavon, Klaus-Robert M¨uller,

and Wojciech Samek. 2017. Explaining recurrent neural network predictions in sentiment analysis. In

Proceedings of the 8th Workshop on Computational

Approaches to Subjectivity, Sentiment and Social

Media Analysis, pages 159-168.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2014. Neural machine translation by jointly learning to align and translate.arXiv preprint arXiv:1409.0473. Yoshua Bengio, Patrice Simard, and Paolo Frasconi.

1994. Learning long-term dependencies with gradi-

ent descent is difficult.IEEE transactions on neural networks, 5(2):157-166.

Kyunghyun Cho, Bart van Merri

¨enboer, Dzmitry Bah-

danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches.Syntax, Semantics and Structure in Statis- tical Translation, page 103.

Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang,

Ting Liu, and Guoping Hu. 2017. Attention-over-

attention neural networks for reading comprehen- sion. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), volume 1, pages 593-602.

Adji B Dieng, Chong Wang, Jianfeng Gao, and John

Paisley. 2016. TopicRNN: A recurrent neural net-

work with long-range semantic dependency. In

ICLR 2016.

Ian J Goodfellow, David Warde-Farley, Mehdi Mirza,

Aaron Courville, and Yoshua Bengio. 2013. Maxout

networks.arXiv preprint arXiv:1302.4389.

Jiatao Gu, James Bradbury, Caiming Xiong, Vic-

tor OK Li, and Richard Socher. 2017. Non- autoregressive neural machine translation.arXiv preprint arXiv:1711.02281.

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsu-

ruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks.arXiv preprint arXiv:1611.01587. Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. InAdvances in Neural Informa- tion Processing Systems, pages 1693-1701.

Sepp Hochreiter and J

¨urgen Schmidhuber. 1997.

Long short-term memory.Neural computation,

9(8):1735-1780.

Rie Johnson and Tong Zhang. 2016. Supervised and

gion embeddings. InInternational Conference on Machine Learning, pages 526-534.Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and

Jan Kleindienst. 2016. Text understanding with

the attention sum reader network.arXiv preprint arXiv:1603.01547.

Diederik P Kingma and Jimmy Ba. 2014. Adam: A

method for stochastic optimization.arXiv preprint arXiv:1412.6980. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky.

2016. Visualizing and understanding neural models

in nlp. InProceedings of NAACL-HLT, pages 681- 691.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos San- tos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding.arXiv preprint arXiv:1703.03130. Thang Luong, Hieu Pham, and Christopher D Manning.

2015. Effective approaches to attention-based neu-

ral machine translation. InProceedings of the 2015

Conference on Empirical Methods in Natural Lan-

guage Processing, pages 1412-1421.

Andrew L Maas, Raymond E Daly, Peter T Pham, Dan

Huang, Andrew Y Ng, and Christopher Potts. 2011.

Learning word vectors for sentiment analysis. In

Proceedings of the 49th annual meeting of the as-

sociation for computational linguistics: Human lan- guage technologies-volume 1, pages 142-150. Asso- ciation for Computational Linguistics.

Bryan McCann, James Bradbury, Caiming Xiong, and

Richard Socher. 2017. Learned in translation: Con- textualized word vectors. InAdvances in Neural In- formation Processing Systems, pages 6297-6308.

Takeru Miyato, Andrew M Dai, and Ian Good-

fellow. 2016. Adversarial training methods for semi-supervised text classification.arXiv preprint arXiv:1605.07725.

Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017.

All-but-the-top: simple and effective postprocess- ing for word representations.arXiv preprint arXiv:1702.01417.

Tsendsuren Munkhdalai and Hong Yu. 2017. Neu-

ral semantic encoders. InProceedings of the con- ference. Association for Computational Linguistics.

Meeting, volume 1, page 397. NIH Public Access.

Ankur P Parikh, Oscar T

¨ackstr¨om, Dipanjan Das, and

Jakob Uszkoreit. 2016. A decomposable attention

arXiv:1606.01933. Jeffrey Pennington, Richard Socher, and Christopher

Manning.2014. GloVe: Globalvectorsforwordrep-

resentation. InProceedings of the 2014 conference on empirical methods in natural language process- ing (EMNLP), pages 1532-1543.

257Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt

Gardner, Christopher Clark, Kenton Lee, and Luke

Zettlemoyer. 2018. Deep contextualized word repre- sentations.arXiv preprint arXiv:1802.05365. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever.

2017. Learning to generate reviews and discovering

sentiment.arXiv preprint arXiv:1704.01444. Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2018.

On the convergence of Adam and beyond. InInter-

national Conference on Learning Representations.

Tim Rockt

¨aschel, Edward Grefenstette, Karl Moritz

Hermann, Tom

´as Kocisk`y, and Phil Blunsom. 2015.

Reasoning about entailment with neural attention.

arXiv preprint arXiv:1509.06664.

Alexander M Rush, Sumit Chopra, and Jason Weston.

2015. A neural attention model for abstractive sen-

tence summarization. InProceedings of the 2015

Conference on Empirical Methods in Natural Lan-

guage Processing, pages 379-389. Abigail See, Peter J Liu, and Christopher D Manning.

2017. Get to the point: Summarization with pointer-

generator networks. InProceedings of the 55th An- nual Meeting of the Association for Computational

Linguistics (Volume 1: Long Papers), volume 1,

pages 1073-1083.

Richard Socher, Alex Perelygin, Jean Wu, Jason

Chuang, Christopher D Manning, Andrew Ng, and

Christopher Potts. 2013. Recursive deep models

for semantic compositionality over a sentiment tree- bank. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642. Alessandro Sordoni, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2016. Iterative alternating neu- ral attention for machine reading.arXiv preprint arXiv:1606.02245.

Rupesh Kumar Srivastava, Klaus Greff, and J

¨urgen

Schmidhuber. 2015. Highway networks.arXiv

preprint arXiv:1505.00387.quotesdbs_dbs44.pdfusesText_44

[PDF] fonction itérative factorielle

[PDF] fonction itérative php

[PDF] operation factorielle

[PDF] différence entre algorithme itératif et algorithme récursif

[PDF] expression de couturiere

[PDF] fonction récursive

[PDF] automobile in corsa

[PDF] pélican volant de marey (1882)

[PDF] dynamisme d'un cycliste

[PDF] le futurisme mouvement artistique

[PDF] futurisme caractéristiques

[PDF] futurisme définition

[PDF] l5a les clans majeurs pdf

[PDF] l5a pdf

[PDF] l5a 4eme edition pdf