[PDF] Detecting Abuse on the Internet: Its Subtle

View - Download Detecting Abuse on the Internet: Its Subtle

Previous PDF

Next PDF

Detecting Abuse on the Internet:

It's Subtle

Sunyam Bagga

Master of Science

School of Computer Science

McGill University

Montreal, Quebec

December, 2019

A thesis submitted to McGill University in partial fulllment of the requirements of the degree of Master of Science c

Sunyam Bagga, 2019

DEDICATION

This thesis is dedicated to my mother, whose love and support I am incredibly grateful for. ii

ACKNOWLEDGEMENTS

I would like to sincerely thank my research advisors, Professor Derek Ruths and Professor Andrew Piper, for their guidance, knowledge and men- torship. I have immensely grown as a researcher in the past two years, and they deserve all the credit for that. I would also like to thank Jack Moncado for some insightful discussions on antagonism and helping out with initial data annotations. Finally, a shout out to the NDL lab members for their constant support and encouragement! iii

ABSTRACT

Abusive language in online discourse negatively aects a large number of social media users. Many computational methods have been proposed by the Natural Language Processing (NLP) community to address this issue of online abuse. The existing studies, however, have been limited in that they focus on detecting the more explicit forms of abuse leaving the subtler forms of abuse largely untouched. This is problematic since subtle abuse can be just as damaging as overt abuse. This thesis makes three core contributions to address this gap in the literature. First, we propose a novel task of detecting a subtler form of abuse, namely unpalatable questions, inspired from the theory of impoliteness. Second, we collect, annotate, and make publicly available a context-rich dataset for the task using data from a diverse set of online communities on Reddit. A notable characteristic of our dataset is that the conversational context was explicitly considered during annotation. Third, we implement a wide array of machine learning and deep learning models using state-of-the-art NLP techniques to detect unpalatable questions. We also investigate the benets of incorporating conversational context into the computational models. Our results show that modeling subtle abuse is feasible but dicult due to the language involved being highly nuanced and context-sensitive. We hope that future research in the eld will also account for subtler forms of abuse since they do their harm passing unnoticed through existing detection systems. Detecting abuse { both overt and subtle { on the Internet would help enhance user's online experience and facilitate civil and productive discussions. iv ABR EGE Le langage abusif utilise sur les plateformes virtuelles a des impacts negatifs sur un grand nombre d'usagers des media sociaux. Plusieurs methodes de cal- cul ont ete proposees par la communaute Natural Language Processing (NLP) pour resoudre la problematique de l'abus virtuel. La recherche sur ce sujet est toutefois limitee et se concentre majoritairement sur la detection de formes explicites d'abus alors que les formes plus subtiles d'abus passent inapercues. Cette problematique est inquietante sachant que les formes d'abus plus subtiles peuvent ^etre tout aussi dommageable que les formes d'abus explicite. Pour combler cette lacune, cette these contribue au domaine de recherche de trois manieres dierentes. Premierement, nous proposons une nouvelle technique inspiree de la theorie linguistique de l'impolitesse an de detecter les formes subtiles d'abus, notamment les questions deplaisantes. Deuxiemement, nous collectons, annotons et rendons public une banque de donnee qui comprend l'information contextuelle associe aux formes d'abus subtiles. Cette base de donnees est construite est a l'aide de donnees provenant d'un ensemble diver- sie de communautes en ligne sur Reddit et se demarque car elle inclut le con- texte de la conversation qui a donne lieu a l'abus. Troisiemement, nous avons implemente plusieurs modeles d'apprentissage automatique et d'apprentissage approfondi a l'aide de techniques NLP de pointe pour detecter les questions deplaisantes. Nousetudionsegalement les avantages qui decoulent de l'integration du contexte de la conversation qui a donne lieu a l'abus dans l'analyse dans des modeles informatiques. Nos resultats demontrent qu'il est possible de modeliser les abus subtils est realisable mais cette t^ache reste incroyablement dicile en raison du langage tres nuance et dependant du contexte qui est utilise. Nous esperons que les recherches futures dans le domaine incluront v egalement les formes d'abus subtiles, car elles causent des dommages qui ne sont pas detectees par les systemes de detection existants. Detecter les abus - a la fois explicites et implicites - sur Internet contribuerait a ameliorer l'experience virtuelle des usagers et faciliterait des discussions saines et civiles. vi

TABLE OF CONTENTS

DEDICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . iii ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv ABR EGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Summary of Contributions . . . . . . . . . . . . . . . . . . 5

1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Related Work and Background Information . . . . . . . . . . . . 7

2.1 Abuse Detection . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 Feature-Based Approaches . . . . . . . . . . . . . . 7

2.1.2 Deep Learning Approaches . . . . . . . . . . . . . . 9

2.1.3 Incorporating Context . . . . . . . . . . . . . . . . . 9

2.2 Impoliteness Theory . . . . . . . . . . . . . . . . . . . . . 12

2.3 What is an Unpalatable Question? . . . . . . . . . . . . . 14

2.4 Rhetorical Question Detection . . . . . . . . . . . . . . . . 17

2.5 Text Classication . . . . . . . . . . . . . . . . . . . . . . 19

2.5.1 Feature Vectors . . . . . . . . . . . . . . . . . . . . 19

2.5.2 Word Embeddings . . . . . . . . . . . . . . . . . . . 20

2.6 Traditional Machine Learning Algorithms . . . . . . . . . . 22

2.6.1 Logistic Regression . . . . . . . . . . . . . . . . . . 22

2.6.2 Support Vector Machines . . . . . . . . . . . . . . . 24

2.7 Deep Learning Algorithms . . . . . . . . . . . . . . . . . . 25

2.7.1 Convolutional Neural Networks . . . . . . . . . . . . 25

2.7.2 Recurrent Neural Networks . . . . . . . . . . . . . . 25

2.8 Recent Advancements in Deep Learning for NLP . . . . . . 28

2.8.1 Contextualized Word Embeddings . . . . . . . . . . 28

2.8.2 Transformers and BERT . . . . . . . . . . . . . . . 29

2.9 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 30

vii

2.9.1 Classication Metrics . . . . . . . . . . . . . . . . . 30

2.9.2 K-fold Cross Validation . . . . . . . . . . . . . . . . 33

2.9.3 Grid Search . . . . . . . . . . . . . . . . . . . . . . 34

3 Data and Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . . 35

3.1 Raw Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.1.1 Data Source . . . . . . . . . . . . . . . . . . . . . . 35

3.1.2 Question Extraction . . . . . . . . . . . . . . . . . . 36

3.2 Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.2.1 Mechanical Turk Web Service . . . . . . . . . . . . 41

3.2.2 Task Design . . . . . . . . . . . . . . . . . . . . . . 43

3.2.3 Annotation Quality Control Measures . . . . . . . . 46

3.3 Annotated Data . . . . . . . . . . . . . . . . . . . . . . . . 47

3.3.1 Data Description . . . . . . . . . . . . . . . . . . . 47

3.3.2 Annotator Agreement . . . . . . . . . . . . . . . . . 49

3.3.3 Comparison with Existing Datasets . . . . . . . . . 50

3.3.4 Perspective API . . . . . . . . . . . . . . . . . . . . 52

4 Computational Modeling . . . . . . . . . . . . . . . . . . . . . . . 56

4.1 Traditional Machine Learning Approach . . . . . . . . . . 56

4.1.1 Feature Categories . . . . . . . . . . . . . . . . . . . 57

4.1.2 Model Training . . . . . . . . . . . . . . . . . . . . 60

4.2 Deep Learning Approach . . . . . . . . . . . . . . . . . . . 61

4.2.1 Model Pipeline . . . . . . . . . . . . . . . . . . . . . 61

4.2.2 Transfer Learning . . . . . . . . . . . . . . . . . . . 66

5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 68

5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

6 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . 79

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 viii

LIST OF TABLESTablepage

2{1 Frequency of dierent subcategories observed in a random

sample ofunpalatablecomments from our dataset. . . . . . . 14

2{2 Examples covering dierent aspects of unpalatable questions

from the annotated dataset. . . . . . . . . . . . . . . . . . . 16

3{1 Confusion matrix for the rule-based regular expression approach. 39

3{2 Confusion matrix for the constituency parsing approach. . . . . 39

3{3 Summary statistics for the published batches on Mechanical Turk. 44

3{4 Distribution of condence scores in the annotated dataset. . . . 48

5{1 Classication results for Logistic Regression on the complete

dataset:condence2 f0:6;0:8;1:0g. . . . . . . . . . . . . . 70

5{2 Classication results for Logistic Regression on high agreement

dataset:condence= 1:0 . . . . . . . . . . . . . . . . . . . . 71

5{3 Classication results for deep learning models on the complete

dataset:condence2 f0:6;0:8;1:0g. . . . . . . . . . . . . . 74

5{4 Classication results for deep learning models on high agreement

dataset:condence= 1:0 . . . . . . . . . . . . . . . . . . . . 75

6{1 Rest of the classication results for Logistic Regression on the

complete dataset:condence2 f0:6;0:8;1:0g. . . . . . . . . 82

6{2 Rest of the classication results for Logistic Regression on

high-agreement dataset:condence= 1:0 . . . . . . . . . . . 83

6{3 Rest of the classication results for deep learning models on the

complete dataset:condence2 f0:6;0:8;1:0g. . . . . . . . . 84

6{4 Rest of the classication results for deep learning models on

high-agreement dataset:condence= 1:0 . . . . . . . . . . . 85 ix

LIST OF FIGURESFigurepage

2{1 This gure illustrates whereunpalatable questionst in a

hypothetical spectrum of online abuse (adapted from [41]). . 15

3{1 Example of a constituent parse tree generated using the Stan-

ford CoreNLP library [54]. . . . . . . . . . . . . . . . . . . . 38

3{2 Instructions for the crowdsourcing task as seen by Mechanical

Turk Workers. . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3{3 Additional examples for the crowdsourcing task as seen by

Mechanical Turk Workers. . . . . . . . . . . . . . . . . . . . 44

3{4 Work

ow design for the crowdsourcing task using Mechanical Turk's API. . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3{5 Distribution of unpalatable questions across dierent subreddits. 48

3{6 Box plot of toxicity scores returned by Perspsective API's

classier when provided with only the question text. . . . . 52

3{7 Box plot of toxicity scores returned by Perspsective API's

classier when provided with the full reply text. . . . . . . . 53

4{1 The skeletal architecture for our deep learning models cor-

responding to the scenario when there is a single input: questiontext or fullreplytext. . . . . . . . . . . . . . . . . 62

4{2 Architecture of a Bi-directional Long Short Term Memory

(LSTM) network. This is one example of aSeq2Vecencoder. Other encoders include LSTM, Stacked Bi-LSTM, and CNN. 63

4{3 Architecture for our deep learning models which include the

dense feature vector constructed using the hand-engineered feature categories. . . . . . . . . . . . . . . . . . . . . . . . 64

4{4 The skeletal architecture for our deep learning models that

incorporate conversational context. This corresponds to the scenario when we have two inputs:preceding commenttext and fullreplytext. . . . . . . . . . . . . . . . . . . . . . . . 65 x

CHAPTER 1

Introduction

1.1 Introduction

Social media sites have become an increasingly popular medium of com- munication in modern society. This is evident from the fact that there are over

2 billion monthly active users on Facebook. Other social curation sites like

Reddit also boast over 330 million active users. With this high rate of user en- gagement, abuse and antisocial behaviour is omnipresent in online discourse. According to a recent survey, 41% of Americans have personally experienced some form of online harassment with 18% having faced severe forms of harass- ment, e.g. physical threats and sexual harassment [72]. To ensure people's safety online and maintain civil discourse, dierent social media platforms implement several mechanisms such as content mod- eration, muting or blocking users from posting etc. Such mechanisms are essential in today's world to discourage and prevent abusive behaviour on- line. Its importance is underscored by a recent white paper published by the UK Government which puts forward a new system of accountability for tech companies [35]. It is, however, infeasible tomanuallymoderate online com- munities due to the sheer enormity of content produced every day { Twitter, for example, receives over 500 million tweets per day. Manual moderation in such a scenario would require humans to read millions of tweets daily which would take an impractical amount of time and other resources. Consequently, there is a growing interest in the elds of Natural Language Processing (NLP) and computational social science to develop algorithmic models that can au- tomatically detect oensive language on social media. 1 Long before detecting online abuse gained attention, there had been sig- nicant research in a related eld of linguistic impoliteness. Impoliteness can be dened as the use of strategies to attack the interlocutor's face { a per- sona that one presents in a conversation [32] { and create social disruption [16]. The most notable contribution in this eld is by Jonathan Culpeper who introduced his theory of impoliteness [16] as a parallel to Brown and Levin- son's theory of politeness [7]. More recently, Culpeper oered conventionalized impoliteness formulae for English language derived from his corpus [17] [18]. The formulae were divided into several categories or forms: insults, threats, dismissals, unpalatable questions, silencers, and others. Abuseis an umbrella term which can cover several ne-grained negative expressions. There exists a plethora of abuse detection studies employing dierent terminology and eectively solving overlapping sub-tasks: personal attacks [93], bullying [20] [12], nastiness [77], harassment [33] [94], toxicity [83], hostility [52], racism or sexism [90], oensive language [62] [21], aggression [9] [12], abusive language [63], hate speech [30] [88] [21] [26] [23], and others. A comparison of these sub-tasks with the impoliteness forms proposed by Culpeper [18] reveals a fundamental gap in the literature { most studies tend to focus on detecting the more explicit forms of abuse leaving the subtler forms of abuse largely untouched. This is concerning since subtle abuse can be just as damaging as explicit abuse for some people [61]. This is also noted in a recent survey by Jurgens et al. [41] calling on the NLP community to rethink and expand what constitutes abuse. There can be a number of such subtle forms of abuse. Here, we focus on detectingunpalatable questions, one of the impoliteness forms proposed by Culpeper [18]. In this thesis, we make three core contributions to address this missing gap in the literature. 2 First, we propose a novel task of detecting unpalatable questions inspired from impoliteness theory. We dene theunpalatable questiontask as detecting a negatively phrased question designed to antagonise its recipient in online discourse (discussed in detail in Section 2.3). There exists several linguistic studies which show that being asked an unpalatable question puts the recipient in a vulnerable position to receive further verbal attacks [6] [92]. In order to promote healthy and civil online discourse, we argue that detecting such covert forms of oensive language is just as important as detecting the more explicit forms. Second, we introduce a context-dependent dataset for the task of detect- ing unpalatable questions. The data comes from a diverse set of online com- munities (orsubreddits) on the popular social media site Reddit. Most other existing datasets for abuse detection [93] [90] [21] [28] [33] only include anno- tations for stand-alone comments or tweets. We are aware of only three other datasets which contain full discussion threads from Wikipedia [42], Instagram [52], and Fox News [30]. However, [42] used an existing toxicity classier called

Perspective API

1to label their data which they acknowledge is a limitation

of their model { their approach will not work for comments where the original toxicity classier itself fails. In comparison, we perform manual annotation where our coders explicitly consider conversational context. Although Gao et al. [30] consider context during manual annotation, our dataset is much bigger in size consisting of 10,909 comments in comparison to their 1,528 comments. Furthermore, we provide at least ve annotations for each comment in com- parison to only two annotations in [30] (see Section 3.3.3 for a comparison with existing datasets).1 http://www.perspectiveapi.com 3 Third, the task of detecting unpalatable questions motivates a method- ological advance as well. A major limitation of existing abuse detection studies { also pointed out in recent work [10] [58] [30] { is that a comment is treated as a single-utterance in isolation, ignoring any additional context provided by other comments in the discussion. This is problematic since abuse is inher- ently contextual and must be interpreted as part of a discussion. It becomes a major issue when working with subtler forms of abuse such as unpalatable questions. Castelle [10] shows how existing machine learning models fail on data which requires contextual enrichment to determine the oensiveness of a comment, achieving a F1-score of 0.3. Moreover, in the impoliteness literature, Culpeper emphasises the importance of context when determining whether a given piece of text is oensive or not [18] [17]. In this work, we investigate the benets of incorporating conversational context in the computational models. For computationally modeling unpalatable questions, we employ several machine learning and deep learning approaches. Using traditional machine learning algorithms such as Logistic Regression, we experiment with a set of hand-crafted features inspired from related literature on abuse detection and rhetorical question detection. We also experiment with deep learning mod- els including Convolutional Neural Networks (CNN) and dierent variants of Recurrent Neural Networks (RNN). We utilise multiple word embedding mod- els such as word2vec [56], GloVe [70], Fasttext [4], and also train our own word embeddings from scratch using Reddit data. We also utilise the recent advancements made in transfer learning for NLP by using deep contextual- ized word representations [71] and Bidirectional Encoder Representations from Transformers [22]. Finally, we incorporate contextual information into all of our computational models in order to investigate whether it improves perfor- mance. All models are evaluated across a set of diverse classication metrics 4 { both threshold-free and threshold-specic { using the standard practice of stratiedk-fold cross validation.

1.2 Summary of Contributions

The contributions of this thesis are summarized below: Inspired from theory of impoliteness, we propose a novel task of detecting unpalatable questions in online discourse. This is a subtler form of abuse which has been largely ignored by existing abuse detection studies [41]; however, such subtle abuse can be just as harmful as overt abuse [61]. We collect, annotate, and make publicly available a diverse context-rich dataset for detecting unpalatable questions. Our dataset consists of

10,909 comments from a set of Reddit communities, with each comment

annotated by at least ve dierent coders. We experiment with several machine learning and deep learning models using state-of-the-art NLP techniques. Moreover, this thesis is one of the few studies to investigate the benets of incorporating conversational context into the computational models. We highlight the importance of using appropriate classication metricsquotesdbs_dbs46.pdfusesText_46

[PDF] langage ada

[PDF] langage c exercices corrigés

[PDF] langage c somme de 2 entiers

[PDF] langage c++

[PDF] langage calculatrice ti-83 plus

[PDF] langage de programmation pdf

[PDF] langage de texto

[PDF] Langage des fonctions, algébrique et lié au graphique

[PDF] langage et mathématiques

[PDF] langage javascript cours

[PDF] langage javascript debutant

[PDF] langage mathématique de base

[PDF] langage naturel maths

[PDF] langage pascal exercices corrigés pdf

[PDF] langage pascal informatique