[PDF] Filtering COMPS Chat Transcripts for Computer Modeling using

View - Download Filtering COMPS Chat Transcripts for Computer Modeling using

Previous PDF

Next PDF

Filtering COMPS Chat Transcripts for Computer Modeling using Common Vocabulary discussions. developedfortheseissuesarepresentedhere. xbuild/updateacommonvocabulary xinspecttranscripts xfiltertextforissueswiththetypedlexicon xremoveorspeciallyhandlecommonvocabulary oFilters for special charactersadd a unique token identifying the character used. ƒ$g02 for *, $g03 for #, $g04 for @, $g05 for ^. ƒFor *, ^, and #, attached word kept if on common list ƒFor *, only one token added if * surrounded the *start and end* of words.

oFilters for stretched letterscollapse the stretched letters to a word on the common list when possible.

ƒNo English non-hyphenated words have more than double letters. ƒAll cases of duplicate letters changed to double letters.

ƒAll combinations of double/single letters checked: longest resultant word on common list (closest to original) chosen.

ƒAll words 1 edit away from misspelled word checked. ƒOriginal Google 10,000 list is in frequency order: Choose resulting word with highest frequency.

What Filtering Programs Do

Vocabulary Inspection

Foundation.

Vocabulary Basis

publicwebpages2 o>90%everydayEnglishusagecoverage xNeededtoremovecommonpersonnamesfromlist oSetdifferencewith1990UScensusdata3 o90%USpopulationnamecoverage. oSlangandabbreviations. oNamesthataremoreoftenregularwords andshownindiagrambelow

Nathaniel Bouman

Dept. of Computing and Information Sciences, Valparaiso University oFRPSMUHG ³-MYM´ SURNOHP MQG ³3RLVRQ´ SURNOHP PUMQVŃULSPVB oTotal occurrences is how often a word occurred in total. oTranscript appearances is how many transcripts a word appeared in. o³2Q FRPPRQ JRUG ILVP"´ PMUNHG 1 LI ³\HV´ MQG 0 LI ³QR´B oResearch students used programs to identify and add words to common words list. xChartshownabovehasselecteddata.

AbstractContinuing Challenges

Future Work

Acknowledgements

Word "Poison" Total

Occurrences

"Poison"

Transcript

Appearances

"Java" Total

Occurrences

"Java"

Transcript

Appearances

On

Common

Word List?

we91225543551 think27925444541 were621953301 asking3336221 skipping00330 wayyy11000 won5618111 totalmortgage00110 encapsulate00540 *instantiating00110 #tired11000 Original TextText after Word FiltersText after Word and Vocab Filters

This document is for testing filtersthis document is for testing filtersthis document is for testing filters

suchhhas stretchingletttttersssssuchas stretchingletterssuch as $g01letters using *stars*for *emphasisusing $g02stars for $g02emphasisusing $g02 stars for $g02 emphasis

or, *the same thing*with multiple wordsor $g02the same thing with multiple wordsor $g02 the same thing with multiple words

#and#thehashtag@atand ^carets$g03and $g03$g04and $g05$g03 and $g03 $g04 and $g05 spellinerroprsof variouztypesspellingerrorsof varioustypesspelling errors of various types and finally, junk: ixmvbulzewand finally junk ixmvbulzewand finally junk $g01 incombinationwiththeprobability. changedto³OHPPHUV´butshouldnotbe). updatedwithlanguagetrends. modelingofCOMPStranscripts. roomforimprovement. versionsofcomputermodelingprograms.

StamatinaKalafatis,andYesukheiJagvaral.

References(allaccessed27July,2017):

1.google-10000-english. Initial commit 29 Mar. 2012.

2.Franz, Alex and Thorsten Brants. All Our N-gram are

Belong to You. 3 Aug. 2006. https://research.googleblog

3.Meranda, Deron. Names of People.

http://deron.meranda.us/data/

4.Norvig, Peter. How to Write a Spelling Corrector. Feb.

2007 to Aug. 2016. http://norvig.com/spell-correct.html

Common words which occurred frequently

Common word not already on list to be

manually added

Example of stretched letters phenomena

Problem-specific word considered common

Problem specific words notto be added to

common word list.

Examples of special characters phenomena

10,000 most

common wordsNames

Common words

without names

Set difference

program

Vocab comparison and

inspection programs

³-MYM´

transcripts

³3RLVRQ´

transcripts

Set Union

programCommon words without names and with manual additions

Transcript vocab

absorb program

Transcript vocab

absorb program

Manually identified

additionsquotesdbs_dbs17.pdfusesText_23

[PDF] 10000 most common english words with meaning pdf

[PDF] 1000ml over 12 hours

[PDF] 100mhz 5g

[PDF] 101 ambulance for sale

[PDF] 101 ambulance kerala

[PDF] 101 creative writing exercises pdf

[PDF] 101 great answers to the toughest interview questions pdf

[PDF] 101 interview questions and answers pdf

[PDF] 101 number ambulance

[PDF] 101 online ambulance

[PDF] 101 phone number

[PDF] 101 toughest interview questions and answers that win the job

[PDF] 1040 fillable form

[PDF] 1040 form 2017 schedule a

[PDF] 1040 form 2017 schedule c