[PDF] Learning Bilingual Lexicons Using the Visual Similarity of Labeled

View - Download Learning Bilingual Lexicons Using the Visual Similarity of Labeled

Previous PDF

Next PDF

Learning Bilingual Lexicons Using

the Visual Similarity of Labeled Web Images

Shane Bergsma and Benjamin Van Durme

Department of Computer Science and Human Language Technology Center of Excellence

Johns Hopkins University

sbergsma@jhu.edu, vandurme@cs.jhu.edu

Abstract

Speakers of manydifferentlanguagesuse the Inter-

net. A common activity among these users is up- loading images and associating these images with words (in their own language) as captions, file- names, or surrounding text. We use these ex- plicit, monolingual, image-to-word connections to translations. Bilingual pairs of words are proposed as translations if their corresponding images have similar visual features. We generate bilingual lex- icons in 15 language pairs, focusing on words that have been automatically identified as physical ob- jects. The use of visual similarity substantially improves performance over standard approaches based on string similarity: for generated lexicons with 1000 translations, including visual informa- tion leads to an absolute improvement in accuracy of 8-12% over string edit distance alone.

1 IntroductionBilingual lexicon induction is the task of finding words orphrases across natural languages that share a common mean-

ing. In the machine translation (MT) community, such trans- lations are usually obtained from aligned parallel text. For most language pairs, and most domains, parallel data is un- available, and therefore a range of methods have been de- veloped to find translations directly from monolingual text [Fung and Yee, 1998; Rapp, 1999; Koehn and Knight, 2002; Haghighiet al., 2008]. Bilingual lexicons have many uses beyond MT, e.g. in cross-language information retrieval. To find translations using monolingual data, words are associated with information that is preserved across lan- guages. Previous systems have exploited the similar spelling of translationsin related languages

Koehnand Knight, 2002;

Haghighiet al., 2008], and their similar frequency distribu- tion over time [Schafer and Yarowsky, 2002; Klementiev and

Roth, 2006

]. A seed lexicon has also been used to project context words from one language into another; translations are then identified as bilingual pairs of words with high con- textual similarity[

Fung and Yee, 1998; Rapp, 1999]

We exploit the universality ofvisualinformation to build

bilingual lexicons. Billions of images are added to sites likeFigure 1:Matching words through their images: Images retrieved

from the web for the English wordcandle(top) and the Spanish wordvela(bottom). The matching between detected

SIFTkeypoints

is shown for a pair of images.

FacebookandFlickreverymonth.1

Usersnaturallylabeltheir

imagesas theypostthemonline,providinganexplicitlinkbe- tween a word and its visual representation. Since images are labeled with words in many languages, we propose to gen- erate word translations by finding pairs of words that have a high visual similarity between their respective image sets. Figure 1 illustrates our approach for a particular word pair. We use Google"s image search to automatically acquire im- ages for the wordscandlein English andvelain Spanish. We then use computer vision techniques to detect scale-invariant keypointsin each image. These keypointsare used to produce a visual similarity score for everycandle/velaimage pair. We generate a single score forcandle/velaby combining the vi- sual similarity across all image pairs. Using 20 images for each word, our approach ranksvelaas the most likely trans- lation forcandleout of 500 translation candidates, despite there being no identical imagesshared by the two image sets. To our knowledge, this is the first work to induce word translations through labeled images. An unexplored alterna- tive to ourapproachwould be to have (monolingual)speakers of different languages provide words for thesameimages.1 Facebook recently tweeted thatover 750 million images were uploaded over the recent NewYear"s weekend alone: twitter.com/ facebook/status/22372857292005376

1764Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence

For example, the monolingual speakers could play the ESP game [von Ahn and Dabbish, 2004]in different languages, but with the same set of images. Or, we might pay annotators to label images in their native language using online anno- tation services such as Amazon"s Mechanical Turk. Unlike these alternatives, our approach can make use of the many billions of web images and labels that already exist. 2 We show that visual similarityenables improvements over standard approaches to bilingual lexicon induction. We au- tomatically determine a large class ofphysical objectwords where one would expect consistent visual representations across languages. We evaluate our method in a realistic and large-scale lexicon induction task using these words. We also show how our method can provide useful semantic informa- tion for resolving other, monolingual, linguistic ambiguities.

2 The visual similarity of bilingual words

Foragivenword,we automatically: (1)acquireacorrespond- ing set of images, (2) extract visual features from these im- ages, (3) compute the visual similarity of two words using their associated image sets, and (4) use this similarity to rank translation pairs for bilingual lexicon induction. 3

2.1 Using image search engines

Search engines provide a natural way to collect labeled im- ages, given the vast effort that has been expended to refine their widely-usedimageretrieval services. Searchenginesre- trieve images based on the image caption, file-name, and sur- rounding text [Feng and Lapata, 2010]. To automatically re- trieve images, we providea word or phraseas an HTTPquery to the search engine, and directly download the uniformly- sized thumbnails that are returned (rather than downloading the source images directly). For English words, we used

Google"s Image Search

(www.google.com/imghp), while for foreign words, we used the correspondingforeign Google website (all with default settings). For experiments usingW imagesfor a givenword (e.g., Figure 2(a) below), we take the firstWimages returnedby Google. We used Google because previous research has shown that its results are competitive with "hand prepared datasets"[Ferguset al., 2005].Also, in related ongoing work, we achieve higher accuracy using Google images than using images obtained from Flickr.

2.2 Visual features

We convert each image to a representation based on a finite set of visual features. A range of visual features have been exploredin the vision literature, usually in the contextof sup- portingcontent-basedimageretrieval [Deselaerset al., 2008]. Often such features correspond only to local parts of the im- age, and the spatial relationship between these parts is not iar to NLP researchers. We adoptthis bag-of-wordsapproach forourtwotypesoffeatures: colorfeaturesand

SIFTfeatures.

2 Our approach is also independent of the verbosity of a given annotator. Knowledgeable web users will naturally label pictures of orioles,magpiesandcockatoos, whereas a solicited annotator might be inclined to tag all these image with the simple labelbird. 3 Scripts and experimental data are publicly available at: www.clsp.jhu.edu/˜sbergsma/LexImg/

Color histogram

Deselaerset al.

histogram performs well...and can be recommended as a simple baseline for many applications." To create a colorhis- togram, we partition the color space and count the number of image pixels that occur in each partition. We partition colors using the first hexadecimal digit in each pixel"s R, G and B values. This results in a16 3 =4096-dimensionalvector space. Each color partition and its count is used as a feature dimen- sion and its value, respectively, in this color vector space.

SIFTkeypoints

SIFTkeypoints are distinctive local image features that are invariant to scaling and rotation, and robust to illumination, noise and distortion [Lowe, 2004]. They are widely used in vision research, including work that intersects with NLP [Feng and Lapata, 2010]. We identify

SIFTkeypoints using

David Lowe"s publicly-available software:

www.cs.ubc. ca/˜lowe/keypoints/ .SIFTfeatures are taken from im- ages converted to gray-scale. Figure 1 shows the location of SIFTkeypoints detected in two images. We added arcs to il- lustrate keypoints that are close in key-point space. Each

SIFTkeypointisitselfamulti-dimensionalvector. We

convert this bag-of-vectors into a bag-of-words representa- tion by mapping each keypoint to a dimension in a quantized SIFTfeature space. First, we cluster a random selection of

430 thousand keypoints (from our English image data) into

Kcluster centroids using the K-means algorithm. We found the final clustering distortion to be robust to differentrandom initializations. Using the signal processing terminology,each resulting cluster centroidis acodewordin theK-dimensional SIFTcodebook. To quantize the keypointsfor a particular im- age, we map each keypoint to its nearest-neighborcodeword. Each dimension in the resulting feature vector correspondsto a codeword; each value is the count of the number of key- points mapping to that word.

2.3 Combining image similarities

Leteandfbe visual feature vectors for a pair of images. We measure the distance between these vectorsusing their cosine similarity:cosine(e,f)= e·f |e||f| . Many distance functions have been used in the literature and improving this function could be fruitful future work (cf.[Deselaerset al., 2008]). Each word has a corresponding set of images. LetEand Fdenote two such sets in a source and target language. To produce a single word-to-word visual similarity score, sim( E,F), we combine the similarities of all image pairs us- ing one of two scoring functions: A

VGMAXor MAXMAX.

For eache?E,A

VGMAXfinds the best matching image

inF. It averagesthese top-matchesto producea single score: A

VGMAX(E,F)=1

|E|? e?E max f?F (cosine(e,f))(1) M AXMAX, on the other hand, takes the single best match- ing image-to-image similarity as the word-to-word score: M

AXMAX(E,F)=max

e?E max f?F (cosine(e,f))(2) 1765

3 Creating a lexicon of physical objects

We assumethat wordsforconcreteobjects, suchas machines, tools and living things, will have consistent color and key- point features in their associated images. Words that repre- sent more abstract concepts, such asprocrastination,forgot andintolerant, could be visually represented in myriad ways, or might have many irrelevant images in their automatically- compiled image sets. The latter words might therefore be problematic to visually-align across languages. We therefore propose to initially focus on finding transla- tions forphysical objects: words that are both likely to occur in image labels and to have consistent visual representations. A multilingual lexicon of physical objects would have one obvious application: it could be used to extend the reach of multilingual image search engines [Etzioniet al., 2007]. We propose automatic methods for creating a lexicon of physical objects. We first explore a precise but low-coverage pattern-based approach and then a higher-coverage but nois- ier approach based on distributional similarity with a seed lexicon. While our experiments use single-token words, ex- tending our approach to phrases is straightforward.

3.1 Physical objects via pattern matching

We first collect English words filling the following pattern: {image,photo,photograph,picture}of{a,an} We requirethe filler to have a nounpart-of-speechtag and the word after the filler tonothave a noun part-of-speech tag. We count how often each word fills this pattern in Lin et al [2010]"s web-scale, part-of-speech-tagged N-gram cor- pus. We rank words by their conditional probability of co- occurring with this pattern. We filter words that occur in the corpus as nouns less than 50% of the time; we also manually filtered 29 potentially offensive terms. After filtering, the top

500 remaining words were taken as our English lexicon.

The resulting lexicon contains many physical objects (like helicopter,finger,andsword), but also some more general or more abstract concepts:organization,situation,logo,and product. Matching these words based on their visual features represents a challenging task for our approach. While it would be possible to apply this same process to other languages, we want to first evaluate the power of visual similarity independently of the quality of our approach"s lin- guistic components. We thus built corresponding lexicons in foreign languages by directly translating the English words using Google Translate ( translate.google.com/). We take the one-best translation returned by Google Translate and create lexicons in Spanish, German, French, Italian and Dutch. Since differentEnglish words may have the same for- eign translation, the foreign lexicons can be less than 500 words. We use Google Translate because it gives high-coverage translations for the 15 language pairs we experimentedwith. 4 However, note that using a single translation from Google Translate might miss translations for words with multiple senses, and thus make our task more difficult. 4 We did not previously have electronic dictionaries for all these pairs. In Section 5 we also make use of in-house electronic dictio- naries for evaluation in Spanish-English and French-English.

3.2 Physical objects via distributional similarity

The above patterns only identify a small fraction of the phys- ical objects that might be amenab le to visual representation. We create a larger list by finding words that occur in similar contexts to a seed list of physical objects, i.e., words that are distributionally similar. For example, our English seed list has the wordshelicopter,motorcycleandtruck; the larger list has similar wordssubmarine,tractor,andlorry. We use a seed lexicon of 100 physical objects in each lan- guage. Our English seeds are the top 100 words as ranked by the pattern-basedapproach(excludingwordsoccurringfewer than 50 times in the N-gramdata). The foreignseed lists con- sist of the Google translations of the English seed list. We exploit the availability of large corpora in each lan- guage to rank a list of unigrams by their contextual similarity with the seeds. Contextual similarity is defined as the cosine similarity between context vectors, where each vector gives the counts of words to the left and right of the target unigram. We get counts from English and foreign Google N-gram data [Linet al., 2010; Brants and Franz, 2009]. Rather than building the vectors explicitly, we use the locality-sensitive hash algorithm of Van Durme and Lall [2010]to build low- dimensionalbitsignaturesina streamingfashion. Thisallows for fast, approximate cosine computation. We rank the uni- grams by their average similarity with their ten most-similar seeds. The top 20,000 highest-ranked unigrams comprise the final physical object lexicon in each language.

4 Experiments Part 1: 500-word lists

4.1 Set-up

EvaluationWe first test on the 500-word lists created via pattern-matching (§3.1). Here, each source word, indexed byi, has a translation in each target lexicon; let this be at positiontr(i). For each source word"s image set,E i ,werank all foreign image sets,F j , by their similarity withE i .The goal is to haveF tr(i) ranked highest, i.e.,rank Ei (F tr(i) )=1.

We use the following evaluation measures:

•MRR: Mean-reciprocal rank of correct translation: MRR = 1 500
500
i=11 rankEi(F tr(i) (closer to1is better). •Top-Naccuracy: Proportion of instances where the correct translation occurs within the topNhighest- ranked translations. We useN=1, 5 and 20. DataWe use our English-Spanish lists to perform prelimi- nary experiments and to set the parameters of our algorithm (including theλparameters described below). Our final re- sults are the average MRR and Top-Naccuracies across all pairs from English, Spanish, German, French, Italian and Dutch, excluding English-Spanish. Images for each language are collected and processed as described in§2. The proposed rankings are evaluated against the Google translations.

Comparison approachesLetw

E andw F be source and target word strings which have corresponding image setsE andF. We compare the following similarity functions:

1.Random: Randomly score eachE,Fpair.

1766

SystemMRR Top-1 Top-5 Top-20

AVGMAX36.0 31.0 40.8 48.8

M

AXMAX31.5 27.0 35.2 42.0

Table 1:

500-word lists experiment (%): AVGMAXperforms better

than M AXMAXon English-Spanish bilingual lexicon induction. 0 10 20 30
40
50
60

0 5 10 15 20

Number of imagesTop-20 (%)

Top-1 (%)

(a) 0 10 20 30
40
50
60

10 100 1000 10000

Size of codebookTop-20 (%)

Top-1 (%)

(b) Figure 2:500-word lists experiment: Performance of English- Spanish lexicon induction improves with (a) more images per word and (b) more codewords (clusters of

SIFTkeypoints).

2.Color Histogram: Compute visual similarity using

color features only:simcolor(E,F). 3. SIFTs: Compute visual similarity usingSIFTfeatures only:sim SIFT (E,F). 4. SIFTs+Color: Use a linear combination of theSIFTand color histogram similarities: sim SIFT (E,F)+λ 0 simcolor(E,F).

5.Normalized Edit Dist. (NED): Compute the character-

level (orthographic) similarity ofw E andw F using the widely-used edit distance measure. NED uses dynamic programmingto computethe minimumnumberof inser- source stringw E into the target stringw F . It normalizes this edit distance by the length of the longer string. 6.

SIFTs+Color+NED: Use a linear combination of the

two visual and one orthographic measure: sim SIFT (E,F)+λ 1 simcolor(E,F)+λ 2 NED(w E ,w F

4.2 Part 1 results

We first provide results on our English-Spanish development data. We use this data to investigate three key components of our algorithm: the scoring function (default A

VGMAX), the

number of images in each image set (default 20) and the SIFT codebookdimensionality(default20,000). For simplicity, we investigate these components using only

SIFTfeatures.

Table1showsthatwegetaconsistentgainusing A

VGMAX rather than MAXMAXscoring. Our approachthereforelever- ages not just the exact image matches in the image sets, but aggregate information over many weaker matches. The number of images that we use in each image set has a strong impact on both performance and efficiency (com- puting A

VGMAXincreases quadratically with the number of

imagesineach imageset). WhiletheTop-1accuracyplateaus around20 images (Figure 2(a)), the Top-20scores are still in-System

MRR Top-1 Top-5 Top-20

Random1.4 0.2 0.9 4.1

Color Histogram

19.6 14.4 23.2 35.6

SIFTs32.1 27.4 35.7 45.3

SIFTs+Color36.7 31.1 41.453.7

Normalized Edit Dist.

41.7 37.3 45.852.9

SIFTs+Color+NED53.6 48.0 59.5 68.7

quotesdbs_dbs19.pdfusesText_25

[PDF] 1000 most common english words xkcd

[PDF] 1000 most common words in french

[PDF] 1000 most common words in german

[PDF] 1000 most common words in italian

[PDF] 1000 most common words in japanese

[PDF] 1000 most common words in korean

[PDF] 1000 most common words in portuguese

[PDF] 1000 most common words in spanish

[PDF] 1000 regular verbs pdf

[PDF] 1000 spanish verbs pdf

[PDF] 1000 useful expressions in english

[PDF] 1000 words essay about myself

[PDF] 1000 words essay about myself pdf

[PDF] 10000 cents to dollars

[PDF] 10000 most common english words with examples and meanings