Project WordGraph

New approaches for the acquisition of lexical information from text corpora based on graph theory

WordGraph

Short description

The goal of the research project WordGraph is to develop new approaches for the acquisition of lexical information from text corpora. These approaches are based on graph theory. In particular, we are investigating node similarity algorithms such as SimRank for the induction and extension of bilingual lexicons.

Sponsor
German Research Foundation (DFG)
Long description

Most natural language processing systems rely heavily on information about words and their meanings as provided by a lexicon. However, a lexicon is never complete. Language evolves constantly, among others due to morphological productivity, sense extensions, loans from other languages, and the constant introduction of new technological and scientific terminology. Since the manual maintenance of lexicons is not only slow, but also susceptible to inconsistencies, automatic acquisition of lexical information has become an important research area and a practical necessity for large systems working with real data.

The goal of the DFG-sponsored research project WordGraph is to develop new approaches for the acquisition of lexical information from text corpora. These approaches are based on graph theory.

Relationships between words in a text can be naturally represented by a graph which has words as nodes and relationships between them as edges. The nodes and edges in such a textual graph are of various types. Node types correspond to word classes (e.g. nouns, verbs, adjectives), and edge types represent different kinds of dependencies between them (e.g. syntactic dependencies, joint occurrence in a coordination, co-occurrence). The meaning of a word is characterized by its relationships (links) to the other words (nodes) in the word graph. The connectivity structure of the word graph thus contains valuable information about words and their meanings.

In particular, we are investigating node similarity algorithms such as SimRank for the induction and extension of bilingual lexicons.

 

Ressourcen

As part of this ongoing project, we have created resources that we believe to be useful for other researchers on lexical acquisition as well as the general NLP research community. We provide these resources as a service to the community.

 

Noun Coordination Data

 

Large dataset of nouns that occur together in a coordination (such as "X and Y"). Extracted from Wikipedia.

Each line contains a single coordination. Each word is annotated with part of speech tag and lemma, separated by slashes: Word/Tag/Lemma

Examples:

complexity/NN/complexity and/CC/and length/NN/length
history/NN/history and/CC/and cultural/JJ/cultural heritage/NN/heritage

Luft/NN/Luft und/KON/und Wasser/NN/Wasser
der/ART/d Starbesetzung/NN/Starbesetzung und/KON/und der/ART/d technischen/ADJA/technisch Raffinessen/NN/Raffinesse

 

 

Adjective-Noun modification data

 

List of adjectives modifying nouns (extracted from Wikipedia)

Each line contains a single adjective-noun pair.

Examples:

left-wing       ideology
political       party
religious       leader

chemisch        Element
deutsch         Film
grell           Lampe

 

 

Verb-object data

 

List of verbs and their direct object (extracted from the Wikipedia-derived parse trees above).

Each line contains a single verb-object pair.

Examples:

turn#off        brain
outwit  enemy
rouse   suspicion

abfahren        Strecke
weiterentwickeln        Technik
annehmen        Ruf

 

 

Cross-lingual Relatedness Thesaurus

 

We used graph similarity algorithms to create a bilingual semantic relatedness thesaurus. For every English word, there are ten German words deemed related by the algorithm. and vice versa. The method used to create this resource will be described in a forthcoming publication (accepted at LREC2010).

The data comes in gzipped text files, each containing blocks of one word and ten related words, each on its own line. The lines of the related words are indented with a TAB character. The next block is separated by an empty line.

Example:

(lion,n)
        (Panther,n)
        (Nashorn,n)
        (Löwe,n)
        (Büffel,n)
        (Jaguar,n)
        (Leopard,n)
        (Tiger,n)
        (Puma,n)
        (Elefant,n)
        (Antilope,n)

(Möwe,n)
        (gull,n)
        (swan,n)
        (goose,n)
        (duck,n)
        (teal,n)
        (flamingo,n)
        (loon,n)
        (grebe,n)
        (cormorant,n)
        (tern,n)

 

 

Parsed Wikipedia Data

 

We have parsed the text of English and German Wikipedia articles using BitPar. This is one of the few large collections of compararable text parsed with the same parser.

The data comes in an archive that bundles gzipped files, each containing about 500 parsed sentences. Each line consists the parse tree of one sentence encoded as structure of nested brackets.

Example sentence "It is one of 58 counties of Gansu.":

(TOP
 (S/fin/.
  (NP-SBJ/3s/base
   (PRP/3s It))
  (VP/3s
   (VBZ/n is)
   (NP-PRD/pp
    (NP/base
     (QP
      (\<QP\[CD\]IN/of|CD\>
       (CD one)
       (IN/of of))
      (CD 58))
     (NNS counties))
    (PP/of/NP
     (IN/of of)
     (NP/base
      (NNP Gansu)))))
  (. .)))

 

 

Co-occurrence Data

 

List of co-occurring word tuples extracted from Wikipedia. The word-word coocurrences were extracted using a cooccurrence window of 3 consecutive words. The files contain tables in the form of Stefan http://www.collocations.de/software.html. The l1 column contains the left word, the l2 column the right word. The right word is appended with "_x" where x is the position in the context window. Example ""Aachen amtlich_3"" means that the word "amtlich" occurred to the right of "Aachen", with two words inbetween. Also included are frequency counts of the left word (f1), the right word (f2, positions are distinguished), counts of the pair f, and log-likelihood statistics am.log.likelihood. The pairs are filtered for space reasons, only pairs with occured more than once, with individual word frequencies > 100 and a log-likelihood ration > 3.87 are included.

(Tables coming soon)

 

Lexicon Induction Test Dataset

Comparative evaluation of methods for bilingual lexicon induction is hampered by the lack of a common evaluation methodology and a common test dataset. Together with Reinhard Rapp (Johannes Gutenberg University Mainz), we propose a common test dataset for the evaluation of lexicon induction experiments. We hope that this data will serve as a basis for a standard evaluation.

 

General Contact IMS

Pfaffenwaldring 5 b, 70569 Stuttgart

 

Webmaster of the IMS

  • Write e-mail
  • If you have any problems with the website, please directly contact the webmaster.
To the top of the page