Deep Semantic Analogies

Various evaluation benchmarks regarding semantic analogies for English and for German

Deep Semantic Analogies

Maximilian Köper, Christian Scheible, Sabine Schulte im Walde

This resource comprises various evaluation benchmarks regarding semantic analogies for English and for German:

  • de_trans_Google_analogies:

The Google semantic/syntactic analogy datasets were introduced in Mikolov et al. (2013). The datasets contain analogy questions of the form A:B::C:D, meaning A is to B as C is to D, where the fourth word (D) is unknown. We constructed German counterparts of the datasets through manual translation and subsequent cross-checking by three human judges. We omitted the relation type “adjective–adverb”, because it does not exist in German. The final task set contains 18,552 analogy tasks.

  • en/de_sem-para_SemRel:

The paradigmatic semantic relation dataset also contains analogy tasks. Here, the paradigmatic relation between A and B is the same as between C and D. The dataset was constructed from antonymy, synonymy, and hypernymy relation pairs collected by Lenci & Benotto for English and by Scheible & Schulte im Walde for German, using the methodology described in Scheible and Schulte im Walde (2014). The questions cover the semantic relations adj antonym, noun hypernym, noun synonym, noun antonym and verb antonym. Overall, this dataset constitutes a deep semantic challenge, containing very specific, domain-related and potentially low-frequent semantic details that are difficult to solve even for humans. For example, the tasks include antonyms such as biblical:secular::deaf:hearing or screech:whisper::ink:erase. The English dataset contains 7,516 analogies and the German dataset contains 2,462 analogies.

  • en_sem-para_BLESS

In the same way, we created an analogy dataset with 10,000 unique analogy questions from the hypernymy and meronymy relations in BLESS (Baroni and Lenci, 2011), by randomly picking semantic relation pairs.

Next to analogies, the resource also contains:

  • de_new-rated_Schm280:

Schm280 (Schmidt et al., 2001)  translated 280 word pairs from WordSim353 (Finkelstein et al., 2001). As they did not re-rate the German relation pairs after translation, we collected new ratings for the German pairs from 10 subjects, applying the same conditions as the original WordSim353 collection. The dataset contains 280 translated and newly rated word pairs for WordSim350.


Maximilian Köper, Christian Scheible, Sabine Schulte im Walde (2015)
Multilingual Reliability and “Semantic” Structure of Continuous Word Spaces
In: Proceedings of the 11th International Conference on Computational Semantics (IWCS). London, UK.



Logo of Research Group SemRel

Research Group SemRel

This image shows Sabine Schulte im Walde

Sabine Schulte im Walde

Prof. Dr.

Akademische Rätin (Associate Professor)

To the top of the page