Alexander Moore: A Classification Based Methodology for the Evaluation of Neural Probabilistic Language Models

BCCN Berlin and TU Berlin

 

Abstract

Vector space models of semantics are increasingly popular in natural language processing. Improvements in deep learning, particularly convolutional neural networks, have increased the need for robust numerical representations of words’ meanings. Traditionally Vector space models were assessed via human similarity ratings. More recently they have been tested using analogy tasks, these aim to check whether or not particular lexical or semantic relationships are conserved in vector space. We propose a novel classification based methodology to evaluate these models. Our methodology is data driven and doesn’t require direct human participation; it assesses the extent to which distinct semantic concepts overlap in vector space.
We used our new method to evaluate two vector space models, Continuous Bag of Words (CBOW) and Skip-Gram. We were particularly interested in checking if these models could capture complex features while embedding proper nouns. We decided to use a large newspaper corpus and focus on three complex features: gender, profession and animacy (this feature related to improper nouns). Using our classification methodology we assessed how well the models encoded these complex features. We also observed how the performance of these models changed as we varied context window size (an important hyperparameter). In the end we found that CBOW models were better able to encode complex features with regard to proper nouns whereas Skip-Gram models were better suited when it came to improper nouns. The relationship to window size differed, with CBOW models performing better with smaller window sizes whereas Skip-Gram models performed better with larger window sizes.

Organized by

Benjamin Blankertz / Robert Martin

Location

BCCN Berlin, lecture hall, Philippstr. 13 Haus 6, 10115 Berlin