Home Language & Literature COGNITIVE APPROACH TO NATURAL LANGUAGE PROCESSING

# Evaluation

In this section, we present the experimental setting for the evaluation and comparison of our system with state-of-the-art algorithms.

## Experimental setting

We evaluated our algorithm with three fine-grained datasets: Senseval-2 English all-words1 (S2) [PAL 01], Senseval-3 English all-words[1] [2] (S3) [SNY 04], SemEval-2007 all-words[3] (S7) [PRA 07] and one coarse-grained dataset, SemEval-2007 English all-words[4] (S7CG) [NAV 07b], using WordNet as a knowledge base. The descriptions of the datasets are presented in Table 6.1.

The results of the evaluation are presented as F1, which is calculated as:

This measure determines the weighted harmonic mean of precision and recall. Precision is defined as the number of correct answers divided by the number of provided answers and recall is defined as the number of correct answers divided by the total number of answers to be provided. In our evaluation, we excluded labeled points in this calculation. Experimentally we noticed that precision is always equal to recall, since the system is always able to provide an answer.

We evaluated two different versions of the system, one using a uniform probability distribution to initialize the strategy space of the games and the other using information from sense labeled corpora (see section 6.4.2). Furthermore, to make the evaluation unbiased, we present the mean and standard deviation results of our system over 25 trials with different sizes of randomly selected labeled points.

 Dataset Text N C Tot. N S2 1 670 2195 2387 S2 2 997 1836 S2 3 720 1916 S3 1 783 2472 2007 S3 2 633 1426 S3 3 591 1881 S7 1 111 593 455 S7 2 150 798 S7 3 194 1035 S7CG 1 368 1287 2268 S7CG 2 379 1473 S7CG 3 499 1926 S7CG 4 677 1666 S7CG 5 345 1410

Table 6.1. Number of target words and senses for each text of the datasets

• [1] www.hipposmond.com/senseval2
• [2] http://www. senseval.org/senseval3
• [3] http://nlp.cs. swarthmore.edu/semeval/tasks/index.php
• [4] http://lcl.uniroma1.it/coarse-grained-aw

 Related topics