Home Language & Literature COGNITIVE APPROACH TO NATURAL LANGUAGE PROCESSING
In this section, we present the experimental setting for the evaluation and comparison of our system with state-of-the-art algorithms.
We evaluated our algorithm with three fine-grained datasets: Senseval-2 English all-words1 (S2) [PAL 01], Senseval-3 English all-words  (S3) [SNY 04], SemEval-2007 all-words (S7) [PRA 07] and one coarse-grained dataset, SemEval-2007 English all-words (S7CG) [NAV 07b], using WordNet as a knowledge base. The descriptions of the datasets are presented in Table 6.1.
The results of the evaluation are presented as F1, which is calculated as:
This measure determines the weighted harmonic mean of precision and recall. Precision is defined as the number of correct answers divided by the number of provided answers and recall is defined as the number of correct answers divided by the total number of answers to be provided. In our evaluation, we excluded labeled points in this calculation. Experimentally we noticed that precision is always equal to recall, since the system is always able to provide an answer.
We evaluated two different versions of the system, one using a uniform probability distribution to initialize the strategy space of the games and the other using information from sense labeled corpora (see section 6.4.2). Furthermore, to make the evaluation unbiased, we present the mean and standard deviation results of our system over 25 trials with different sizes of randomly selected labeled points.
Table 6.1. Number of target words and senses for each text of the datasets
|< Prev||CONTENTS||Next >|