COST250 Speaker Recognition Reference System

8. Results

This section presents some recognition results with the Reference System.

8.1 Polycost

The Polycost database is publicly available from ELRA. Together with a set of four pre-defined baseline experiments, it is useful as a reference database. It provides a way for researchers to compare the performance of their recognition systems, since the same recognition experiment can be performed at many sites. Similarly, this Reference System also provides a way to compare other recognition systems, but also to compare the difficulty of recognition tasks and data.

In this section, the Reference System is tested on the Polycost database. The presented results should be reproducible at any site with access to the Polycost database and the Reference System. A miniature version of BE3 is also described in the calibration section and is used to test the installation of the Reference System.

Figure 1 shows DET-curves for the three speaker verification baseline experiments with the default setup of the Reference System. Note that only the default setup corresponds to a "true" reference system in the sense that it can be used as a reference when comparing two different recognition systems to the common Reference System. Of course, when the system is used for other purposes (such as a teaching tool or a research vehicle for trying new system parts or settings) system settings can be varied. This has been done in Table 4 and Figure 2 where results are presented for different codebook sizes in the VQ-classifier. The default setting of 64 VQ codewords in the client and 64 in the non-client model was chosen based on these results.


Figure 1. DET-curves for the Reference System (with codebook size 64) on the three speaker verification baseline experiments on the Polycost database. 

---------------------------------------------------------------
  Polycost, BE1      |  codebook size, client model
                     |
               EER   |	  16      32      64*   128     256
---------------------+-----------------------------------------
  cb. size,    16    |  14.8
  non-client   32    |  13.9    11.9
  model        64*   |  14.8    12.5    12.8*
              128    |  14.2    12.0    11.9    11.9
              256    |  14.2    12.4    12.4    11.9    11.6
---------------------------------------------------------------

---------------------------------------------------------------
  Polycost, BE2      |  codebook size, client model
                     |
               EER   |   16      32      64*    128     256
---------------------+-----------------------------------------
  cb. size,    16    |  12.8
  non-client   32    |  12.8    11.9
  model        64*   |  12.4    11.0    11.0*
              128    |  12.0    11.2    10.6    11.6
              256    |  11.4    11.0    10.8    11.3    11.2
---------------------------------------------------------------

---------------------------------------------------------------
  Polycost, BE3      |  codebook size, client model
                     |
               EER   |	  16      32      64*    128     256
---------------------+-----------------------------------------
  cb. size,    16    |  18.2
  non-client   32    |  17.6    17.0
  model        64*   |  17.1    16.9    15.7*
              128    |  17.5    16.4    16.3    15.7
              256    |  18.2    16.7    16.5    16.0    15.8
---------------------------------------------------------------
Table 4. Test set EER for various codebook sizes in client and non-client model and the three speaker verification baseline experiments on the Polycost database. Codebook size 64 corresponds to the default Reference System setting (marked with a '*').


Figure 2. Test set EER for various codebook sizes and the three speaker verification baseline experiments on the Polycost database. Codebook size is the same in client and non-client model. Codebook size 64 corresponds to the default Reference System setting.