COST250 Speaker Recognition Reference System

5. Calibration Test

To test your Reference System installation you shall run a miniature experiment. The result file from this experiment can be compared across sites and platforms to verify that the results are re-producible. The test should take no more than 3-15 minutes to run depending on your machine.

Within the test the following three tasks are carried out:

The easiest way to run the test is: 
     cd refsys/$version
     make test
Or, if you have problem with using the makefile you can run the test "manually": 
     cd refsys/$version/work
     ./runtest.tcl polycost/test
     makedet.csh polycost/test/results.llk > polycost/test/results.det
     eer.csh polycost/test/results.llk
It may be useful to redirect output to a file for later reference. This is done differently with different shell programs. For example, if you are using C-shell (csh or tcsh) you can do 
     make test >& work/polycost/test/test.log
to send all log output from the test to the given file. If you want to see what is happening while you run the test you can use tee to send output to file and to the screen: 
     make test |& tee work/polycost/test/test.log

5.1 Comparing results

Copies of all the files that were created during the calibration test on different platforms at various sites are stored in a directory structure 
     refsys/$version/work/calib/site/architecture.os/polycost/test
The tag architecture.os can be for instance i386.Linux or hp700.HP-UX, i.e. the first part indicates the hardware architecture and the second part indicates the operating system. You can compare your score file to those for i386.Linux produced at KTH by typing: 
     make compare
which runs two simple diff:s, one for the score (results.llk) file and one for the DET-curve (results.det). If no differences are reported, then your results are equal to those of the reference (at least at the score output and DET-curve levels).

5.2 Reporting results

Once you have installed the Reference System, run the calibration test and compared your score file to the given reference, please report your findings to the authors: Pack all produced files together into one file, and send it as an email attachment to melin@speech.kth.se. Using tar: 
     cd refsys/$version
     tar cvf mySite.myArch.myOS.tar work/polycost/test
     gzip mySite.myArch.myOS.tar
     -- send mySite.myArch.myOS.tar.gz by email --
Please also include bugfixes, problem reports or comments of any kind.