Teleping: Automatic evaluation of spoken dialogue systems
Opponent: Janine Wicke
This project deals with the construction of an evaluation system for spoken dialogue systems. Various properties of such systems are discussed, as well as required properties of evaluation measures. A suggestion of such a system and measure, with required properties is made, and implemented. The evaluation system Teleping uses log files of old dialogues for a spoken dialogue system to test its functionality by utterance verification. Teleping is shown to correctly diagnose 114 out of 129 tried dialogues. The greatest benefit of this system is the automation with which it operates, which allows for effortless, but comprehensive evaluation. It is constructed so as to be used on real time telephone dialogue systems.