Sign learning via game-based interaction
TIVOLI aims to create learning application for sign language signs, in the form of a computer game featuring sign recogntion via webcam and a signing avatar. The target group is children with communication disorders.
Today many children need signing as a support in their daily communication, but the opportunities and motivation to learn are very different. At the same time the TV and computer games form a large part of many children and young people's everyday lives. We want to create a computer game in which the child learns to sign by recieveing instructions from a signing avatar, and then using sings to control the game (via camera). In this way, we believe that learning will automatic and fun and that the child rapidly can increase their vocabulary. A bonus is that the game can be used in social contexts and the child can play with others on equal terms.
Technically, the project will aim to develop robust techniques for capturing sign language signs via motion capture for avatar animation, and also to incorporate and adapt state-of-the art visual tracking techniques to analyze the signs of the user.
Group: Speech Communication and Technology
Jonas Beskow (Project leader)
Maria Olsson (DART)
Sandra Derbring (DART)
Morgan Fredriksson (Liquid Media)
Funding: PTS - Post och Telestyrelsen (Tivoli)
Duration: 2011-09 - 2013-09
Keywords: sign language, signing avatars, sign recognition, communication disorders, children, computer game
Tivoli - teckeninlärning via spel och interaktion. Slutrapport projektgenomförande. Technical Report, PTS Innovation för alla. [pdf] (2014).
Proceedings of the 1:st european Symposium on Multimodal Communication. [pdf] (2013). The Tivoli System – A Sign-driven Game for Children with Communicative Disorders. In
Proceedings of the 2013 Workshop on Multimodal Corpora: Beyond Audio and Video. [pdf] (2013). A Kinect Corpus of Swedish Sign Language Signs. In
(2012). Visual Recognition of Isolated Swedish Sign Language Signs.