Binyam Gebrekidan Gebre, Max Planck Institute for Psycholinguistics
Automatic gesture stroke detection has important applications in a) human-computer interaction and b) video content annotation. Success in the latter application, which is the subject of my talk, has important implications in speech, gesture and sign language research. In my talk, I will present a work-in progress annotation model that allows a user to a) track hands/face movements b) extract spatial and temporal features and c) detect gesture strokes. The hands/face tracking is done with color matching algorithms and is initialized by the user. The stroke detection is done using training data and machine learning algorithms (both generative and discriminative). The key element in this model are the features used. In our experiments, we used only image features, but the stroke is known to be synchronized with the speech segments that are co-expressive with it. Therefore, it will be enlightening to hear the views of speech researchers on how speech features can be incorporated in gesture stroke detection.