Multimodal interaction and people with disabilities

The opportunities offered by modern multimodal technology are important for people with disabilities because they create new forms of communication. Disabilities affect a wide range of peoples' abilities and functions. There are no panaceas or universal themes - but a common one is problems with communication.

Communication problems may be due to

sensory impairments
deafness, blindness etc
physical impairments
for instance, lack of motor control of the speech articulatory system
cognitive impairments
These may affect, human-human communication, human-computer interaction and human-human communication which is mediated by computer.

Multiple modalities are very important because a deficit in on channel of communication may be overcome by expliotation of another. Sensory deficiencies can be sub-divided into conditions where there is a complete lack of useful communication (i.e. deafness or blindness). In that case it is necessary to translate information normally presented on the unavailable channel into a form accessible on one - or more - of the remaining channels. For instance, a great amount of effort has been expended on making the kind of information displayed on computer screen accessible to blind people, using speech and non-speech sounds.

Partial loss of a sensory channel (i.e. a hearing loss short of total deafness or partial sight) often is remediated (to some extent) by enhancing input to that channel. Thus, the most important prosthesis for people with a hearing loss is the hearing aid, which amplifies input to the ear. Similarly most visual impairments can be remediated by eye glasses. However, it can also often be desirable or necessary to supplement the enhanced input on the primary channel with information transformed to one of the other channels. For instance, a deaf person may use visual aids to help improve their speech, to see characteristics of their voice that they cannot hear - even with the assistance of a hearing aid.

The reason we communicate particular information on one channel is usually because it is more efficient. Therefore to try to communicate the same information on another channel is likely to be less efficient. Thus there can be problems of 'bandwidth'. These may be remediated by making better use of multiple channels, ideally with some level of redundancy.

At the extreme, is it always possible to transform modalities? For instance, what is the appropriate non-visual representation of the Mona Lisa?

We can review the level that has been reached in applying such ideas, but we will also highlight that there is a lot of scope for more research as to how to make use of novel modes of communication.