RSVP-iconCHAT

Background

RSVP-iconCHAT is sponsored by the National Science Foundation (NSF) under grant #HCC-0914808 and jointly investigated by Dr. Patel and Dr. Erdogmus.

Task

Single-switch augmentative and alternative communication (AAC) systems are typically simplifications of multi-array systems that share elements of the same interface layout, but support some form of scanning, such as linear or row-column. From a development standpoint, this conversion technique makes it easy to transform almost any AAC system into a single-switch system; however, it means that most of these systems were originally designed for large screens and users with much greater mobility. The purpose of this project is to create a small-footprint, icon-based AAC system specifically for use with binary signals, such as switches.

Approach

The interface builds upon prior work in the CadLab, specically a system called IconCHAT that leveraged semantic frames to allow for free-order message construction. We created an accompanying language model that supports unordered word prediction and tested the system using motoric input. We are currently working with Dr. Erdogmus's team in the Electrical and Computer Engineering (ECE) department to integrate a brain-controlled interface (BCI) based on surface-level electroencephalography (EEG) and detection of P300 brain waves.

Technologies Used

C, Git, Linux, Matlab, Mercurial, NLTK, OS X, Python, SDL (LibSDL/Pygame), WordNet