Pre-Presentation Notes Slides and presentation materials are available online at: karlwiegand.com/csun2014 RSVP-iconCHAT: A Single-Switch, Icon-Based AAC Interface Karl Wiegand Rupal Patel, Ph.D. Northeastern University (USA) March 1, 2013 Outline The Vision Background and Scope Approach Evaluation Results Part 1: The Vision http://www.emotiv.com/ http://www.neurosky.com/ Brain-Computer Interfaces (BCI) http://www.thalmic.com/en/myo/ http://research.microsoft.com/en-us/um/redmond/groups/cue/MuCI/ Muscle-Computer Interfaces (MuCI) http://www.vuzix.com/consumer/products_m100/ http://www.google.com/glass/ Wearable Computing Systems How can we think about the future of AAC interfaces? Part 2: Background and Scope Current AAC Interfaces Conversion to Single-Switch Row-Column Scanning Observations Maintains familiarity. Focus is on the vocabulary. Required screen size is tied to vocabulary size or navigation complexity. Users usually perform the search twice. Searches involve repetitive movements. Our Collaborative Effort Creation of a icon-based, BCI-controlled AAC system NSF Grant HCC-0914808 Jointly investigated: Dr. Deniz Erdogmus (ECE) Dr. Rupal Patel (SLPA and CCIS) BCI uses P300 brainwave (unary signal) The P300 Wave Requirements and Scope Based on icons, not letters: Some letter-based BCI systems exist. Icons have the potential to be faster. Users may be pre-literate or have language impairments. Minimize head, neck, and eye movements: Movement can dislodge the BCI equipment. Work with a unary signal (P300): Expand to support other signals if available. Part 3: Approach Driving Questions How can we decouple the required screen size from the vocabulary size? Can we focus on the message instead of the vocabulary? How much work can be shifted from the user to the system? Idea #1: RSVP Rapid Serial Visual Presentation Used in psychology, speed-reading, lie detection, and prior letter-based AAC Orhan, U., II, K. E. H., Erdogmus, D., Roark, B., Oken, B. & Fried-Oken, M. (2012). RSVP keyboard: An EEG based typing interface.. In ICASSP (pp. 645-648) . IEEE . ISBN: 978-1-4673-0046-9 Idea #2: Semantic Frames Semantic frames, CxG, and PAS (Fillmore) To Give ( Agent, Object, Beneficiary ) "I gave the item to him." "The item was given to him by me." WordNet, FrameNet, "Read the Web," NELL Easier to convert from semantic to surface RSVP-iconCHAT Example Usage Wiegand, K., Patel, R., & Erdogmus, D. (2010). Leveraging Semantic Frames and Serial Icon Presentation for Message Construction. ISAAC Conference for Augmentative and Alternative Communication, Barcelona, Spain, July 2010. Characteristics De-emphasized textual representation Semantic frames can be populated in any order Configurable switch modality Configurable ordering patterns Configurable frame complexity Complexity vs. Real Estate Observations Required screen space is now tied to message complexity. Full vocabulary is hidden/filtered. Prediction/ordering controls speed of construction. Some classifications are tricky -- where should illocutionary acts (e.g. small-talk) go? Part 4: Evaluation User Group #1: "ND" Non-disabled (ND), to provide a theoretical upper bound on performance 24 English-speaking adults 10 males and 14 females Ages 19 - 43 (mean of 24) User Group #2: "SMI" Speech and motor impairments (SMI) Ages 33 - 56 (mean of 41) ID Sex Motor Speech Mode P1 F Mild Mild Unaided P2 M Mild Moderate Unaided P3 F Moderate / Severe Mild Unaided and Switch P4 M Severe Severe Caregiver Constrained Message Elicitation More open-ended than "copy phrase" More comparable than real-world usage Closed vocabulary controlled via single-action picture cards Example Picture Cards Study Setup 30 shuffled cards per person Space bar as switch Starting RSVP speed of 700ms Adjustable by +/- 100ms 106 words tagged in up to 8 roles Unlimited time and alphabetic ordering Part 5: Results Quantitative Summary No nonsensical utterances Average of 5 selections (verb + 4) Average speed of last 5 utterances: 70s (ND) vs. 107s (SMI) RSVP speeds w/ positive motor response: 700ms (ND) vs. 1200ms (SMI) Similar learning curves for both groups Mid-experiment errors may have been exploration Qualitative Feedback All users get restless w/ alphabetic ordering Even alphabetic ordering can be surprising Numerous users asked about other switching methods and multi-modal adaptations Numerous users favorably mentioned the automatic syntax modification/correction Epilogue: Closing Thoughts Potential Improvements Slow down near more likely words Filter unlikely words based on frames Skip to more likely alphabetic positions Carousel: Current Applications Small-screen and mobile systems Perhaps combined with a hand-held controller Multi-modal or analog input combinations: Push the switch harder to go faster Directional switches "Oops" functionality Involuntary responses: muscle twitch, BCI Can then leverage predictive reordering Initial results expected later this year Thanks to Dr. Rupal Patel, Dr. Deniz Erdogmus, and the National Science Foundation (Grant #0914808). Thank you for listening! karlwiegand.com/csun2014