Pre-Presentation Notes Slides and presentation materials are available online at: karlwiegand.com/csun2013 CSUN 2013: Novel Approaches to Icon-Based AAC Karl Wiegand Rupal Patel, Ph.D. Northeastern University March 1, 2013 My name is Karl Wiegand, and I'm a Ph.D. student in computer science at Northeastern University... Outline Why Icons? Current Systems and Questions SymbolPath RSVP-iconCHAT How You Can Help Why Icons? Bad: Not fully generative Vocabulary requires screen space Letter-based research is often inapplicable Good: Supports limited recall Doesn't require literacy Often faster ...and why is speed so important? On Speed of Communication Typical AAC is < 20 words per minute vs. Speech is often 150 - 200 words per minute Higginbotham, D. J., Shane, H., Russell, S., & Caves, K. (2007). Access to AAC: present, past, and future. AAC: Augmentative & Alternative Communication, 23 (3), 243-257. Current Icon-Based AAC http://www.speakforyourself.org/ This is SpeakForYourself, a modern, icon-based AAC application for Android and iOS. I should note that I'm not affiliated with SpeakForYourself, I just like their application and admire their work. ...one of the problems some users of SpeakForYourself have encountered is accidentally touching multiple buttons. This is a problem often encountered by ANY touch-screen application. To avoid this problem, one of the current solutions is to attach a physical grid to the device... Some Questions Can we handle accidental selections? Upper limb motor impairments (e.g. tremors) Can we handle syntax and word order? Pre-literacy or aphasia from stroke Can we make communication even faster? One Idea: Continuous Motion "QUICK" http://www.swype.com/ SymbolPath: A Prototype "I need more coffee." Wiegand, K., & Patel, R. (2012). SymbolPath: a continuous motion overlay module for icon-based assistive communication. ASSETS '12, (pp. 209-210). New York, NY, USA: ACM. SymbolPath in Action How It Works: Language Model Semantic frames Verbs have "frames" that take certain "roles" Each word can fill certain "roles" Give ( Agent, Object, Beneficiary ) Free-order word statistics (co-occurrence) Display the final sentence choices (for now) Fillmore, C. J. (1976). Frame semantics and the nature of language. Annals of the New York Academy of Sciences, 280 (Origins and Evolution of Language and Speech), 20-32. Wiegand, K., & Patel, R. (2012). Non-Syntactic word prediction for AAC. In Proceedings of the Third Workshop on Speech and Language Processing for Assistive Technologies, (pp. 28-36). Montréal, Canada: Association for Computational Linguistics. Initial User Questions Who are the intended users? Current users of icon-based AAC Especially users with upper-limb motor impairments Meant for integration with other systems Which boards does it have? When can we try it out? What We've Learned So Far Two adults with speech and motor impairments: "It's fun!" Suggested sentences can be amusing (i.e. "wrong") It doesn't actually require touch input: Broad/flat stylus, joysticks, paddles, etc. It doesn't work well for people with spasms Future Addition: "Finish Line" Future Addition: "Finish Area" Other Planned Additions Support for multiple and individualized boards Prediction problem: Statistics from real AAC users provide better prediction Help, please? Let's Dream Together a Little Bit... From the mother of a young AAC user: I want my daughter to be able to say everything. Every single thing that pops into her head -- from "I want water" to "That rhinoceros is purple" to "Yellow pineapple makes my lips itch." My Response "YES!" ...But How? One obvious solution: use letters What if we can't or don't want to? Problems we need to solve: How to display the vocabulary? How to navigate the vocabulary? How to do all of this stuff quickly? Back to the Drawing Board What if we actually knew what just popped into the user's head? Brain-Computer Interfaces (BCI) or Brain-Machine Interfaces (BMI) Often designed assuming a minimal number of input signals Primarily used for people with Locked-In Syndrome (LIS) ...But Also for Novelty http://www.emotiv.com/ http://www.neurosky.com/ On Brain-Computer Interfaces Different signals: Electroencephalography (EEG) Functional Magnetic Resonance Imaging (fMRI) and many others... Different techniques and approaches: Motor imagery Steady state visual evoked potentials (SSVEP) P300 and Event Related Potentials (ERP) and many others... The P300 Wave A Letter-Based Example P300 Speller (or Matrix Speller) http://www.etsu.edu/cas/bcilab/ The P300 is often elicited through event-related potentials (ERP) signals, such as through RSVP, which stands for... Related Project: RSVP-Keyboard Orhan, U., II, K. E. H., Erdogmus, D., Roark, B., Oken, B. & Fried-Oken, M. (2012). RSVP keyboard: An EEG based typing interface.. In ICASSP (pp. 645-648) . IEEE . ISBN: 978-1-4673-0046-9 Another type of RSVP is what some people call "keyhole RSVP," such as used by this project, RSVP-Keyboard. RSVP-Keyboard is a joint project between Professor Deniz Erdogmus at Northeastern and the Center for Spoken Language Understanding at Oregon Health and Science University... RSVP-iconCHAT Wiegand, K., Patel, R., & Erdogmus, D. (2010). Leveraging Semantic Frames and Serial Icon Presentation for Message Construction. ISAAC Conference for Augmentative and Alternative Communication, Barcelona, Spain, July 2010. RSVP-iconCHAT in Action Initial User Questions Who are the intended users? Primarily people with locked-in syndrome (LIS) Users of BCI-controlled wheelchairs Do I have to concentrate? Not exactly, but some users are "better" than others What does it feel like? What We've Learned So Far Experiments with non-AAC users and letter-based RSVP (RSVP-Keyboard) Non-AAC users get frustrated (surprise!) Staying attentive can be tiring Prediction is extremely important for speed and comfort It doesn't work well for people with spasms Future Directions Improve the signal detection and word prediction Begin testing with current AAC users Translate the system and approaches into a portable package http://www.emotiv.com/ Why We Need Help Prediction is extremely important Prediction is primarily based on data More realistic data is better There isn't a lot of real AAC data out there We need to train our algorithms on realistic AAC data How You Can Help Getting ideal data creates privacy concerns Asking current AAC users is a good start We'd like to ask the families, friends, and teachers of real AAC users What do you talk about? How? When? What would you like to talk about? Boards, logs, or word lists would be great! CadLab Corpus Project karlwiegand.com/csun2013/ Survey Demographics karlwiegand.com/csun2013/ The Core of the Survey karlwiegand.com/csun2013/ Thank you for listening! karlwiegand.com/csun2013 Grant #0914808