Semantic Disambiguation of Non-Syntactic and Continuous Motion Text Entry for Icon-Based AAC Karl Wiegand Northeastern University Boston, MA USA October 21, 2012 Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments What is AAC? Augmentative and Alternative Communication Three major categories: Assisted communication Physical boards with letters, words, or images Electronic devices with integrated Text-to-Speech (TTS) systems ...intended for anyone for whom spoken communication is extremely difficult or infeasible. Who uses AAC? People of all ages People with: cerebral palsy (CP) amyotrophic lateral sclerosis (ALS) brain and spinal cord injuries neurological disorders (e.g. aphasia) muscular dystrophy paralysis, autism, and more... Used by people with a wide variety of conditions, such as CP or ALS, or neurological disorders, such as aphasia from stroke. Current AAC System SpeakForYourself, an icon-based AAC application for iOS and Android Scope and Definitions Target users are primarily non-speaking and may have upper limb motor impairments Target users may also have language impairments (e.g. aphasia) "Icon-based AAC" includes systems that use words, icons, or a combination of both "Non-syntactic" is non-standard syntax or inconsistent syntax Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments Problem Statement Current icon-based AAC systems assume: Syntactic Order Intended Set Discrete Entry Assumption 1: Syntactic Order Users will select icons in the syntactically correct order of the target language. Without syntactic order, how do we handle directional utterances? (near vs. like) Users do not always select icons in syntactic order (Van Balkom and Donker-Gimbrere, 1996) Using AAC devices is slow (Todman, 2000; Wolpaw et al, 2002; Muller and Blankertz, 2006) Assumption 2: Intended Set Users will select exactly the icons that are desired -- no fewer or more. Without this, how do we complete subsets (predict) or prune supersets (correct)? Motor impairments and tremors may result in missing or additional selections (Ball, 2005) Letter-based text entry systems detect accidental and deleted selections Assumption 3: Discrete Entry Users will make discrete movements or selections, either physically or with a cursor. Selection is important; path is irrelevant Recent letter-based systems have started to remove this assumption (Goldberg, 1997; Kushler and Marsden, 2008; Rashid and Smith, 2008) Removing this assumption enables the use of continuous input signals Thesis Statement These three assumptions are problematic and burdensome to users. Algorithms and design approaches can mitigate the need for these design constraints. Alleviating these constraints can: Result in faster, less fatiguing communication Enable the use of new input modalities Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments Project: SymbolPath Relaxation of all three major assumptions "I need more coffee." Project: RSVP-iconCHAT Continuous input signal (BCI) and non-syntactic message construction An icon-based AAC system controlled by a brain-computer interface. Goals For current AAC: Completion and correction Continuous motion For future AAC: Faster communication New input modalities Vowel sounds Electromyographic responses (EMG) Brain-computer interfaces (BCI) Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments Addressing Syntactic Order Semantic frames (Fillmore, 1976) Verbs have a "frame" with semantic roles: Give ( Agent, Object, Beneficiary ) WordNet, FrameNet, "Read the Web" Verb-first message construction (Patel et al, 2004) Any order in RSVP-iconCHAT Emphasize that last bullet is CURRENT WORK; all else is prior work. Addressing Intended Set Subset completion and superset pruning N-grams; Compansion (McCoy et al, 1998) Semantic grams (Wiegand and Patel, 2012) "I like to play chess with my brother." brother, chess brother, i brother, like brother, play chess, i ... brother, chess, i brother, chess, like brother, chess, play chess, i, like chess, i, play ... Semantic grams is CURRENT WORK. Addressing Discrete Entry Physical path or signal characteristics Letter-based continuous motion (Goldberg, 1997; Kushler, 2008) Relative positioning vs. exact locations (Rashid, 2008) Merge semantic salience with path attributes SymbolPath considers: Starting and ending locations Movement speed Pauses, stops, or sudden changes in direction Jitter and tremor SymbolPath is CURRENT WORK uses a linear mixture model that incorporates information about... Outline Background on AAC Problem Statement and Thesis Projects and Goals Theories and Approaches Implementation and Experiments Proposed Corpus Experiments Semantic roles: Sem-grams vs. WordNet & FrameNet vs. tuples (left words, verb, right words) Contextual cues: Location, time of day, discourse markers Syntactic reordering: FrameNet vs. N-gram-based permutations This is all FUTURE WORK that I plan to do as part of my dissertation. Three potential corpora: Crowdsourced AAC-like corpus, Enron emails, and transcripts from The Boston Home. Proposed User Experiments RSVP-iconCHAT: Create a sentence Describe a picture scene Clinical trial with regular feedback SymbolPath: Type vs. draw Respond to a prompt App Store release and feedback This is all FUTURE WORK. Special thanks to my advisor, Dr. Rupal Patel, and the National Science Foundation (Grant #0914808). Thank you for listening!