Pre-Presentation Notes Slides and presentation materials are available online at: karlwiegand.com/thesis Disambiguation of Imprecise User Input Through Intelligent Assistive Communication Karl Wiegand Northeastern University Boston, MA USA June 2013 Thesis Statement "Intelligent interfaces can mitigate the need for linguistically and motorically precise user input to enhance the ease and efficiency of assistive communication." Thesis Strategy "Intelligent interfaces..." User-specific, adaptive, and context-sensitive "...can mitigate the need for linguistically and motorically precise user input..." Demonstrated by algorithms and corpus studies "...to enhance the ease and efficiency of assistive communication." Demonstrated by implementations and user studies Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments SMCR Model of Communication Affected by distortion to any component Intelligent components can mitigate the risks of distortion; trend in HCI What if there is distortion from the Source? Who Uses AAC? Stephen Hawking and Roger Ebert People of all ages People with: cerebral palsy (CP) -- 53% use AAC (Jinks and Sinteff, 1994) amyotrophic lateral sclerosis (ALS) -- 75% use AAC (Ball et al, 2004) brain and spinal cord injuries neurological disorders paralysis, autism, muscular dystrophy, and more... What is AAC? Physical Boards Electronic Systems Letter-Based Icon-Based Current AAC Application SpeakForYourself, an icon-based AAC application for iOS and Android Current AAC Application SpeakForYourself, an icon-based AAC application for iOS and Android Scope and Definitions Target users are primarily non-speaking and may have upper limb motor impairments Target users may also have developing literacy or language impairments "Icon-based AAC" includes systems that use words, icons, or a combination of both Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments Problem Statement Current icon-based AAC systems assume: Syntactic Order Intended Set Discrete Entry Assumption 1: Syntactic Order Users will select icons in the syntactically correct order of the target language. Disambiguate directional utterances Users do not always select icons in syntactic order (Van Balkom and Donker-Gimbrere, 1996) Using AAC devices is slow (Beukelman et al, 1989; Todman, 2000; Higginbotham et al, 2007) Assumption 2: Intended Set Users will select exactly the icons that are desired -- no fewer or more. Complete subsets and prune supersets Motor and cognitive impairments may result in missing or additional selections (Ball, 2004) Letter-based text entry systems detect accidental and deleted selections Assumption 3: Discrete Entry Users will make discrete movements or selections, either physically or with a cursor. Selection is important; path is irrelevant Recent letter-based systems have started to remove this assumption (Goldberg, 1997; Kristensson and Zhai, 2004; Kushler and Marsden, 2008; Rashid and Smith, 2008) Some input methods are naturally continuous (e.g. brain waves, vocalizations) Problem Summary Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments Project: SymbolPath Relaxation of all three major assumptions "I need more coffee." Initial Feedback Two adults and one child with speech and motor impairments: "It's fun!" Suggested sentences can be amusing (i.e. "wrong") and longer than normal It doesn't actually require touch input: Broad/flat stylus, joysticks, paddles, etc. It doesn't work well for people with spasms Future Addition: "Finish Line" Project Goals Functional test-bed for: Free order message construction Completion and correction Continuous motion Faster, less fatiguing communication New input modalities Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments Addressing Syntactic Order Statistical MT (Soricut and Marcu, 2006) Semantic frames, CxG, and PAS (Fillmore, 1976) Give ( Agent, Object, Beneficiary ) WordNet, FrameNet, "Read the Web" Verb-first message construction (Patel et al, 2004) > Free order in SymbolPath (Wiegand and Patel, 2012) Subset completion and superset pruning N-grams; Compansion (McCoy et al, 1998) > Semantic grams (Wiegand and Patel, 2012) "I like to play chess with my brother." Addressing Intended Set Bigrams brother, chess brother, i brother, like brother, play chess, i ... Trigrams brother, chess, i brother, chess, like brother, chess, play brother, i, like brother, i, play ... Set-Completion Example Original Sentence: “Hey, they’re in first, by a game and a half over the Yankees.” Target Stem: game Input Stems: yanke, hey, first, half N1 Candidate List: game, stadium, like, hour, time, year, day, guy, hey, fan, say, one, two, ... S1 Candidate List: game, got, like, red, time, play, team, sox, hour, go, fan, one, get, day, ... Initial Sem-Gram Results Addressing Discrete Entry Physical path or signal characteristics Rotated unistroke recognition (Goldberg, 1997) Letter-based paths (Kristensson and Zhai, 2004; Kushler, 2008) Relative positioning (Rashid, 2008) Merge semantic salience with path attributes > Continuous motion in SymbolPath: Starting and ending locations Movement speed Pauses, stops, and sudden directional changes Outline Communication and AAC Problems to be Addressed Project and Goals Theories and Approaches Implementation and Experiments Proposed Work Corpus Studies "...can mitigate the need for linguistically and motorically precise user input..." Theory Addressability User Studies "...to enhance the ease and efficiency of assistive communication." Practice Usability and applicability > Implementation < Corpus Studies: Overview Venues: ACL, ASSETS, EMNLP, SLPAT Corpora: Blog Authorship Corpus [age, gender, career] Crowdsourced AAC-Like Corpus [standard] Human Speechome Corpus [location, time, role] TalkBank Corpora Evaluation via ranked suggestions and set similarity/differences Proposed Corpus Studies Syntactic reordering: Task: Reorder a shuffled sentence FrameNet vs. N-gram-based permutations Predicting and pruning selections: Tasks: Suggest words to add/remove Sem-grams vs. WordNet+FrameNet vs. tuples Predicting and pruning selections: Location, time of day, and discourse markers User Studies: Overview Venues: ASSETS, CSUN, ISAAC, RESNA Design: Within-subjects to address heterogeneity Current and potential AAC users (12 - 20) Cognitive, speech, and motor assessments Evaluation: Construction speed, length, and error rate Quantification of workload via NASA-TLX Quantification of desirability via Likert scales Proposed User Studies Select vs. draw: Reproduce given utterance (icon set) System 1: Press icons System 2: Draw a line through all icons Prompted response: Describe given picture card System 1: Press icons System 2: Full SymbolPath functionality * Enhanced AAC: Features: Reordering and prediction/pruning Proposed Timeline Thesis (Redux) "Intelligent interfaces can mitigate the need for linguistically and motorically precise user input to enhance the ease and efficiency of assistive communication." Special thanks to the National Science Foundation (Grant #0914808). Thank you for listening! Why Icons? Disadvantages: Not fully generative Vocabulary requires screen space Letter-based research is often inapplicable Advantages: Supports limited recall Doesn't require literacy Often faster (Todman et al, 1994) On Speed of Communication Typical AAC is < 20 words per minute (Higginbotham et al, 2007) vs. Speech is often 150 - 200 words per minute (Beasley and Maki, 1976) Likert Scales Questionnaires w/ Likert items (statements) Suggested scale attributes: Symmetric Equidistant options Odd number of options Usually use 5 options: "strongly disagree" . . . "neither" . . . "strongly agree" Various forms of the same question (5 - 8) NASA's TLX Survey Standardized, researched Likert scales Five, 7-point scales w/ 21 gradations Measure ("very low" to "very high"): Mental Demand Physical Demand Temporal Demand (how rushed were you?) Performance (how successful were you?) Effort Frustration