Impact of Motor Impairment on Full-Screen Touch Interaction karlwiegand.com/csun2015 Karl Wiegand and Rupal Patel Northeastern University (USA) March 6, 2015 1 Outline Touchscreens and Accessibility Study Description Results and Observations Summary and Conclusions 2 Touchscreen Prevalence First appeared in 1960s and 1970s (Johnson; Beck and Stumpe; Hurst) In mobile devices since 1990s (IBM Simon) Popularity exploded in 2007 w/ iPhone Apple sold 1.12+ million iPhones in 2007 Now in almost every mobile device 3 Touchscreen Accessibility Motor impairment from MS, MSA, MD, etc. Can rebind gestures or create new ones Adjust click timing; enable switch control Avoid touch interaction (e.g. Siri or Now) 4 Common Limitations Programmatically prohibited from toggling or adjusting sliding functionality Users are prevented from modifying the location, size, shape, and orientation of many toolbars and buttons Difficult to quantify the effects of these limitations on users and purchasing 5 Motivating Example: Swype Originally conceived as an interface for assistive communication Well-received by non-disabled users 6 Motivating Questions Most mobile devices now have touchscreens Increasing research on accessibility Fitts and Steering Laws [Fitts, 1954; Accot and Zhai, 1996] Swabbing/sliding is easier [Wacharamanotham et al, 2011] Buttons need to be bigger [Chen et al, 2013] What about functional compensation? Can we learn realistic, layout-agnostic interaction patterns for an individual user? 7 Motor Optimization GUI (MoGUI) 8 MoGUI Example 9 Study Overview 2 cross-balanced sessions: taps vs. slides 4x4 grid = 16 locations Pseudo-random shuffling (a la Latin Squares) 10 levels of 3 rounds each 1, 2, 3, ...10 balloons per round = 165 total Track all hits, misses, and timing 10 Participants Residents at the Boston Home Various levels of speech and motor impairments 10 females and 5 males Ages 35 - 71 (mean of 56) 8 right-handed; 7 left-handed (3 due to MS) All were familiar with touchscreen tablets, but only 8 regular users (7 iPads, 1 Kindle) 11 Method 10.1" Android tablet (ASUS Transformer) Comfortable, landscape position s.t. all areas were fully reachable 9 users preferred a 45-degree angle on a table 1 user preferred a lowered table and flat tablet 2 users preferred the tablet in a wheelchair mount 2 users preferred the tablet in their lap 1 user cradled the tablet in one arm Choice of finger or stylus 6 wanted stylus, but needed it positioned 12 Interview Questions Did you find any areas of the screen easier or more difficult to reach than others? Did you prefer tapping, sliding, a combination of both, or neither? Were the balloon targets too big or small? What would you like to see improved in touchscreen tablets? 13 Variability: Multiple Taps (LH) 14 Variability: Finger Dragging (RH) 15 Variability: Hand Resting (RH) 16 Variability: Thumb Usage (RH) 17 Results: Misses by Handedness 18 Left Overlays from all sessions Right Results: Locations by Handedness Left Right Mean speed-to-target in pixels/second 19 Results: Directions by Handedness Mean speed-to-target in pixels/second Left Right 20 Participant Feedback 3 users preferred tapping; 5 preferred sliding; 5 preferred a combination of both; and 2 had no preference 10 users noted that sliding required planning Overall, 8 participants felt that sliding felt "faster" and "easier," but for short distances Randomness prevented motor learning 21 Study Observations Varied tablet and hand/arm positions Tablet being held, flat/tilted on lap, on desk, tilted on table, held in wheelchair mount Needed to disable auto-rotation for 1 user Use of fingers, thumb, stylus, and knuckles Ghost tapping, spastic tapping, stylus friction, and finger humidity Repeated margin activation and triggering of Google Now functionality 22 Summary Sliding not significantly faster than tapping for arbitrary targets; no motor learning 16% accidental slides; 43% accidental taps High variance in individual motor patterns; weak correlations by handedness Gamified calibration Static improvements through personas: Handedness → margins, button locations Tap/slide preferences → input sensitivity 23 Special thanks to the Continuous Path Foundation and the National Science Foundation (Grants #HCC-0914808 and #SBE-0354378). Thank you for listening! karlwiegand.com/csun2015 24