EarTouch: Facilitating Smartphone Use for Visually Impaired People in Public and Mobile Scenarios

CHI 2019 [DOI]

Interacting with a smartphone using touch and speech output is challenging for visually impaired people in public and mobile scenarios, where only one hand may be available for input (e.g. with the other one holding a cane) and privacy may not be guaranteed when playing speech output using the speakers. We propose EarTouch, an one-handed interaction technique that allows the users to interact with a smartphone touching the ear on the screen. Users hold the smartphone in a talking position and listen to speech output from ear speaker privately. EarTouch also brings us an step closer to the inclusive design for all users who may suffer from situational disabilities. 

Facilitating Temporal Synchronous Target Selection through User Behavior Modeling

IMWUT 2019 [DOI]

Temporal synchronous target selection is an association-free selection technique: users select a target by generating signals (e.g., finger taps and hand claps) in sync with its unique temporal pattern. However, classical pattern set design and input recognition algorithm of such techniques did not leverage users' behavioral information, which limits their robustness to imprecise inputs. In this paper, we improve these two key components by modeling users' interaction behavior. We also tested a novel Bayesian, which achieved higher selection accuracy than the Correlation recognizer when the input sequence is short. The informal evaluation results show that the selection technique can be effectively scaled to different modalities and sensing techniques.

SENTHESIA: Adding Synesthesia Experience to Photos

2018 GIX Innovation Competition Semi-Finalist

An integrated multi-modal solution for adding synesthesia experience to ordinary food photos. Using natural language processing and visual recognition, SENTHESIA learn from the comments and photos from these websites and more materials online, to extract proper and delicate description from the semantic space, and analysis the key elements of taste and sense experience. Visually impaired users can also benefit from vivid descriptions beyond photos. Proposed and designed the concept, now cooperating with an interdisciplinary team to implement prototype. 

Tap-to-Pair: Associating Wireless Devices using Synchronous Tapping

IMWUT 2018 [DOI]

Currently, most wireless devices are associated by selecting the advertiser’s name from a list, which becomes less efficient and often misplaced. We propose a spontaneous device association mechanism that initiates pairing from advertising devices without hardware or firmware modifications. Users can then associate two devices by synchronizing taps on the advertising device with the blinking pattern displayed by the scanning device.  We believe that Tap-to-Pair can unlock more possibilities for impromptu interactions in smart spaces.

Prevent Unintentional Touches on Smartphone

When interacting with smartphones, the holding hand may cause unintentional touches on the screen and disturb the interactions, which is annoying for users. We develop capacitive image processing algorithm to identify the patterns of unintentional touches, such as the process of touches "growing up" and the corresponding relationships among touches on the screen etc. Through mechanical and artificial test, our algorithm rejected 96.33% of unintentional touches while rejecting 1.32% of intentional touches.