HCC-Large: Using the Internet Without Using the Eyes: Models of Online Transactions for Non-Visual Interaction

The Internet has become the primary medium for accessing information and for conducting many types of online transactions, including shopping, paying bills, making travel plans, applying for college or employment, and participating in civic activities. The primary mode of interaction over the Internet is via graphical browsers designed for visual navigation. This seriously limits the access of people with impaired vision or blindness, a population that is large and growing ever larger. Existing assistive technology for non-visual Internet access typically forces users with visual impairments into an inefficient, sequential mode of information access. To do better, two kinds of models are needed. First, we need to build computational models to represent the structure of web pages and online transactions, and to present them effectively using non-visual modalities. Second, we need to better understand how users' mental models for online transactions are built and utilized; we then need to align the computational models with the users' mental models, so as to combine their strengths and significantly improve the efficiency of non-visual interactions. In previous work, the PI developed the HearSay non-visual web browser, which permits users to perform basic non-visual web browsing and search, contextual browsing, and online form-filling. However, HearSay does not take full advantage of the interaction context or the unique perceptual and processing strengths of people with visual impairments. In the current project, the PI seeks to combine basic computational and psychological research designed to produce accessibility technology embodying the synergy of computational modeling and users' mental models. In terms of computational research, the PI will: (i) automatically track the interaction context of user browsing actions; (ii) automatically build models for transactions that users perform online; and (iii) develop ways in which users can interact with transaction models through non-visual modalities efficiently and effectively. In terms of psychological research, user studies will be conducted to examine (i) how people build mental models for online transactions, and (ii) how they use modality-specific cues and their own short-term memory to utilize these mental models. The PI will incorporate the findings from these user studies into the computational models for online transaction processing, so as to align them with the users' mental models. Broader Impacts: The ultimate goal of the PI's research is to empower people with visual impairments to lead completely independent lives with the help of the Internet. To this end, the PI has planned an extensive dissemination campaign involving workshops, collaborations with institutions that serve people who have visual impairments, and online dissemination of HearSay prototypes and HearSay component technologies. HearSay will also provide a means, in principle, for anyone who wishes to have non-visual Internet access (e.g., listening to Internet content while driving).