Complexities of practical Web automation

Web automation, a process of automating browsing actions on behalf of the user, has the potential to bridge the divide between the ways visually-impaired and sighted people access the Web. The proliferation of portable small-screen devices (smartphones, smart watches, etc.) presents both an opportunity and a challenge for the development of a new generation of automated non-visual user interfaces for sighted users. The goal of this paper is to provide guidance for the designers of future user interface automation systems. In this paper, we systematize and analyze the complexities involved in building a practical, usable, and accessible system for web automation. We review the published literature and present lessons learned from the authors' own years of experience in developing novel approaches to web automation for visually-impaired Web users.