LLM-powered Text Entry in Virtual Reality

Large language models (LLMs) have demonstrated exceptional performance across various language-related tasks, offering significant potential for enhancing text entry in Virtual Reality (VR). We introduce an LLM-powered text entry system for VR, which integrates multiple input modalities and utilizes a fine-tuned LLM as a keyboard decoder. The LLM-based decoder achieved 93.1% top-1 decoding accuracy on a word gesture typing dataset and 95.4% on tap typing, highlighting its potential for VR text entry applications. Our demonstration shows how LLMs can support tap typing and word-gesture typing through raycasting and joystick-based inputs, potentially accommodating various user preferences and enhancing the adaptability of VR text input methods.