Robust human-computer interaction system guiding a user by providing feedback

We introduce a human-computer interaction system which collaborates with a user by providing feedback during user activities. The goal of the system is to help a user complete a high-level activity that has been represented hierarchically. While a user is performing the high-level activity, our system analyzes what sub-events a user has already completed and what sub-events are needed next in order for the user to finish the activity. The representations of human activities are constructed using a previously developed context-free grammar based representation scheme. We focus on a game named 'pentagram game' to illustrate our system. In the experiments, our system shows the ability to guide the user to complete the 'pentagram game' by providing explicit feedback. The feedback not only contains atomic level instructions but also describes higher-level long-term goals of the composite activities.