Eye and hand coarticulation during problem solving reveals hierarchically organized planning

Abstract

During everyday activities, such as when preparing a cup of coffee or traveling across cities, we often plan ahead and execute sequences of actions. However, much remains to be understood about how we plan and coordinate sequences of actions (e.g., eye and hand movements) to solve novel problem-solving tasks. This study investigates how participants use gaze and cursor movements to plan and execute problem-solving tasks, for which solutions are not provided in advance but rather have to be found from scratch. We report three key findings. First, we found that participants segment the problem into sequences of gestures; within each gesture, gaze selects a target and remains fixed until the cursor reaches it, then moves to the next target. Second, we observed coarticulation in both cursor-cursor and gaze-cursor movements, occurring within individual gestures and, to a lesser degree, between successive gestures. Third, we demonstrate for the first time that the initial fixation in a sequence is biased toward the direction of the subsequent fixation. This implies that the angular position of the gaze reliably predicts the next fixation’s direction, suggesting a forward-looking coarticulation between successive gaze fixations. Together, these findings suggest that participants employ a hierarchical planning strategy: they divide the problem into gesture sequences and plan multiple eye and cursor movements in advance to efficiently reach both current and upcoming gesture targets. This hierarchical motor plan demonstrates a structure where targets (or subgoals) are defined and achieved through the coordinated actions of the eyes and hand, highlighting the importance of integrated eye-hand planning in complex task performance.