Eye and hand coarticulation during problem solving reveals hierarchically organized planning

Abstract

While the ways in which we plan ahead have been investigated since the inception of cognitive science, much remains to be understood about how we plan and coordinate sequences of actions (e.g., eye and hand movements) necessary for effective problem-solving. This study investigates how participants use gaze and cursor movements to plan and execute problem-solving tasks, revealing three key findings. First, participants segment the problem into sequences of gestures; within each gesture, gaze selects a target and remains fixed until the cursor reaches it, then moves to the next target. Second, we observed coarticulation in both cursorcursor and gaze-cursor movements, occurring within individual gestures and, to a lesser degree, between successive gestures. Third, the angular position of gaze reliably predicts the direction of the next fixation, indicating forward-looking coarticulation between successive gaze fixations. Together, these findings suggest that participants employ a hierarchical planning strategy: they divide the problem into gesture sequences and plan multiple eye and cursor movements in advance to efficiently reach both current and upcoming gesture targets. This hierarchical motor plan demonstrates a structure where targets (or subgoals) are defined and achieved through the coordinated actions of the eyes and hand, highlighting the importance of integrated eye-hand planning in complex task performance.