Abstract
Virtual reality (VR) has garnered much interest as a training environment for motor skill acquisition, including for neurological rehabilitation of upper extremities. While the focus has been on gross upper limb motion, VR applications that involve reaching for, and interacting with, virtual objects are growing. The absence of true haptics in VR when it comes to hand-object interactions raises a fundamentally important question: can haptic-free immersive virtual environments (hf-VEs) support naturalistic coordination of reach-to-grasp movements? This issue has been grossly understudied, and yet is of significant importance in the development and application of VR across a number of sectors. In a previous study (Furmanek et al. 2019), we reported that reach-to-grasp movements are similarly coordinated in both the physical environment (PE) and hf-VE. The most noteworthy difference was that the closure phase—which begins at maximum aperture and lasts through the end of the movement—was longer in hf-VE than in PE, suggesting that different control laws might govern the initiation of closure between the two environments. To do so, we reanalyzed data from Furmanek et al. (2019), in which the participants reached to grasp three differently sized physical objects, and matching 3D virtual object renderings, placed at three different locations. Our analysis revealed two key findings pertaining to the initiation of closure in PE and hf-VE. First, the respective control laws governing the initiation of aperture closure in PE and hf-VE both included state estimates of transport velocity and acceleration, supporting a general unified control scheme for implementing reach-to-grasp across physical and virtual environments. Second, aperture was less informative to the control law in hf-VE. We suggest that the latter was likely because transport velocity at closure onset and aperture at closure onset were less independent in hf-VE than in PE, ultimately resulting in aperture at closure onset having a weaker influence on the initiation of closure. In this way, the excess time and muscular effort needed to actively bring the fingers to a stop at the interface of a virtual object was factored into the control law governing the initiation of closure in hf-VE. Crucially, this control law remained applicable, albeit with different weights in hf-VE, despite the absence of terminal haptic feedback and potential perceptual differences.
Introduction
Virtual reality (VR) has garnered much interest as a training environment for motor skill acquisition, including for neurological rehabilitation of upper extremities (Schultheis and Rizzo 2001; Sveistrup 2004; Holden 2005; Rizzo and Kim 2005; Rose et al. 2005; Adamovich et al. 2009; Cheung et al. 2014). The ability to reach-to and grasp virtual objects is paramount to any VR-based manual training. Devices that provide haptic feedback in VR tend to be expensive, bulky, and restrict natural motion in the virtual environment (Borst and Volz 2005; Pacchierotti et al. 2017; Culbertson et al. 2018). Haptic-free immersive virtual environments (hf-VEs) are a more viable option. However, the extent to which hf-VEs support naturalistic coordination of reach-to-grasp movements is mostly unknown. To fill this gap, in a previous study (Furmanek et al. 2019), we asked participants to reach to grasp physical objects of different sizes located at different distances, and to grasp virtual renderings of these objects, in an immersive hf-VE. We found that reach-to-grasp movements are similarly coordinated in both the physical environment (PE) and immersive hf-VE. The most noteworthy difference was that the closure—which begins at maximum aperture and lasts through the end of the movement—was longer in hf-VE than in PE. Understanding the factors underlying this difference is essential to modify a given hf-VE so that it provides for more naturalistic reach-to-grasp movements.
Much attention has been given to the processes that govern/trigger the initiation of closure during reach-to-grasp movements. Based on the observation that maximum aperture occurs approximately when hand transport deceleration is maximum, it was initially suggested that the initiation of closure is triggered by the reduction of hand transport velocity (Jeannerod 1984; Paulignan et al. 1991a, b). This suggestion was expanded by Rand and colleagues (2006a, b, 2008), who proposed that the initiation of closure is governed by a control law describing the dependence of the control action on the dynamics of the arm and hand in relation to the object location and size. According to this view, the distance of the hand from the object at closure is predicted by a linear function combining the state estimates of transport velocity, transport acceleration, and aperture. This control law has been shown to adequately describe the initiation of closure across different movement speeds (Rand et al. 2006b), object distances (Rand et al. 2010), and disease states, such as Parkinson’s disease (Rand et al. 2006a).
To our knowledge, a comparison of the control laws governing the initiation of closure in PE and hf-VEs has not been made before despite reasons to believe that altered perceptual information and/or task constraints in hf-VEs might affect the planning, execution, and control of manual actions (Harris et al. 2019). For example, egocentric depth cues in VE may be processed differently by brain circuits than the natural depth cues in the real world, contributing to uncertainty in perceptual estimates of the object size and distance. Specific to hf-VEs, an absence of terminal haptic feedback may increase uncertainty in perception of the object size as haptics cannot be used to calibrate grasp (Bingham et al. 2007; Fukui and Inui 2013; Renner et al. 2013). Furthermore, the control law governing the initiation of closure in hf-VE might also differ because in PE, the closure is stopped instantaneously as soon as the fingers contact a physical object, but when reaching to grasp a virtual object, voluntary effort must be exerted to arrest finger movement to prevent interpenetration into that object (Prachyabrued and Borst 2012). Knowledge about whether lack of, or uncertainty about, key perceptual information in hf-VE compared to PE changes the control law used to govern closure is crucial as we aim to improve the fidelity of VR applications for manual skills training and rehabilitation.
In the present study, we reanalyzed data from Furmanek et al. (2019) to investigate whether the same control law governs the initiation of closure for reach-to-grasp movements performed in PE and our immersive hf-VE. We hypothesized that slower closure initiated farther from the object in the hf-VE than the PE condition that was noted by Furmanek et al. (2019) resulted from a difference in the control law governing the initiation of closure between the two environments.
To investigate the source of variation in the control low governing the initiation of closure in PE and hf-VE, we followed an information-theoretic approach to model selection (Burnham and Anderson 2002). This approach chooses the best model from a given set of a priori candidate models—by comparing the models against each other, instead of the null hypothesis, and balances the goodness of fit with simplicity. We considered the null model (containing only the intercept) and all the different combinations of the state estimates (aperture, transport velocity, and transport acceleration) to test whether the control law relies on each of these estimates in both PE and hf-VE and whether the strength of reliance on each of these estimates differs between the two environments.
Methods
Participants
Fifteen right-handed subjects (13 male, 12 female; mean±1SD age: 23.9±6.8 years old) with no reported muscular, orthopedic, or neurological health concerns, voluntarily participated in Furmanek et al. (2019) after providing verbal and written consent. Data from these participants were reanalyzed. The experimental task and procedure are described in brief below. Further details about the experimental task and procedure can be found in Furmanek et al. (2019).
Experimental task
The participants performed the task of reaching toward and grasping three differently-sized physical objects and matching 3D virtual renderings. The objects were cuboids of identical height (8 cm) and depth (2.5 cm) but different widths: small: 3.6 cm; medium: 5.4 cm; large: 7.2 cm (Fig. 1). The physical objects were 3D printed using PLA thermoplastic (mass: small: 30 g; medium: 44 g; large: 59 g) and covered with glow-in-the-dark paint. The participants viewed blue-colored virtual renderings of these objects in a custom 3D immersive hf-VE designed in UNITY (ver. 5.6.1f1, Unity Technologies, San Francisco, CA) and presented via head-mounted display (Oculus Rift DK2, Oculus Inc., Menlo Park, CA). The objects, in PE or hf-VE, were placed at three different distances from the starting position—24 cm (near), 30 cm (middle), 36 cm (far)—and oriented at a 65° angle along the vertical axis to minimize excessive wrist extension (Fig. 1).
Experimental procedure
The participants were seated in a chair with the right arm and hand placed on a table in front of them (Fig. 1). At the start position, the thumb and index finger straddled a 1.5 cm wide wooden peg located 12 cm in front and 24 cm to the right of the sternum, with the thumb depressing a switch. Lifting the thumb off the switch marked movement onset. A six-camera motion tracking system (PPT Studio N™, WorldViz, Inc., Santa Barbara, CA; sampling rate: 75 Hz) recorded the 3D motion of IRED markers attached to the participant’s wrist (at the center of the segment running between the ulnar and radial styloid process), thumb nail, and index fingernail. A pair of IRED markers attached to the HMD co-registered the participant’s head motion to render the virtual environment. When reaching to grasp a virtual object in hf-VE, the participants viewed the tips of their thumb and index finger as two 3D spheres (green in color, 0.8 cm diameter), reflecting the real-time 3D position of the respective IRED marker. When reaching to grasp a physical object, all lights were turned off so that only the illuminated IRED markers attached to the thumb and index finger, as well as the glow-in-the-dark object, were visible to the participants. The overhead light was turned on after every block of 12 trials to prevent acclimatization of the vision to the dark environment.
Prior to the experiment, the participants were allowed to practice the reach-to-grasp task in both PE and hf-VE until they felt comfortable in both environments. The experiment consisted of 216 trials, each trial lasting 3 seconds: 108 trials in PE and 108 trials in hf-VE (3 objects × 3 distances/object × 12 trials/ or each size–distance combination in each environment). The trials were blocked for each size-distance combination and each environment. The order of the presentation of the three objects and their placement was randomized for each block of size-distance combination.
In each trial, an auditory cue—a beep—signaled the participants to reach for, grasp, and lift the object. The participants were considered to have grasped a physical object the moment the tips of their thumb and index finger came into contact with that object’s lateral surface, and aperture did not change more than 1 mm for at least 50 ms. The participants were considered to have grasped a virtual object the instance a custom collision detection algorithm detected that both 3D spheres had come in contact with the virtual object. Once grasp of the virtual object was detected, the object changed color from blue to red. The participants lifted and raised each object briefly before returning their hand to the starting position.
Data processing and feature extraction
All data were analyzed offline using custom MATLAB routines (Mathworks Inc., Natick, MA). For each trial, the time series data were cropped from movement onset (and the moment the switch was released) to movement offset (the moment aperture velocity became zero in PE, and the moment the collision detection criterion was met in hf-VE) and filtered (6 Hz, fourth-order low-pass Butterworth filter).
For each condition (environment, object size, and object distance), mean values were calculated on a per-participant basis for the following outcome variables: (a) Movement time, defined as the interval between movement onset and movement offset; (b) Maximum aperture, which marked the initiation of closure or closure onset (CO) and which we refer to as aperture at CO; (c) Opening time, defined as the interval from movement onset to CO; (d) Closure time, defined as the interval from CO to movement offset; (e) Transport distance, defined as the distance between the initial and final positions of the wrist; (f) Opening distance, defined as the distance between the wrist’s position at movement onset and the object; (g) Closure distance, defined as the distance between the wrist’s position at CO and the object; (h) Maximum closure velocity; (i) Maximum closure deceleration; (j) Transport velocity at CO; and (k) Transport acceleration at CO.
Statistical analysis
A 2 × 3 × 3 repeated measures analysis of variance (rm-ANOVA) with within-subject factors of the Environment (PE, hf-VE), Object size (Small, Medium, Large), and Object distance (levels: Near, Middle, Far) was used to evaluate the effects of the Environment, Object size, and Object distance on each kinematic variable. All data mostly fulfilled the conditions of normality, homogeneity of variance, and sphericity; when the assumption of sphericity was not met, Greenhouse-Geisser correction was applied, and in these cases, the corrected p-values and the Greenhouse-Geisser correction factor (ε) are reported. Pearson’s correlation tests were used to examine the relationship between opening/closure time and movement time in both PE and hf-VE, as well as between opening/closure distance and transport distance in both PE and hf-VE. Each test was performed in SPSS 21 (IBM Inc., Chicago, IL), and each test statistic was considered significant at the two-tailed alpha level of 0.05. All effect sizes are reported as partial eta-squared (ηp2).
To investigate which state estimates best explained variation in closure distance in PE and hf-VE, we followed an information-theoretic approach to model selection using RStudio. This approach uses the Akaike Information Criterion (AIC; or quasi-AIC (QAICc) for over-dispersed data to choose a set of plausible models from a given set of a priori candidate models (Burnham and Anderson 2002). AICc for a given model is computed as: , where is the likelihood of the parameters in xthe model and K = number of estimable parameters in the model. For example, a simple linear regression model, y = a + bx, has three estimable parameters: in intercept a, the slope b, and the variance (s2). This definition of AIC is applicable for large samples (n). In the small-sample case (i.e., n/K < 40), . To compute QAICc, the number of model parameters is increased by 1 to account for overdispersion.
QAICc serves as an estimator of out-of-sample prediction error and thereby the relative quality of statistical models for a given set of data. QAICc estimates the quality of each model relative to each of the other models. Specifically, a smaller QAICc value reflects better performance/complexity trade-off. Thus, QAICc provides a means for selecting the model with the best performance/complexity trade-off. In the present study, we considered eight candidate models, including the null model (containing only the intercept) and all the subset models of the following full model: CD = a + b(ACO) + c(TVCO) + d(TACO), by allowing the intercept to differ by participant, where CD = closure distance, ACO = aperture at CO, TVCO = transport velocity at CO, and TACO = transport acceleration at CO.
Results
Figure 2 provides spatial trajectories of the reach (left) and grasp (right) components of the reach-to-grasp movements performed in PE and hf-VE for a representative object size-distance pair (medium object, middle distance). Table 1 summarizes the mean (±1SD) values of each movement variable across all participants (n = 13). Table 2 summarizes the outcomes of rm-ANOVAs. Test statistics only for significant effects are presented below (see Table 2 for non-significant outcomes.
Closure lasted longer and was initiated farther from the object, with lower peak deceleration, in hf-VE than in PE
Closure lasted longer in hf-VE than in PE (rm-ANOVA: F1,12 = 19.24, p = 0.001, ηp2 = 0.62), and this effect of environment was mediated by object size (F2,12 = 7.69, p = 0.003, ηp2 = 0.39; Fig. 3A). Closure was initiated farther from the object in hf-VE than in PE (F1,12 = 15.63, p = 0.002, ηp2 = 0.57), an effect also mediated by object size (F2,12 = 6.77, p = 0.005, ηp2 = 0.36; Fig. 3B). The maximum closure velocity did not differ between the two environments (F1,12 = 1.77, p = 0.208, ηp2 = 0.13; Fig. 3C), but maximum closure deceleration was higher in hf-VE than in PE (F1,12 = 8.55, p = 0.015, ηp2 = 0.42; Fig. 3D). Thus, closure lasted longer and was initiated farther from the object, with lower peak deceleration, in hf-VE than in PE.
Closure scaled in time but not distance from the object in both PE and hf-VE
To examine whether the initiation of closure occurred at a consistent time relative to movement onset or grasping of the object in PE and hf-VE, we analyzed the relationship between opening/closing time and total movement time. Opening time showed a strong positive relationship with movement time in both PE (Pearson’s correlation, ρ = 0.85, p < 0.001; Fig. 4A) and hf-VE (ρ = 0.76, p < 0.001; Fig. 4B), and closing time showed a weak positive relationship with movement time in both PE (ρ = 0.50, p < 0.001; Fig. 4A) and hf-VE (ρ = 0.43, p < 0.001; Fig. 4B). The slopes of the regression lines for both opening time (0.72 in PE vs. 0.71 in hf-VE) and closing time (0.27 in PE vs. 0.29 in hf-VE) were similar between the two environments.
To examine whether the initiation of closure occurred at a consistent distance relative to the initial or final transport position in PE and hf-VE, we analyzed the relationship between opening/closing distance and total transport distance. Opening distance showed a strong positive relationship with transport distance in both PE (ρ = 0.98, p < 0.001; Fig. 4C) and hf-VE (ρ = 0.91, p < 0.001; Fig. 4D). In contrast, closing distance did not show a significant relationship with transport distance in PE (ρ = 0.11, p = 0.505; Fig. 4C) or hf-VE (ρ = 0.14, p = 0.306; Fig. 4D). The slopes of the regression lines for both opening distance (0.98 in PE vs. 0.94 in hf-VE) and closing distance (0.024 in PE vs. 0.060 in hf-VE) were similar between the two environments.
Similar control laws governed the initiation of closure in PE and hf-VE
Of the eight models used to identify the control law governing the initiation of closure in PE, only two models yielded nonzero weights, wi, (the probability that a given model is the best model among all models). Support for the first model was 999 times stronger than the second model (evidence ratio = w1/w2 = 0.9900/0.0010 = 999; χ2 = 16.66, p < 0.001, Table 3). The first model yielded both the best fit (log-likelihood closest to zero) and the best performance-to-complexity tradeoff (lowest QAICc). This model included a coefficient for ACO, TVCO, and TACO, supporting the idea presented by Rand and coworkers (Rand et al. 2006a, b) that the distance at closure is predicted by a linear function combining the state estimates of transport velocity, transport acceleration, and aperture.
Of the eight models used to identify the control law governing the initiation of closure in hf-VE, only two models yielded non-negligible wi. Support for the first model was 3.2 times stronger than the second model (evidence ratio = w1/w2 = 0.73/0.27 = 2.50; Table 3). While the model with the best fit (model 2, log-likelihood closest to zero) included a coefficient for all three variables, the model with the best performance-to-complexity tradeoff (model 1, lowest QAICc) did not include a coefficient for aperture at CO. Nonetheless, the first and second models yielded close goodness of fit (log-likelihood values: –85.05 and –84.86, respectively) and the inclusion of ACO in the model did not significantly improve the fit (χ2 = 0.38, p = 0.537). In short, the initiation of closure in hf-VE was not as dependent on aperture at CO as it did in PE.
To further explore the contribution of different state variables to the respective control laws, we analyzed the relationship between the values of closure distance and each state variable averaged across all participants. Closure distance showed a negative relationship with aperture at CO in both environments (PE: Pearson’s r = –0.78, p = 0.013; hf-VE: r = – 0.81, p = 0.008; Fig. 5A, left), a strong positive relationship with transport velocity at CO (PE: r = 0.94, p < 0.001; hf-VE: r = 0.98, p < 0.001; Fig. 5B, left), and a weak negative relationship with transport acceleration at CO (PE: r = –0.60, p = 0.088; hf-VE: r = –0.64, p = 0.067; Fig. 5C, left). The strong relationship between closure distance and transport velocity at CO in both PE and hf-VE indicates that it is the most important state variable contributing to the control laws. This result confirms that of the model selection in that the model with the best fit and performance-to-complexity tradeoff for PE, and all four models with nonzero weight, wi, for hf-VE, included a coefficient for transport velocity at CO (Table 3).
To substantiate that the state estimate of aperture contributed to the variability in the control law governing the initiation of closure between PE and hf-VE, we compared the relationship between each state variable and closure distance between PE and hf-VE. In both PE and hf-VE, closure distance showed a strong negative relationship with aperture at CO (PE: Pearson’s r = -0.78, p = 0.013; hf-VE: r = –0.81, p = 0.008; Fig. 5A, left) and a very strong positive relationship with transport velocity at CO (PE: r = 0.94, p < 0.001; hf-VE: r = 0.98, p < 0.029; Fig. 5B, left) but did not show a significant relationship with transport acceleration at CO (PE: r = –0.060, p = 0.088; hf-VE: r = –0.064, p = 0.067; Fig. 5C, left). At the individual participant level, the slope of the regression line between closure distance and aperture at CO was steeper in hf-VE than in PE (paired t-test: t2,13 = 4.40, p = 0.029; Fig. 5A, right). Crucially, the slope of the regression line did not differ for transport velocity at CO (t2,13 = –1.90, p = 0.082; Fig. 5B, right) and transport acceleration at CO (t2,13 = 1.81, p = 0.095; Fig. 5C, right).
Finally, to test whether a difference in the relationship between aperture at CO and transport velocity at CO might be a possible reason for why the initiation of closure in hf-VE did not depend on aperture at CO as it did in PE, we compared relationships among the three state parameters between PE and hf-VE. In both PE and hf-VE, aperture at CO showed a strong relationship with transport velocity at CO (PE: Pearson’s r = -0.72, p = 0.023; hf-VE: r = –0.88, p = 0.002; Fig. 6A, left) but no significant relationship with transport acceleration at CO (PE: r = 0.38, p = 0.309; hf-VE: r = 0.21, p = 0.594; Fig. 6B, left). Transport velocity at CO showed a strong negative relationship with transport acceleration at CO in PE (r = –0.72, p = 0.029) but no significant relationship in hf-VE (r = –0.62, p = 0.072; Fig. 6C, left). At the individual participant level, the slope of the regression line between aperture at CO and transport velocity at CO was steeper in hf-VE than in PE (paired t-test: t2,13 = 4.47, p < 0.001; Fig. 6A, right). Crucially, the slope of the regression line between aperture at CO and transport acceleration at CO did not differ between PE and hf-VE (t2,13 = –0.11, p = 0.917; Fig. 6B, right), and the slope of the regression line between transport velocity at CO and transport acceleration at CO was only marginally steeper in hf-VE than in PE (t2,13 = 2.21, p = 0.047; Fig. 6C, right).
Discussion
In the present study, reanalysis of the Furmanek et al. (2019) data indicated that the closure phase was also initiated farther from the object, with smaller peak deceleration, in immersive hf-VE than in PE. These differences presumably reflect the fact that when reaching to grasp in hf-VE, the voluntary effort needed to actively bring the fingers to a stop at the interface of a virtual object. PE does not impose this physical constraint, as the fingers automatically stop when they contact the physical object. However, the goal of this study was not merely to identify differences in the initiation of closure between the two environments; the goal of the present study was to investigate potential differences and similarities in the control laws governing the initiation of closure in PE and an immersive hf-VE. To do so, a linear regression model—previously suggested by Rand et al. (2008) to predict the distance from the object at which the closure is initiated from the object from the limb state parameters of transport velocity, transport acceleration, and peak aperture—was applied to kinematic data from each environment. The model comparisons yielded two important insights: (1) Transport velocity was the dominant predictor of closure distance in both environments, and the relationship between transport velocity at CO and closure distance was nearly identical between environments, and (2) Aperture was less informative to the control law in hf-VE.
In hf-VE, two factors significantly influence the speed-accuracy tradeoff in reach-to-grasp movements. The first is the well-documented finding that VR increases the perceptual uncertainty about object distance and size (Armbrüster et al. 2008; Ogawa et al. 2018; Harris et al. 2019). The increase in perceptual uncertainty about the spatial properties of objects, which is hypothesized to cause slower movements in hf-VE (Viau et al. 2004; Magdalon et al. 2011), was likely a major factor in smaller peak transport velocity found in the present study. The second factor influencing the speed-accuracy tradeoff is the precision demand of grasping. The lack of physical properties of the object in hf-VE not only requires the voluntary effort needed to actively bring the fingers to a stop at the interface of a virtual object, but also increases the demand for precision of the finger aperture in the closure phase of the movement.
Investigations of the speed-accuracy tradeoff for reach to grasp have shown that the full reach-to-grasp movement is poorly described by the Fitts law (Bootsma et al. 1994). Instead, it may be better characterized by Welford’s (1968) theoretical position that object distance and size place separate constraints upon movement time (McIntosh et al. 2018). However, Rand et al. (2008) have suggested that the precision of the closure phase specifically may be reasonably well described by principles of speed-accuracy tradeoff as described by the standard form of the Fitts’ law (Fitts 1954). Based on the finding that controlled manipulations of movement speed did not alter the relationship between transport velocity at CO and closure distance, Rand et al. (2006b) reasoned that variability in the opening process might be more acceptable as long as the transport-aperture coordination at the time of the phase transition to closure is tightly controlled. Our data support this assertion by Rand for movements conducted in hf-VE with one notable exception. Rand et al. (2006b) found faster movements (i.e., movements with greater peak transport velocity) were associated with higher transport velocity at CO, and the closure initiated farther from the object. In contrast, we observed that the movements performed in hf-VE, despite having significantly lower peak transport velocity than those performed in PE (Furmanek et al. 2019), were characterized by higher transport velocity at CO and closure distance farther from the object. Indeed, this finding appears to be consistent with previous empirical results suggesting that precision demands of grasping are associated with an extended deceleration phase, suggesting an important role for feedback-based correction (MacKenzie et al. 1987; Bootsma et al. 1994; MacKenzie and Graham 1997). We suggest that a parsimonious explanation for this difference between our result and that of Rand et al. (2006b) is that movement plan at the time of movement initiation factored the specific task and precision demands of the closure in hf-VE to extend the relationship between transport velocity at CO and closure distance learned over a lifetime of practice in PE. The participants treated the virtual object as if it were a physical object with greater precision demands for grasping (Gentilucci et al. 1991; Iyengar et al. 2009; Magdalon et al. 2011). The striking similarity of the relationship between transport velocity at CO and closure distance offers valuable insights into the critical importance of preserving this aspect of the phase transition between the opening and closing of the grasp for successful reach to grasp. We suggest that the preservation of this relationship may be a critical benchmark for any VE in which reach-to-grasp tasks are performed.
Our second insight was that aperture was less informative to the control law in hf-VE. This, too, appears to relate directly to uncertainty about spatial properties of the object and precision demands of the task. It has been shown that calibration of grasp size with reach distance allows executing naturalistic reach-to-grasp movements even without haptic terminal feedback (e.g., Bingham et al. 2007). A lack of any opportunity to calibrate grasp size with reach distance in the present study might have led to uncertainty about object size in hf-VE. Indeed, the finding that closure distance scaled differently with object size in PE and hf-VE speaks directly to the perceptual uncertainty about object size in hf-VE. Furthermore, aperture at CO was less independent in hf-VE than in PE, ultimately resulting in the aperture at CO having a weaker influence on the initiation of closure. Indeed, the strong negative relationship between transport velocity and aperture at CO in hf-VE (Fig. 6A, left) speaks to the deliberate slowing of hand transport in situations with higher precision demand, as expected of hf-VE.
The present findings, along with those of Furmanek et al. (2019), show that immersive hf-VE can partially support naturalistic reach-to-grasp movements. These results are especially relevant to the study of the role that terminal feedback may have in naturalistic reach-to-grasp movements and to reproducing naturalistic reach-to-grasp movements in immersive hf-VEs. When reaching to grasp a physical object, one of the important roles of terminal haptic feedback is to convey, via somatosensory signals, that the object has been grasped successfully. This feedback may benefit subsequent actions by updating and recalibrating forward models controlling reach-to-grasp movements so that the grasp size and reach distance can be scaled appropriately (Coats et al. 2008; Cavina-Pratesi and Hesse 2013). Notably, in the present study, when a collision detection algorithm detected that both 3D spheres (denoting the 3D position of the fingertips in hf-VE) had come into contact with the virtual object, the object’s color changed from blue to red to provide [terminal] visual feedback that the object had been grasped. This visual feedback could presumably substitute for the haptic feedback available upon grasping a physical object, and thus the reach-to-grasp movements were coordinated similarly in both PE and hf-VE (Prachyabrued and Borst 2014; Geiger et al. 2018). Our future work will investigate how bio-inspired collision detection algorithms can help thinning the remaining gap between prehensile actions performed in PE and immersive hf-VE.
To conclude, the present findings addresses the concern that the planning, execution, and control of manual actions in hf-VEs than in PE may involve distinct mechanisms (Harris et al. 2019). Such a distinction implies that VR might have certain limitations in its utility for training and rehabilitation. However, we show that albeit with different weights in hf-VE, a control law that involves sensorimotor integration of the internal state estimates of the velocity and acceleration of the arm and perceptual estimate object size dictates the initiation of closure in both PE and hf-VE. This finding suggests that haptic-free immersive virtual environments support naturalistic coordination of reach-to-grasp movements and can be a viable option for low-cost home-based personalized training and rehabilitation, as well as for behavioral and neurophysiological investigations of reach-to-grasp movements and it’s underlying control mechanisms.
Author contributions
M.M., M.Y., M.P.F., and G.T. conceived and designed research; M.P.F. performed experiments; M.M., M.P.F., and M.P.F. analyzed data; M.M., M.Y., M.P.F., and G.T. interpreted results of experiments; M.M. prepared figures; M.M. drafted manuscript; M.M., M.Y., M.P.F., and G.T. edited and revised manuscript; M.M., M.Y., M.P.F., and G.T. approved final version of manuscript.
Conflict of interest
The authors declare that no competing interests exist.
Acknowledgments
This work was supported by NIH grants #R01NS085122 and #2R01HD058301, and NSF grants #CBET-1804550 and #CMMI-M3X-1935337, to Eugene Tunik. We thank Alex Hunton and Samuel Berin for developing the VR platform.