Effect of visual and haptic feedback on grasping movements
Abstract
Perceptual estimates of three-dimensional (3D) properties, such as the distance and depth of an object, are often inaccurate. Given the accuracy and ease with which we pick up objects, it may be expected that perceptual distortions do not affect how the brain processes 3D information for reach-to-grasp movements. Nonetheless, empirical results show that grasping accuracy is reduced when visual feedback of the hand is removed. Here we studied whether specific types of training could correct grasping behavior to perform adequately even when any form of feedback is absent. Using a block design paradigm, we recorded the movement kinematics of subjects grasping virtual objects located at different distances in the absence of visual feedback of the hand and haptic feedback of the object, before and after different training blocks with different feedback combinations (vision of the thumb and vision of thumb and index finger, with and without tactile feedback of the object). In the Pretraining block, we found systematic biases of the terminal hand position, the final grip aperture, and the maximum grip aperture like those reported in perceptual tasks. Importantly, the distance at which the object was presented modulated all these biases. In the Posttraining blocks only the hand position was partially adjusted, but final and maximum grip apertures remained unchanged. These findings show that when visual and haptic feedback are absent systematic distortions of 3D estimates affect reach-to-grasp movements in the same way as they affect perceptual estimates. Most importantly, accuracy cannot be learned, even after extensive training with feedback.
the execution of a well-aimed reach-to-grasp action entails the computation of different object features, such as the position of the object in the environment (extrinsic object properties) and the physical properties of the object in terms of shape and dimension (intrinsic object properties). Among the different visual cues, binocular cues (vergence and binocular disparity) play an important role in specifying this information (Melmoth and Grant 2006; Servos et al. 1992). An accurate estimate of extrinsic features, like object distance, which is mostly specified by vergence and accommodation, is also necessary for the accurate estimate of intrinsic object properties from horizontal binocular disparities (Melmoth et al. 2007; Servos 2000). Thus, taken together, binocular cues can in principle unambiguously specify object shape size and location (Foley 1980; Rogers and Bradshaw 1993; Servos 2000). Nevertheless, systematic distortions in object depth and distance perception have been found in binocular, just as in monocular, conditions (Domini and Caudek 2013; Foster et al. 2011; Servos 2000; Todd et al. 1995). By using perceptual judgments and manual size estimation tasks, studies originally reported biases in distance estimation. Distances in near space were overestimated, whereas distances in far space were underestimated (Bingham and Pagano 1998; Foley 1980; Servos 2000). In addition, the distance at which the objects were presented also affected the perceived object depth, resulting in an overestimation of the perceived depth of objects close to the observer and underestimation of objects further away (Johnston 1991; Volcic et al. 2013).
Systematic biases comparable to those reported in perceptual tasks have also been observed in experiments with action tasks, where subjects were required to perform movements in impoverished sensory conditions, i.e., in the absence of visual feedback of the hand or haptic feedback about the object to be reached at or grasped. In particular, numerous studies reported distortions in distance estimation when measured by reaching actions (Bingham et al. 2000, 2001, 2007; Bingham and Pagano 1998). The few studies testing grasping actions also revealed systematic biases in object depth estimates when feedback was absent (Bingham et al. 2007; Campagnoli et al. 2012; Foster et al. 2011).
What these studies suggest is that the potential information available before the start of a reach-to-grasp action does not necessarily yield an accurate estimate of object distance and size. It is evident, therefore, that visual and haptic feedback, which are always present during natural grasping movements, play a fundamental role in the accurate performance of reach-to-grasp actions (2010). The visual feedback of the hand is used both to guide the hand and the fingers toward the final grasping position (Smeets and Brenner 1999) and to determine the relative distance between hand and object through relative disparities (Melmoth et al. 2007). Haptic feedback, which takes place at the end of the movement, provides additional information about intrinsic properties of the object, such as shape and size. It can be assumed that this feedback corrects the initially biased visual estimate of the object size to allow accurate future grasping movements (Bingham et al. 2007; 2010, 2012).
In the present study we pursued two aims: 1) to investigate the specific function and use of binocular information in visuomotor processes and 2) to understand the extent to which systematic errors in reach-to-grasp actions executed without feedback can be reduced through training sessions in which haptic and visual feedback are provided.
The study of reach-to-grasp in impoverished or altered sensory environments is doubtless the most direct way to address the first aim. Virtual reality setups permit the selective presentation of feedbacks, both visual and haptic, allowing systematic investigation of their effect on reach-to-grasp actions (Bingham et al. 2000, 2007; Bradshaw and Elliott 2003; Hibbard and Bradshaw 2003; Melmoth et al. 2007; Schenk 2012; Servos 2000; Servos et al. 1992; Smeets et al. 2006). In this respect, some studies have tested grasping and reaching movements by asking subjects to perform the action beside the object, rather than performing it as an object-oriented action (Goodale et al. 1994), or by limiting the availability of visual information to the phase preceding the movement execution, letting subjects perform the movement based only on the memory of the object (Goodale et al. 1994). These tasks, defined as “pantomimed actions,” give rise to a pattern of errors similar to that found in perceptual tasks. Therefore “pantomimed actions” have been attributed to the workings of the ventral stream, which is involved in perceptual processing (Goodale et al. 1991, 1992, 1994; Whitwell et al. 2014). The classic explanation of this phenomenon is that the absence of feedback, especially haptic feedback, disengages the dorsal stream from its main role of controlling motor actions, which are thought to be unaffected by systematic biases typical of perceptual processing. The general consensus is that the presence of haptic feedback is crucial for defining a movement as “natural,” even though distortions of shapes have been also reported in studies in which the haptic feedback was provided at the end of the movement (Cuijpers et al. 2008; Hibbard and Bradshaw 2003).
However, it could also be speculated that tasks defined as “pantomimed actions” give rise to biases not because they engage different cortical areas but because they lack information, provided by haptic and visual feedback, that is fundamental for the correct execution of a movement. This speculation led us to the second aim of this study, which focuses on the role of visual and haptic feedback in the calibration of directed actions. In this regard, studies of the use of incongruent or mismatched feedback about the object position and/or dimension have shown that the system can rapidly adapt to these new visuomotor mappings that are maintained even once feedback is removed (Coats et al. 2008; Gentilucci et al. 1995; Redding and Wallace 2006; Rossetti et al. 1993). These results suggest that visuomotor actions in reaching and grasping tasks may be grounded on simple mechanisms of associative learning based on error signals from visual and haptic feedback (Bingham et al. 2007; Bingham and Pagano 1998; Cuijpers et al. 2008; Domini and Caudek 2013; Foster et al. 2011; Mon-Williams and Bingham 2007). Additionally, they show that processes underlying the computation of the movement transport phase and grip phase can be calibrated either together or separately, depending on the specific feedback provided (Bedford 1989; Bingham et al. 2000; Coats et al. 2008; Marotta et al. 2005; Mon-Williams and Bingham 2007).
Our goal here is to study calibration in actions when veridical feedback is provided. Previously, Bingham and colleagues (2007) reported an important result on the possibility of removing biases from grasping movements thanks to haptic feedback. Using a trial-by-trial design, their study described a transfer of information from trials with veridical haptic feedback to trials without feedback. However, they did not consider other information involved in the online control of object-oriented actions, i.e., vision of the moving hand and, specifically, vision of the fingers grasping the target. Moreover, this study focused on trial-by-trial calibration only, whereas evidence of longer-lasting calibration is still lacking.
An important issue in the study of calibration and visuomotor adaptation processes is to verify whether the visual system can learn to achieve veridical object distance and depth estimates once online control and haptic feedback are no longer available. In addition, the individual role of these feedbacks in achieving this aim remains an open question.
With the present study we addressed these questions by testing the execution of reach-to-grasp actions in four groups of subjects trained with four different combinations of haptic and visual feedback. They performed reach-to-grasp actions in three consecutive blocks: a Pretraining, a Training, and a Posttraining block.
In the Pretraining and Posttraining blocks only vision of the object was allowed. In the Training blocks four different combinations of haptic and visual feedback were given along with vision of the target object: 1) vision of the thumb moving toward the object, allowing the subject to guide the finger toward the anterior surface of the object to be reached, 2) vision of the thumb and vision of the index finger, providing visual information about the fingers' span to guide the hand to the object contact points (Smeets and Brenner 1999), 3) vision of the thumb and haptic feedback of the object, providing additional information about the object's position and its relative depth, and 4) vision of the thumb and index finger along with haptic feedback of the object, providing the full combination of the aforementioned information. Hence, vision of the thumb was always present during the Training sessions in all experimental conditions. In a custom-built virtual reality setup (Nicolini et al. 2014) we were able to selectively provide the desired combination of haptic and visual feedback.
MATERIALS AND METHODS
Subjects.
Forty-four volunteers, students at the University of Trento (mean age 23.2 yr; 30 women, 14 men), participated in the study. All had normal or corrected-to-normal vision, and none presented neurological or psychiatric diseases. All subjects were naive to the purpose of the experiment and were paid for their effort. The experiments were undertaken with the understanding and written consent of each subject, with the approval of the Comitato Etico per la Sperimentazione con l'Essere Vivente of the University of Trento, and in compliance with national legislation and the Code of Ethical Principles for Medical Research Involving Human Subjects of the World Medical Association (Declaration of Helsinki).
Apparatus and design.
Subjects were seated in a dark room in front of a high-quality, front-silvered 400 × 300-mm mirror. The mirror was slanted at 45° relative to the subjects' sagittal body midline and reflected the image displayed on a ViewSonic 9613 19-in. CRT monitor placed directly to the left of the mirror (Fig. 1A). For consistent vergence and accommodative information, the position of the monitor, attached to a linear positioning stage (Velmex, Bloomfield, NY), was adjusted on a trial-by-trial basis to equal the distance from the subject's eyes to the virtual object. To present visual stimuli in three dimensions (3D), we used a frame interlacing technique in conjunction with liquid crystal FE-1 goggles (Cambridge Research Systems, Cambridge, UK) synchronized to the monitor frame rate. A C++ program controlled stimulus presentation and response recording (Nicolini et al. 2014). All experimental sessions started with the calibration procedure. Subjects' head position and orientation was tracked with three infrared-emitting diodes arranged on the back of the head. The position of each digit was calculated with respect to the three infrared-emitting diodes attached on each distal phalanx. Finally, one infrared-emitting diode was located on the ulnar styloid process to track the wrist position. Head, wrist, index finger, and thumb movements were acquired online at 100 Hz with submillimeter resolution with an Optotrak Certus motion tracker with two position sensors (Northern Digital, Waterloo, ON, Canada). The positions of the fingers' pads were used for the alignment of the visual feedback, a dot representing the pad of the finger, with the finger itself. Head movements updated the subjects' viewpoint to present the correct geometric projection of the stimulus in real time. High-contrast random-dot visual stimuli (Fig. 1B) were disparity-defined and rendered in stereo, simulating a vertically oriented cylinder with an elliptic cross section (height, y-axis: 130 mm; width, x-axis: 30 mm; depth, z-axis oriented along the viewing direction: 20 or 40 mm). Stimuli were simulated at eye height at three distances (420, 470, and 520 mm) and provided consistent vergence and accommodation cues. Because of the small size of the stimulus, the influence of vertical disparity was negligible (Rogers and Bradshaw 1993). In those blocks in which tactile feedback was provided, a real physical object was presented in the same location and with the same dimensions as the visual stimulus. The physical objects were mounted on a platform moved by a second linear positioning stage. The experiment was a block design. All subjects performed three different subsequent blocks. Each block consisted of 60 trials including the different combinations of the three distances and two object depths (10 repetitions). Trials were presented in a randomized order. Subjects were required to reach to grasp the virtual object shown in front of them as if it was physically present behind the mirror (see Fig. 1A). After an auditory cue, subjects were allowed to start moving. To keep the duration of stimulus presentation similar among subjects, the object started being visible at the time of the movement onset. Since the object was visible during the whole grasping action, which typically took ∼2 s to be completed, the participants had enough time to plan and update their action during the whole transport phase. Binocular information available during the reach is indeed paramount in the control of prehension to select the correct motor program in a feedforward strategy (Servos et al. 1992). Moreover, the late kinematic components that we wanted to investigate are shown not to be affected by visual information in the preprogramming phase that, although useful, is not necessary (Bradshaw and Elliott 2003). An auditory cue was provided 2,500 ms after movement initiation, indicating the end of the trial. After that, the object was no longer visible and the subject could move the hand back to the starting point. The reach-to-grasp movement had to be performed within this time interval.
Fig. 1.Experimental apparatus and feedback conditions. A: schematic top view of the experimental apparatus. Subjects reached behind the mirror to grasp the 3D object they could see in the mirror. The monitor distance was adjusted to match the simulated distance of the object. B: schematic representation of the stimulus. The dimension of the cylinder with an elliptic cross section could vary along the z-axis that was oriented along the viewing direction. C: feedback conditions: subjects were trained with only vision of the thumb (Vt), vision of the thumb and index finger (Vt-i), vision of the thumb and haptic feedback of the object (HVt), or vision of the thumb and index finger and haptic feedback of the object (HVt-i).
Procedure.
Each subject was tested in a dark room with his/her head positioned on a chin rest to avoid as many movements as possible. Before starting the experiment, subjects were tested for stereo vision with a custom test in which they were required to report whether a fronto-parallel disparity-defined surface with different degrees of curvature (base-to-peak depth differences between 5 and 30 mm) was bended toward or away from them. Four subjects failed the test and did not take part in the study. The rest of the subjects were subsequently presented with a short subset of practice trials to get accustomed to the virtual environment and to the movement to be performed. Subjects were randomly assigned to one of the four feedback conditions. In each of the conditions tested, subjects ran three different consecutive blocks: a Pretraining block (used as baseline), a Training block, and a Posttraining block. Differences between conditions were related to the kind of feedback provided in the Training block. Subjects started each trial of the experiment with their thumb and index fingertips in contact and resting on the top of a pole. The top of the pole was shifted relative to the body of the subject by about 250 mm to the right from the coronal plane, 150 mm from the sagittal plane, and 300 mm lower than the subjects' line of sight. During each block, subjects were instructed to move toward the virtual cylinder and grasp it along its depth axis. The movement had to be performed as naturally as possible. In the first (Pretraining) and third (Posttraining) blocks of each condition, subjects had to perform the reach-to-grasp movement in the dark and could only see (but not touch) the virtual object. In the second block (Training), subjects were provided with one of the different feedback combinations according to the condition they were assigned to. In the visual thumb (Vt) condition, subjects were provided with visual feedback of their thumb. In the visual thumb-index (Vt-i) condition, the feedback provided was the view of both thumb and index finger (the index finger was visible until it disappeared behind the object, i.e., 15 mm from the contact point). In the haptic and visual thumb (HVt) condition, subjects were provided with the view of the thumb and the haptic feedback of the object. In the haptic and visual thumb-index (HVt-i) condition, the view of both thumb and index finger along with the haptic feedback of the object were given (Fig. 1C). A virtual dot representing the tips of the visible fingers appeared as soon as the fingers entered the subject's visual field and remained visible for the whole trial duration. At the end of each trial the monitor turned black and the subject returned to the starting position. Then the monitor (and, when present, the object) moved to the new position, ready for the start of the next trial.
Data analysis.
The raw positional data were processed and analyzed off-line with custom software. The raw data were smoothed and differentiated with a second-order Savitzky-Golay filter with a window size of 41 points. These filtered data were then used to compute velocities and accelerations in 3D space for each fingertip and the wrist, the Euclidean distance between the fingertips of the thumb and the index finger (grip aperture), and the velocity and acceleration of the change in grip aperture. The dependent measures were the terminal hand position error (THP), the maximum grip aperture (MGA), and the final grip aperture (FGA). The THP was defined as the difference between the position of the thumb fingertip along the z-axis (depth component) at the end of the grasping movement and the position subjects had to reach on the anterior surface of the object. The FGA and the MGA were defined as the Euclidean distance between the fingertips at the end of the grasping movement and the maximum Euclidean distance between the fingertips, respectively. We defined the end of the grasping movement on the basis of the multiple sources of information method (see Schot et al. 2010). The parameters used were the velocities of the index finger, thumb, and wrist, the distance from the starting position, and the velocity and acceleration of the change in grip aperture.
RESULTS
Pretraining block analysis.
At first, subjects' performance for THP, FGA, and MGA was assessed in the Pretraining block to be used as baseline for further analysis. Both the transport and grip phases of the movement were inaccurate, revealing an underestimation of the object distance and an overestimation of the object depth. Most importantly, the distance at which the object was presented affected both biases in a systematic fashion. The distance underestimation increased with object distance (Fig. 2A, left), whereas the overestimation of FGA was the largest at the closest distance and gradually decreased as the object distance increased (Fig. 3A), showing a systematic lack of depth constancy. Similarly, MGA was also modulated by the object distance.
Fig. 2.Hand position analysis. A: terminal hand position error (THP) in the Pretraining (left) and Posttraining (right) blocks as a function of the object distance. Error bars represent the 95% within-subject confidence intervals. B: relation between Pretraining position error and position correction. The position correction is calculated as the amount of correction between the position reached in the Pretraining and Posttraining blocks. C: drift of the position error in the Posttraining blocks. Each panel represents the average drift after each training condition.
Fig. 3.Pre- and Posttraining final grip aperture (FGA) and maximum grip aperture (MGA). A: average FGA in the Pretraining (left) and Posttraining (right) blocks for the 2 objects (gray line = 20 mm; black line = 40 mm) as a function of the object distance. B: average MGA in the Pretraining (left) and Posttraining (right) blocks for the 2 objects (gray line = 20 mm; black line = 40 mm) as a function of the object distance. Error bars represent the 95% within-subject confidence interval.
The general undershoot in reaching distance was evaluated by a one-sample t-test against zero run on the average THP [t(43)= −8.71; P < 0.01]. A repeated-measures ANOVA with distance as main factor and feedback condition as between-participant factor was run on the THP. The feedback condition did not show any effect [F(3,40) = 1.07, P = 0.37]. Note that at this point of the experiment the subjects were not yet exposed to any of the feedback. In contrast, a significant effect of distance outlined that this underestimation was more pronounced at the furthest distance [F(2,80) = 42.15; P < 0.0001] (Fig. 2A, left); no interaction effect between feedback condition and distance was found [F(6,80) = 0.17, P = 0.98]. Similarly, a one-sample t-test against the veridical depth of the objects was run on the FGA variable, showing that FGA was overestimated for both object depths [20 mm: t(43) = 10.87; P < 0.001; 40 mm: t(43) = 2.05; P = 0.04]. A repeated-measures ANOVA with depth and distance as main factors and feedback condition as between-participant factor was then run, showing an effect of depth [F(1,40) = 51.48; P < 0.0001], distance [F(2,80) = 16.09; P < 0.0001], and their interaction [F(2,80) = 4.61; P = 0.01] but no effect of the between-participant factor [F(3,40) = 0.55, P = 0.64]. To better understand the interaction effect, we split the data into two subgroups for the big and small objects and ran separate ANOVAs with distance as main factor. Results showed that the FGA was modulated by distance for the big object only [F(2,86) = 16.31, P < 0.0001] and not for the small object [F(2,86) = 3.03, P = 0.053]. In a similar vein, a repeated-measures ANOVA with depth and distance as main factors and feedback condition as between-participant factor was run also on the MGA, showing a significant effect of distance [F(2,80) = 13.31; P < 0.0001] and depth [F(1,40) = 47.42; P < 0.0001] but not of their interaction [F(2,80) = 1.76, P = 0.17] and no effect of the feedback condition [F(3,40) = 1.81, P = 0.15] or interaction effects of the between-participant factor with any of the main factors.
Hand position: Pre- and Posttraining adjustment.
In the Posttraining blocks, the reached position was less biased compared with the Pretraining blocks, although still underestimated, as shown in Fig. 2A, right. No effect of the feedback condition was found, showing that all feedback conditions produced a similar final adjustment. Most importantly, even after the Training blocks, subjects maintained the same distance bias, that is, they kept performing more poorly as the distance increased.
THP was analyzed with a repeated-measures ANOVA with block (Pretraining and Posttraining) and distance as within-participant factors and feedback condition as between-participant factor. A main effect of block [F(1,40) = 20.74; P < 0.0001] and distance [F(2,80) = 62.63; P < 0.0001] but no effect of the feedback condition [F(3,40) = 0.91; P = 0.44] was found. Because of the absence of both the feedback condition effect and the feedback condition and block interaction [F(3,40) = 1.04, P = 0.38], we can conclude that the different feedback conditions produced similar improvements, but none of the feedback conditions was more prominent in the visuomotor adaptation process. After Training blocks, all groups improved their performance only slightly (Fig. 2A, right), without reaching a full calibration, as shown by the one-sample t-test against zero run on the average THP in the Posttraining block [t(43)= −6.65; P < 0.01]. Most importantly, the amount of correction between the Pretraining and Posttraining blocks varied as a function of the error in the Pretraining block: the larger the error in the Pretraining block, the larger the correction in the Posttraining block [F(1,42) = 14.86; P = 0.0003; R2 = 0.26] (Fig. 2B).
Our results do not provide evidence of a complete calibration of the reaching position but only of a partial adjustment. Thus it might be possible that in the present situation the lack of calibration is a consequence of a drifting behavior (Magne and Coello 2002; Smeets et al. 2006; Wann and Ibrahim 1992). To verify whether this was the case, the THP of the Posttraining blocks was analyzed on a trial-by-trial basis. For each subject, we fitted the THP with a linear regression model as a function of the trial number. The intercept (equivalent to the THP on the first trial) and slope parameters were compared against zero with separate t-tests. Both the intercept [t(40) = −5.47, P < 0.0001] and the slope [t(40) = −3.88, P = 0.0003] were significantly different from zero, showing an instantaneous reappearance of the bias already by the very first trials in the Posttraining blocks and a subsequent moderate drift (see Fig. 2C).
Final and maximum grip apertures: Pre- and Posttraining adjustment.
FGA and MGA were not different between the Pretraining and Posttraining blocks, as shown in Fig. 3. Therefore, the Training blocks did not have any influence in correcting either the overestimation of the FGA or the general lack of depth constancy. As observed in the Pretraining blocks, depth estimates also decreased with distance in the Posttraining blocks (see Fig. 3).
A repeated-measures ANOVA on the FGA with block (Pretraining and Posttraining), distance, and depth as within-participant factors and feedback condition as between-participant factor was run. It revealed a significant effect of distance [F(2,80) = 29.83; P < 0.0001] and depth [F(1,40) = 69.33; P < 0.0001]. FGA performance was consistently modulated by the distance at which the object was presented (Fig. 3A, right). Feedback conditions and block did not show any significant effect [F(3,40) = 0.09; P = 0.96 and F(1,40) = 0.06; P = 0.80, respectively], as well as all interactions between different factors. Similarly to the FGA, the repeated-measures ANOVA on the MGA revealed a main effect of distance and depth [F(2,80) = 22.71; P < 0.0001 and F(1,40) = 56.94; P < 0.0001, respectively] and their interaction [F(2,80) = 3.21; P = 0.04], showing that distance had a stronger effect on the MGA for the bigger object than for the smaller object (Fig. 3B, right). Feedback conditions [F(3,40) = 1.286; P = 0.29] and block [F(1,40) = 0.04; P = 0.84] did not show any significant effect; nor did their interaction [F(3,40) = 2.26; P = 0.09].
Final and maximum grip apertures: distance bias in Training blocks.
The lack of depth constancy, as revealed by the distance effect on both FGA and MGA, is a remarkable finding, since subjects maintained the same bias even after being trained with both visual feedback of the fingers and haptic feedback of the object. In a further analysis, we asked whether this systematic bias was also present within the Training blocks.
In the Vt and Vt-i conditions we analyzed the FGA, since at the end of the grasp the object could not be felt and the feedback of the index finger disappeared behind the object. The repeated-measures ANOVA showed a significant effect of feedback condition [F(1,20) = 11.6; P = 0.002] and, most importantly, a significant effect of distance [F(2,40) = 8.12; P = 0.001] but no interaction effect [F(2,40) = 0.97; P = 0.38]. As shown in Fig. 4A, in the Vt-i condition subjects performed a more accurate FGA, confirming the importance of online control in grasping actions and the use of the two visible fingers for comparing the grip aperture to the depth of the object. However, in both visual feedback conditions the FGA decreased significantly as a function of object distance, showing that even the visual feedback of both fingers is insufficient to counteract the lack of depth constancy (Fig. 4A).
Fig. 4.FGA and MGA in the Training blocks. A: average FGA performed in the Training blocks where vision of the fingers was provided but without haptic feedback, Vt (dark gray) and Vt-i (light gray), for the 20-mm (left) and 40-mm (right) object depths at the 3 object distances. Dotted lines represent the veridical depth of the objects. B: average MGA in the 4 Training blocks (Vt, dark gray; Vt-i, light gray; HVt, dotted dark gray; HVt-I, dotted light gray) for the 20-mm (left) and 40-mm (right) object depths at the 3 distances. Error bars represent the 95% within-subject confidence interval.
In the same vein, the MGA was analyzed in all feedback conditions. Statistical analysis showed a main effect of feedback condition [F(3,40) = 2.84; P = 0.05], depth [F(1,40) = 45.151; P < 0.001], their interaction [F(3,40) = 4.85, P = 0.005], and distance [F(2,80) = 5.976; P = 0.01]. Figure 4B shows that the interaction effect was caused by a bigger difference between the MGA for the small and big objects used in the Vt feedback condition. This effect is probably due to the fact that this condition provided subjects with the least amount of information, giving rise to more uncertain and variable behavior. In those conditions in which tactile feedback was present, subjects performed a larger MGA compared with the other conditions, probably in order to increase the safety margin to avoid collision with the physical object. Remarkably, regardless of the feedback provided, they always showed a lack of depth constancy as a function of object distance, with a decrease of the MGA for objects presented further away (∼2 mm between the closest and furthest distances) (Fig. 4B).
DISCUSSION
The present study highlights two important results in the study of visuomotor behavior: first, extensive training with different feedback produces partial calibration only of the object distance estimation; second, systematic distortions of the object depth estimate are preserved even when grasping actions are performed in a full-feedback condition.
The first result concerns the study of the calibration process that corrects systematic biases present in visuomotor tasks. Data collected in the Pretraining blocks, where subjects did not have access to any feedback but could only rely on binocular cues, show that motor performances mimic the behavior observed in perceptual tasks in two important ways. In the first instance, the FGA always showed an overestimation of the object depth. Second, both the FGA and MGA results clearly revealed a systematic lack of depth constancy, since they both decreased with object distance. The failure of depth constancy is compatible with an incorrect scaling of binocular disparities. As for perceptual estimates (Bingham and Pagano 1998; Johnston 1991; Norman et al. 1996; Volcic et al. 2013), the scaling distance of binocular disparities is overestimated at the closest grasping distance, and this overestimation gradually becomes smaller at larger object distances.
The explanation of depth constancy failure in terms of scaling distance is compatible with the errors in terminal hand position. Even though objects were underreached at all grasping distances, they were less underreached at close distances than at large distances. At the closest distance (420 mm) objects were underreached on average by 27 mm, whereas at the largest distance (520 mm) they were underreached on average by 50 mm. This suggests that the visual space is compressed toward the observer and that the encoding of object distance for reaching is affected by the same systematic biases in the scaling of binocular disparities as revealed by the FGA and MGA data.
In the Posttraining blocks only the positioning of the hand was subject to a partial calibration, since reaching errors were smaller than those observed during Pretraining blocks. In this respect, different studies have shown a drift effect on the reaching position following a session of visuomotor adaptation once the feedback was again removed (Magne and Coello 2002; Smeets et al. 2006; Wann and Ibrahim 1992). In the same vein, we found a similar drift backward, but, most importantly, already by the beginning of the Posttraining block subjects showed a significant undershot, indicating an immediate decay of the proprioceptive information acquired during the Training blocks. Strikingly, the amount of partial calibration did not change based on the kind of feedback provided during the Training blocks, as if only minimal visual feedback about hand position (the thumb in our case) was necessary and sufficient for this calibration to take place. Even though the importance of vision of the hand in grasping actions has been described previously (Connolly and Goodale 1999; Fukui and Inui 2013; Whitwell et al. 2008), the predominant role of vision of the thumb in the present study can be due to the fact that subjects grasped objects along the depth axis at eye height, where only the contact point of the thumb was visible (Volcic and Domini 2014).
Unlike the hand positioning, in the Posttraining blocks MGA and FGA were not affected by the Trainings, even when both haptic and visual feedback of the grasping fingers were provided. These results suggest that the scaling of binocular disparities, which depends on the estimated distance to the object (Brenner and van Damme 1999; Johnston 1991; van Damme and Brenner 1997; Volcic et al. 2013), was not affected by the Training blocks. This is especially surprising since the partial calibration of the hand position does indicate a change in object distance estimate. These compelling results are compatible with the view that the different components of reach-to-grasp actions can be independently adjusted (Coats et al. 2008; Marotta et al. 2005; Volcic and Domini 2014).
Since both overestimation of object depth and failure of depth constancy shown by the FGA and MGA data were preserved in the Posttraining blocks, we can conclude that the intrinsic object information provided during the Trainings could not be learned. This may suggest that this information is only gathered from the visual information available while the grasping action unfolds. However, even when haptic feedback was available throughout the block, the MGA was subject to the same depth constancy failure observed during Pretraining and Posttraining blocks. This second compelling result concerning the similarity between perceptual biases and biases observed for reach-to-grasp actions endorses the hypothesis that processing of visual information for both perceptual judgments and visually guided actions is indicative of similar and interrelated underlying mechanisms (Foster et al. 2011; Franz et al. 2000, 2009; Franz and Gegenfurtner 2008; Schenk 2010, 2012). In particular, the present findings would indicate that depth scaling of binocular disparities is the same for perception and action, a result incompatible with the dual visual systems theory (Goodale et al. 1991; Goodale and Milner 1992). Previous research in support of this theory has pointed to the difference between acting upon a real target and pantomimed action, occurring when the target is not physically present and/or the action is just mimed beside it. In the latter cases, the action is shown to be subject to the different biases similar to those found in perceptual tasks, leading to the assumption that different perceptual processes are engaged depending on the different feedback available (Goodale et al. 1994; Whitwell et al. 2014). Then again, what these studies show is an undeniable inaccuracy in object depth and distance estimates in the absence of haptic feedback, supporting the key role the haptic feedback has in grasping actions (Bingham et al. 2007; Schenk 2012; Whitwell et al. 2014). Nevertheless, the present findings demonstrate that also in the presence of all sensory feedback, haptic and visual, depth constancy is still compromised.
In conclusion, the present findings suggest that metric estimates of extrinsic and intrinsic object properties for guiding grasping actions are in general inaccurate (Hibbard and Bradshaw 2003; Schenk 2010), even when reliable visual information carried by binocular disparities is potentially available during the entire movement duration. Moreover, they show that these biases, typically observed in perceptual tasks, cannot be corrected even after extensive Training sessions during which proprioceptive information of the object distance and depth is available via haptic and visual feedback. Therefore, we can speculate that action may take place without accurate metric information, since non-Euclidean metric relationships are sufficient for motor control (Domini and Caudek 2013). This is indeed the case in real-world situations in which visual feedback of the hand and haptic feedback of the object are typically present during successful grasping actions.
DISCLOSURES
No conflicts of interest, financial or otherwise, are declared by the author(s).
AUTHOR CONTRIBUTIONS
Author contributions: C.B., R.V., and F.D. conception and design of research; C.B. performed experiments; C.B. analyzed data; C.B., R.V., and F.D. interpreted results of experiments; C.B. prepared figures; C.B. and R.V. drafted manuscript; C.B., R.V., and F.D. edited and revised manuscript; C.B., R.V., and F.D. approved final version of manuscript.
REFERENCES
- . Constraints on learning new mappings between perceptual dimensions. J Exp Psychol Hum Percept Perform 15: 232–248, 1989.
Crossref | ISI | Google Scholar - . Natural prehension in trials without haptic feedback but only when calibration is allowed. Neuropsychologia 45: 288–294, 2007.
Crossref | PubMed | ISI | Google Scholar - . Accommodation, occlusion, and disparity matching are used to guide reaching: a comparison of actual versus virtual environments. J Exp Psychol Hum Percept Perform 27: 1314–1334, 2001.
Crossref | PubMed | ISI | Google Scholar - . The necessity of a perception-action approach to definite distance perception: monocular distance perception to guide reaching. J Exp Psychol Hum Percept Perform 24: 145–168, 1998.
Crossref | PubMed | ISI | Google Scholar - . Distortions in definite distance and shape perception as measured by reaching without and with haptic feedback. J Exp Psychol Hum Percept Perform 26: 1436–1460, 2000.
Crossref | PubMed | ISI | Google Scholar - . The role of binocular information in the “on-line” control of prehension. Spat Vis 16: 295–309, 2003.
Crossref | PubMed | Google Scholar - . Perceived distance, shape and size. Vision Res 39: 975–986, 1999.
Crossref | PubMed | ISI | Google Scholar - . The same object and at least three different grip apertures. J Vis 12: 429, 2012.
Crossref | Google Scholar - . Calibrating grasp size and reach distance: interactions reveal integral organization of reaching-to-grasp movements. Exp Brain Res 189: 211–220, 2008.
Crossref | PubMed | ISI | Google Scholar - . The role of visual feedback of hand position in the control of manual prehension. Exp Brain Res 125: 281–286, 1999.
Crossref | PubMed | ISI | Google Scholar - . Consistent haptic feedback is required but it is not enough for natural reaching to virtual cylinders. Hum Mov Sci 27: 857–872, 2008.
Crossref | PubMed | ISI | Google Scholar - . Perception and action without veridical metric reconstruction: an affine approach. In: Shape Perception in Human and Computer Vision. London: Springer, 2013, pp. 285–298.
Crossref | Google Scholar - . Binocular distance perception. Psychol Rev 87: 411–434, 1980.
Crossref | PubMed | ISI | Google Scholar - . Integration of disparity and velocity information for haptic and perceptual judgments of object depth. Acta Psychol 136: 300–310, 2011.
Crossref | PubMed | ISI | Google Scholar - . Grasping visual illusions: consistent data and no dissociation. Cogn Neuropsychol 25: 920–950, 2008.
Crossref | ISI | Google Scholar - . Grasping visual illusions: no evidence for a dissociation between perception and action. Psychol Sci 11: 21–25, 2000.
Crossref | ISI | Google Scholar - . Visual illusions, delayed grasping, and memory: no shift from dorsal to ventral control. Neuropsychologia 47: 1518–1531, 2009.
Crossref | PubMed | ISI | Google Scholar - . Utilization of visual feedback of the hand according to target view availability in the online control of prehension movements. Hum Mov Sci 32: 580–595, 2013.
Crossref | PubMed | ISI | Google Scholar - . Unconscious updating of grasp motor program. Exp Brain Res 105: 291–303, 1995.
Crossref | PubMed | ISI | Google Scholar - . Differences in the visual control of pantomimed and natural grasping movements. Neuropsychologia 32: 1159–1178, 1994.
Crossref | PubMed | ISI | Google Scholar - . Separate visual pathways for perception and action. Trends Neurosci 15: 20–25, 1992.
Crossref | PubMed | ISI | Google Scholar - . A neurological dissociation between perceiving objects and grasping them. Nature 349: 154–156, 1991.
Crossref | PubMed | ISI | Google Scholar - . Reaching for virtual objects: binocular disparity and the control of prehension. Exp Brain Res 148: 196–201, 2003.
Crossref | PubMed | ISI | Google Scholar - . Systematic distortions of shape from stereopsis. Vision Res 31: 1351–1360, 1991.
Crossref | PubMed | ISI | Google Scholar - . Retinal and extra-retinal contribution to position coding. Behav Brain Res 136: 277–287, 2002.
Crossref | PubMed | ISI | Google Scholar - . Task-specific sensorimotor adaptation to reversing prisms. J Neurophysiol 93: 1104–1110, 2005.
Link | ISI | Google Scholar - . Advantages of binocular vision for the control of reaching and grasping. Exp Brain Res 171: 371–388, 2006.
Crossref | PubMed | ISI | Google Scholar - . Dissociation between vergence and binocular disparity cues in the control of prehension. Exp Brain Res 183: 283–298, 2007.
Crossref | PubMed | ISI | Google Scholar - . Calibrating reach distance to visual targets. J Exp Psychol Hum Percept Perform 33: 645–656, 2007.
Crossref | PubMed | ISI | Google Scholar - . A framework for the study of vision in active observers. In: Proceedings of SPIE 9014, Human Vision and Electronic Imaging, edited by , Rogowitz BE, Pappas TN, de R, idder H. Bellingham, WA: SPIE, 2014, 901414.
Google Scholar - . The visual perception of three-dimensional length. J Exp Psychol Hum Percept Perform 22: 173–186, 1996.
Crossref | PubMed | ISI | Google Scholar - . Prism adaptation in alternately exposed hands. Atten Percept Psychophys 75: 1168–1185, 2006.
Crossref | ISI | Google Scholar - . Vertical disparities, differential perspective and binocular stereopsis. Nature 361: 253–255, 1993.
Crossref | PubMed | ISI | Google Scholar - . Prismatic displacement of vision induces transient changes in the timing of eye-hand coordination. Percept Psychophys 54: 355–364, 1993.
Crossref | PubMed | Google Scholar - . Visuomotor robustness is based on integration not segregation. Vision Res 50: 2627–2632, 2010.
Crossref | PubMed | ISI | Google Scholar - . No dissociation between perception and action in patient DF when haptic feedback is withdrawn. J Neurosci 32: 2013–2017, 2012.
Crossref | PubMed | ISI | Google Scholar - . Vision-for-perception and vision-for-action: which model is compatible with the available psychophysical and neuropsychological data? Vision Res 51: 812–818, 2011.
Crossref | PubMed | ISI | Google Scholar - . Robust movement segmentation by combining multiple sources of information. J Neurosci Methods 187: 147–155, 2010.
Crossref | PubMed | ISI | Google Scholar - . Distance estimation in the visual and visuomotor system. Exp Brain Res 130: 35–47, 2000.
Crossref | PubMed | ISI | Google Scholar - . The role of binocular vision in prehension: a kinematic analysis. Vision Res 32: 1513–1521, 1992.
Crossref | PubMed | ISI | Google Scholar - . A new view on grasping. Motor Control 3: 237–271, 1999.
Crossref | PubMed | ISI | Google Scholar - . Sensory integration does not lead to sensory calibration. Proc Natl Acad Sci USA 103: 18781–18786, 2006.
Crossref | PubMed | ISI | Google Scholar - . Distortions of three-dimensional space in the perceptual analysis of motion and stereo. Perception 24: 75–86, 1995.
Crossref | PubMed | ISI | Google Scholar - . The distance used for scaling disparities is the same as the one used for scaling retinal size. Vision Res 37: 757–764, 1997.
Crossref | PubMed | ISI | Google Scholar - . The visibility of contact points influences grasping movements. Exp Brain Res 232: 297–3005, 2014.
Crossref | ISI | Google Scholar - . Visuomotor adaptation changes stereoscopic depth perception and tactile discrimination. J Neurosci 33: 17081–17088, 2013.
Crossref | PubMed | ISI | Google Scholar - . Does limb proprioception drift? Exp Brain Res 91: 162–166, 1992.
Crossref | PubMed | ISI | Google Scholar - . Grasping future events: explicit knowledge of the availability of visual feedback fails to reliably influence prehension. Exp Brain Res 188: 603–611, 2008.
Crossref | PubMed | ISI | Google Scholar - . DF's visual brain in action: the role of tactile cues. Neuropsychologia 55: 41–50, 2014.
Crossref | PubMed | ISI | Google Scholar
