An Augmented Reality Hand–Eye Sensorimotor Impairment Assessment for Spaceflight Operations
INTRODUCTION: Following a transition from microgravity to a gravity-rich environment (e.g., Earth, Moon, or Mars), astronauts experience sensorimotor impairment, primarily from a reinterpretation of vestibular cues, which can impact their ability to perform mission-critical tasks. To enable future exploration-class missions, the development of lightweight, space-conscious assessments for astronauts transitioning between gravity environments without expert assistance is needed. METHODS: We examined differences in performance during a two-dimensional (2D) hand–eye multidirectional tapping task, implemented in augmented reality in subjects (N = 20) with and without the presence of a vestibular-dominated sensorimotor impairment paradigm: the binaural bipolar application of a pseudorandom galvanic vestibular stimulation (GVS) signal. Metrics associated with both the impairment paradigm and task performance were assessed. RESULTS: Medial-lateral sway during balance on an anterior-posterior sway-referenced platform with eyes closed was most affected by GVS (effect size: 1.2), in addition to anterior-posterior sway (effect size: 0.63) and the vestibular index (effect size: 0.65). During the augmented reality task, an increase in time to completion (effect size: 0.63), number of misses (effect size: 0.52), and head linear accelerations (effect size: 0.30) were found in the presence of the selected GVS waveform. DISCUSSION: Findings indicate that this multidirectional tapping task may detect emergent vestibular-dominated impairment (near landing day performance) in astronauts. Decrements in speed and accuracy indicate this impairment may hinder crews’ ability to acquire known target locations while in a static standing posture. The ability to track these decrements can support mission operations decisions. Allred AR, Weiss H, Clark TK, Stirling L. An augmented reality hand–eye sensorimotor impairment assessment for spaceflight operations. Aerosp Med Hum Perform. 2024; 95(2):69–78.
In humans, exposure to microgravity results in a central reinterpretation of information from multiple sensory sources to produce a sensorimotor state appropriate for motor actions in microgravity.4 However, this new adapted state is no longer appropriate for gravity-rich environments such as Earth, and a subsequent central reinterpretation is required to achieve a state appropriate for the gravity-rich environment. Before this adaptation completes, astronauts experience deficits in both perceptual and motor functions, including alterations in locomotor and postural control.6 This sensorimotor impairment can impact their ability to perform mission-critical tasks such as piloting vehicles and operating other complex systems.23
To quantify sensorimotor impairment, field tests including sit-to-stand and prone-to-stand transitions, walking, translating objects, and jumping down from a platform were performed upon return to Earth in 41 long-duration International Space Station crewmembers with an average mission duration of ∼6 mo (spanning 115–341 d).24 Subjects were a mixture of U.S. Orbital Segment astronauts (N = 22) and Russian cosmonauts (N = 16). When defining “recovered” as a return to within 95% of preflight performance, the time-to-stand performance metric in the “sit-to-stand” and “recovery from fall” tests show that 95% of the crew were recovered within 2 d after landing. The tandem walk task required 4 d after landing for 95% of the crew to obtain recovery.
These recovery timelines are confounded by crewmembers undergoing state-of-the-art rehabilitation with astronaut strength, conditioning, and rehabilitation specialists.28 Recovery times are expected to increase without this same level of rehabilitation during future missions. In addition, the tests were performed with the assistance of experts on the ground and the availability of space to perform the tasks. To mitigate the consequences of performing operational tasks in an impaired state, these future missions will require methods of assessing sensorimotor impairment and, consequentially, evaluating an associated operational task performance. These tasks and assessments will need to be performed with limitations in space and assistance during long-duration missions with strict habitat requirements.
Beyond the operational field test task metrics, computerized dynamic posturography (CDP) has been utilized to assess astronaut balance performance via a set of 20-s duration sensory organization tests (SOTs) following short- and long-duration missions8,15 and throughout recovery postflight.23,29 Notably, SOT 1 is a baseline postural test, with eyes open (visual cues) and a stable standing platform (somatosensory cues) accompanying the individual’s vestibular cues. SOT 5, balancing on an anterior-posterior (A-P) sway-referenced platform with eyes closed, has been used to isolate and assess balance performance when vestibular feedback is primarily relied on with limited somatosensory and ankle proprioceptive feedback and no visual cues. During this SOT, the greatest performance decrements have been found for astronauts postflight.23 While a valuable assessment here on Earth, there is no proposed plan to bring a CDP device on long-duration space missions to new gravity environments where operational tasks must be performed.
Due to both the limited assessment tools available for astronauts to assess their sensorimotor impairment and resultant operational risks, there exists a need to develop lightweight, space-conscious assessments for astronauts transitioning from microgravity to non-Earth gravity-rich environments. Additionally, because these assessments must be conducted prior to operational tasks (such as extravehicular activity), these tasks must be accomplished safely in a confined space. For these reasons, augmented-reality-based (AR-based) assessments offer perception of environmental elements for safe operation compared to virtual-reality-based tasks, while also enabling the collection of sensorimotor-dependent variables. AR-based assessments are easily deployable, enable the user to see their physical environment for safety in confined spaces, provide flexibility for new software integration, and are multifunctional for other mission tasks such as procedural guidance. Furthermore, assessments in an AR environment naturally enable hand-movement and eye-gaze observation metrics, which may provide more insight into vestibular and perceptual deficits experienced by the user.
One application leveraging AR as an assessment tool is the implementation of a two-dimensional (2D) hand–eye multidirectional tapping task. It has been shown that vestibular stimulation (provided via head rotations) can improve hand–eye coordination,26 but it is unknown whether such a task has the sensitivity to detect vestibular-dominated sensorimotor impairment in crewmembers transitioning to a gravity-rich environment from microgravity. For rapid aimed movements, Fitts’ Law11 relates movement time, distance, and accuracy, where the time required to reach a target increases with distance and with decreasing target size. This law is widely known to apply to pointing and dragging tasks using a mouse, trackball, stylus, joystick, and touchscreen. A recent study has extended these findings into newer interface technologies, such as AR, with the demonstration of a speed−accuracy trade-off with different interaction types (touchpad, pointing gesture, and raycast), yet no differences were observed in performance (measured by throughput, movement time, error rate, and incorrect click count) due to the transparency of the holographic content.19 This previous work comprised healthy subjects without musculoskeletal or vestibular impairments, thereby motivating the need for future work to examine task sensitivity to vestibular deficits. This research effort investigates the sensitivity of the 2D hand–eye multidirectional tapping task to vestibular disturbances with the Microsoft HoloLens 2 device, which supports hand-tracking and eye-tracking capabilities for the evaluation of various hand–eye performance metrics. Herein, an implementation is explored as a potential assessment tool.
In contrast to astronauts returning to a gravity-rich environment from microgravity, Earth-residing humans are effective at perceiving the direction of gravity and their self-motion in addition to performing sensorimotor tasks. These unimpaired capabilities are primarily due to the effective performance of the vestibular system. Postflight spatial disorientation is likely due both to microgravity-induced deconditioned otolith-mediated reflexes13 and a central nervous system reinterpretation of otolith cues which are not compatible with gravity-rich environments.4,18 To a lesser extent, somatosensory and visual changes are also believed to contribute to sensorimotor decrements5 (however, reported postflight visual disturbances may be due to vertigo rather than physiological or perceptual changes in the optic pathway). Indeed, almost all Shuttle crewmembers have reported illusory sensations of self-motion during reentry,1 even after short-duration shuttle missions (1–2 wk).22
Beyond studying postflight astronauts, vestibular impairment has been imitated in Earth-residing subjects using galvanic vestibular stimulation (GVS) as an impairment paradigm. GVS is a transcutaneous electrical stimulation applied to the mastoid processes, which affects the firing rate of the afferent vestibular neurons.16 In the binaural bipolar configuration, with large surface area electrodes, a low frequency (<1 Hz) pseudorandom GVS stimuli (with a peak current of 5 mA) has been shown to induce acute vestibular impairment, resulting in impaired postural control comparable to astronauts 2–4 h post-landing, examined via a CDP SOT protocol.17
While passing currents through the vestibular neurons in this manner results in vestibular disturbances, there are many inherent differences between GVS-induced vestibular disturbance from a pseudorandom waveform and spaceflight-induced vestibular disturbance. One major limitation is that GVS produces a nonspecific disturbance, whereas crewmembers describe specific illusory sensations of confusing angular head movements with linear motion.18 While it may be possible to recreate these specific readaptation illusions using a head-coupled GVS-waveform,2,3,14 existing works suggest that head-coupled systems result in fairly rapid adaptation to the GVS signal (<4 min after visual or somatosensory conditioning14), or subjects may instead adopt head-control strategies to minimize GVS disruptions.17 Thus, a noncoupled pseudorandom GVS stimulus provides the capability of providing prolonged GVS disturbances9,10 during Earth-based experiments for simulating vestibular-dominated sensorimotor impairment, which does not produce specific post-spaceflight vestibular illusions.
Finally, GVS is mostly a peripheral vestibular disturbance,7 while adaptation to microgravity involves a body weight unloading, fluid shifts, central vestibular and multisensory reinterpretation, and vision changes, all of which are in an inappropriate state once the crewmember returns to a gravity-rich environment. Nevertheless, GVS enables a means of providing a graded level of transient vestibular disruption necessary for evaluating the sensitivity of potential assessment tasks.
In this study, we utilize a 2D hand–eye multidirectional tapping task in AR as a sensorimotor assessment tool for detecting vestibular impairment upon transitioning to a gravity-rich environment from microgravity. The hand–eye task requires minimal space considerations to be performed within a habitat prior to vehicle egress. The chosen task is an ISO-recognized standard and widely used assessment for quantifying human motor and perceptual abilities while performing discrete tasks and is contained within the fine motor skills test battery of NASA. Additionally, vestibular impairment may significantly affect early operations requiring standing balance with hand–eye coordination (e.g., managing spacesuit umbilical interfaces or assembling and maintaining surface infrastructure) after gravity transitions. We leverage a pseudorandom GVS waveform as a form of vestibular-dominated sensorimotor impairment to alter sensory integration in Earth-residing subjects (Fig. 1). The level of susceptibility each subject experiences to the GVS waveform is first characterized via a current “gold standard” assessment tool: tracking postural stability on a CDP device with and without the pseudorandom GVS waveform. We formulate the following two primary hypotheses: 1) we hypothesize that our specific GVS waveform will induce postural stability performance decrements captured via the “gold standard” CDP device; and 2) we hypothesize that a 2D hand–eye Fitts’ Law assessment task (in the form of a multidirectional tapping task) in AR will reveal performance decrements when vestibular-dominated sensorimotor impairment is induced via GVS.
Citation: Aerospace Medicine and Human Performance 95, 2; 10.3357/AMHP.6313.2024
METHODS
Subjects
The study protocol was approved in advance by the NASA Institutional Review Board. All subjects provided written informed consent prior to participating in this study. A total of 20 individuals were recruited for this study (11 women and 9 men, mean age = 22.5 yr ± 2.5 SD, min: 19, max: 30). Subjects reported no history of vestibular dysfunction, and subjects reported varying degrees of virtual reality (VR) and AR experience (17 subjects reported previous use of either VR or AR). Of these 17 subjects, 5 reported prior use of an AR device and 13 reported prior use of a VR device. All but one subject reported daily use of a touchscreen (they reported using a touchscreen 4–6 times a week), and all subjects reported using a computer a minimum of 4–6 times a week.
Equipment
The CDP device at NASA Johnson Space Center was utilized to gather a baseline GVS susceptibility metric for each subject. Both SOT 1 and SOT 5 were conducted, with and without the pseudorandom GVS waveform, in order to gather a vestibular index (SOT 5/SOT 1) in both configurations. Prior to applying GVS, the subjects’ skin around the mastoids were cleaned and exfoliated. For applying GVS, 3-in diameter CarbonFlex insulated electrodes were applied to both mastoid processes, and current was delivered in the binaural bipolar configuration; after saturating the electrode sponges with electrode solution, electrode gel was applied between the electrodes and skin. The AR hand–eye task was administered on the Microsoft HoloLens 2 device, calibrated to each subject’s interpupillary distance. Prior to performing this study, the embedded inertial measurement unit (IMU) sensor in the HoloLens 2 was assessed for electromagnetic interference and sensor performance decrements in the presence of GVS current; no differences were found.
The random GVS waveform was comprised of a sum-of-sines with a frequency content ranging from 0.055–0.37 Hz and a peak-to-peak of ±3 mA. A peak-to-peak GVS amplitude below that was used by MacDougall et al.20 was chosen to provide a fraction of the sensorimotor impairment experienced by astronauts returning to Earth, representing only the vestibular component of impairment 0–4 h post-landing (R + 0A). While the exact fraction of impairment due to compromised vestibular inputs is unknown, this amplitude was chosen based on a pilot study in astronauts (N = 5) at NASA Johnson Space Center simulating post-spaceflight impairment using GVS and a weighted suit (somatosensory disruption), in which the amplitudes of each component were titrated based on astronaut feedback.21
The 2D hand–eye multidirectional tapping task, adapted from the ISO9241-9 standard, is an extension of the Fitts paradigm with the primary benefit of controlling for the effect of target direction.25 Moreover, this task is a subtask of the NASA-developed fine motor skills test battery, designed to determine the effects of microgravity and other stressors on motor skills (manual dexterity) essential for extravehicular and intravehicular activities during spaceflight. The hand–eye task features 16 targets arranged equidistant in a circular array with the principal aim of tapping the targets as quickly and accurately as possible. The sequence in which the subject acquires the targets follows a predefined tapping pattern that alternates the active target in a clockwise procedure across the full diameter of the array, with the starting and ending positions at the apex of the array (Fig. 2D). A single nominal index of difficulty, defined by the movement distance (diameter of the array) and the target widths, was implemented. A diameter of 0.152 m and a target width of 0.025 m was selected in accordance with NASA’s Man-Systems Integration Standards (NASA-STD-3000) for push buttons. This array yielded an index of difficulty of 2.824 b, in line with the 2–8 b range of IDs employed within the literature.25
Citation: Aerospace Medicine and Human Performance 95, 2; 10.3357/AMHP.6313.2024
The array of 16 targets was holographically projected into the user’s physical space, where world-anchoring ensured the array’s position and size remained constant irrespective of the user’s physical movements (Fig. 2C). In this manner, the subject was allowed to position themselves at an appropriate, personalized arm-length distance to interact with the target space. Only pointing interactions with the index finger of the dominant hand were permitted. Raycasting, a common AR gesture interaction for distant manipulation, was disabled within the augmented scene to enforce the use of touch selection. The 2D hand–eye multidirectional tapping task was organized into blocks, where 1 block consisted of 5 full rotations of the array, i.e., 16 target selections otherwise referred to as a sequence. Subjects were permitted to take breaks between sequences, particularly if they felt arm fatigue, to prevent fatigue effects from impacting the results.
Procedure
Subjects first conducted SOT 1 (eyes open with a stable platform) and SOT 5 (eyes closed with an A-P sway-referenced platform) both with and without GVS (Fig. 2A), with shoes removed and loosely harnessed for fall security. Three 20-s trials were collected in each configuration for a total of 12 trials (example trial data in Fig. 2B). For all trials, a lightweight GVS device was secured to their waist via a fanny pack, and they wore the HoloLens 2 compatible GVS headset restraint and electrode equipment. Immediately following the baseline GVS susceptibility task, subjects stepped off the CDP platform (shoes remaining off), doffed their harness, donned the AR headset, and were given an overview of the hand–eye task. Subjects were instructed to select each circle with their index finger, held at a 45° angle with the horizon (to ensure that the headset successfully tracked the finger joints and accurately registered target acquisitions), when the target circle turned green. They were given a demonstration of the acquisition procedure and informed that the order of selection would be the same for each sequence and for each block. Furthermore, subjects were instructed to “complete the task as quickly and accurately as possible.” The experiment was broken into 4 blocks, each with 5 trials, completing the 16-target selection sequence. To bring the subject population (each with a unique set of AR, VR, and computer experience) to a proficient level of conducting the task, a training block was conducted after familiarizing subjects with the tapping motion in AR. Following the training block, subjects repeated the task both with and without GVS turned on (blocks 2 and 3). GVS was administered in a counterbalanced order with half the subjects experiencing the GVS block before the control (no GVS) block and vice versa. At the end of the experiment, subjects were instructed to conduct a final block of the task without GVS (overview provided in Fig. 2C). This final block was collected to examine if learning effects were present after primary blocks (2 and 3) of the experiment.
CDP data collected during SOT 5 was analyzed via three metrics: inferred A-P center of mass sway (A-P sway); inferred medial-lateral center of mass sway (M-L sway); and the vestibular index, which is a measurement of the SOT 5/SOT 1 equilibrium scores (a function of inferred A-P sway). A-P sway is a commonly collected metric during SOT 5 in the literature for both astronauts recovering from flight23,29 and for subjects experiencing binaural bipolar random GVS9,10,17 due to the A-P sway referencing of the CDP platform during SOT 5. However, because this GVS waveform in the binaural bipolar configuration predominantly elicits the illusory perception of roll, M-L sway was also collected as a GVS susceptibility metric. The vestibular index has been used in the existing literature to isolate the effects of GVS on the vestibular cues for postural stability,10,17 and, so, we calculated this metric (SOT 5/SOT 1 ratio) with and without GVS. The medians of the three trials collected for each SOT were used in the following analyses, as has been done previously for studies of astronauts postflight.29 Note that when using means, the magnitudes of the results were minorly impacted, and the significance outcomes (significant or not) were unchanged.
For the 2D hand–eye multidirectional tapping task, six metrics were collected that were considered as candidates to be impacted by sensorimotor impairment. These metrics were categorized as task performance, hand strategy, and balance. Three performance metrics were assessed: accuracy of successful presses (average distance tapped from target center across all 16 targets for successful presses), number of missed presses (number of unsuccessful taps), and the mean time to press (proportional to the time to complete each trial). Two hand-motor strategy metrics were assessed: the total path length drawn from the index finger and the variation in the index finger velocity. One balance metric was assessed: head linear accelerations [the root-mean-square (RMS) of the head accelerations in a plane perpendicular to gravity (the inertial XY-plane)]. Both hand-motor strategy metrics were selected to evaluate characteristics of the way the task was performed, enabling sensitivity and additional benefits of AR hand-tracking to be assessed. The path-length metric characterizes efficiency through space, and the finger-velocity variability is a measure of compensatory control required to counter GVS sway. The balance metric was captured with the IMU onboard the HoloLens 2 device. All metrics were calculated as the median of the five trials for each block. Medians were used because the true underlying distributions of these metrics on an individual level are unknown. Once again, using the mean yields a minor change in metric magnitude with significance unchanged.
Statistical Analyses
After using the CDP device to collect A-P sway, M-L sway, and the vestibular index, we conducted paired two-tailed t-tests between each experimental block (with and without GVS) for each metric. To test for differences between experimental blocks in the 2D hand–eye multidirectional tapping task, we also carried out paired two-tailed t-tests. Because the number of missed presses is ordinal and not normally distributed, a Wilcoxon signed rank test was conducted for this metric. In evaluating differences in the performance metrics, distributions were found to be nonnormally distributed (failing the Shapiro-Wilk test of normality) and contained multiple outliers (scores greater than 1.5 times the interquartile range above the third quartile). Rather than excluding subjects based on their scores during the experiment (i.e., during the GVS and no GVS blocks), data transformations of the dependent variables and subject task proficiency were both considered. Apart from total path length, which was found to follow a log-normal distribution, data transformations did not remedy the data.
We assessed the proficiency of our subjects prior to conducting the task using a separate, independent dataset: performance metrics during the training block. Because subjects were all trained a fixed amount with no final proficiency criteria, this evaluation could not be made a priori. We found that the mean time-to-press metric was nonnormally distributed with two distinct performance clusters (shown in Fig. 3). Clusters were determined using density-based spatial clustering of applications with noise, shrinking the search radius until normality was reached for the largest cluster of subjects (search radius = 0.33 s; minimum neighbors = 1). This methodology revealed two distinct clusters and indicated that an outlier set of subjects (N = 4) were not proficient in the task during training. Further, the nonproficient group identified with this approach lay more than 3.5 standard deviations above the mean of the proficient group’s distribution. These subjects were excluded from the assessment metrics’ statistical analyses, which remediated the distributions comprising the experimental conditions. Of the nonproficient subjects, two reported no VR or AR experience, and the other two reported no AR experience and some limited VR experience. The subject population of proficient subjects (N = 16) consisted of 8 men and 8 women (mean age = 22.4 yr ± 2.7 SD, min: 19, max: 30). For all statistically significant differences found, effect sizes were calculated to assess the sensitivity of each metric.
Citation: Aerospace Medicine and Human Performance 95, 2; 10.3357/AMHP.6313.2024
RESULTS
The effectiveness of our GVS waveform was assessed by comparing the SOT metrics between the no-GVS and GVS conditions. Using the common peak-to-peak A-P sway metric during SOT 5, a statistically significant (t(19) = −3.69; P = 0.002) difference was found. Compared to Shuttle-era astronauts (N = 34) preflight and postflight (R + 0: 1–24 h23), a smaller effect size (0.63 in our study vs. 1.34 in the Shuttle-era astronauts) was found (Fig. 4A). A larger (effect size = 1.2) difference (t(19) = −4.79; P < 0.001; Fig. 4B) was found between no-GVS and GVS conditions when using peak-to-peak M-L sway as the dependent variable. Examining the vestibular index, a similar (effect size = 0.65) difference (t(19) = 3.34; P = 0.003; Fig. 4C) to A-P sway was uncovered between the no-GVS and GVS conditions. Statistical results are summarized in Table I.
Citation: Aerospace Medicine and Human Performance 95, 2; 10.3357/AMHP.6313.2024
The 2D hand–eye multidirectional tapping task revealed multiple performance decrements during GVS as compared to the no-GVS condition when evaluating proficient subjects. Evaluating subjects’ mean time to press, a statistically significant difference was uncovered between the no-GVS and GVS levels (t(15) = −3.09; P = 0.007; effect size = 0.63). The within-subject comparison of the no-GVS and GVS blocks is shown in Fig. 5A. No differences (P = 0.17) were uncovered for the accuracy of successful presses (Fig. 5B). However, differences were observed for the number of missed presses (z = –2.08.1; P = 0.038; effect size = 0.52), with the number of misses increasing in the presence of GVS (Fig. 5C).
Citation: Aerospace Medicine and Human Performance 95, 2; 10.3357/AMHP.6313.2024
Regarding hand-motor strategy metrics, neither hand-movement metrics revealed statistically significant differences. Within-subject comparisons of these two metrics are shown in Fig. 5D and Fig. 5E. Conversely, the balance metric (collected during the AR task), the RMS of subjects’ head accelerations in the inertial XY-plane, revealed a significant difference between experimental groups (t(15) = −2.91; P = 0.011). A small effect size (0.30) was found, indicating an increase in head linear accelerations with the presence of GVS (see Fig. 5F). Significance is maintained when keeping nonproficient subjects in the statistical analysis (t(19) = −2.49; P = 0.022).
DISCUSSION
The application of our GVS waveform resulted in a significant increase in A-P sway compared to the no-GVS condition and had a fraction of the effect experienced in Shuttle-era astronaut data. Because the vestibular disturbances experienced by subjects in this study were intended to be a fraction of the sensorimotor impairment experienced by astronauts 1–4 h postflight, it was expected that the effect size would be smaller than the postflight astronauts whose SOT metrics are a product of their full sensorimotor impairment of posture control.
Furthermore, the M-L sway metric revealed the largest effect size between GVS levels during SOT 5 despite the platform only pivoting in the A-P direction (while remaining fixed in the M-L direction). There is not currently a rich dataset of postflight M-L sway data in the literature for comparison to our M-L sway data; however, this information is unlikely to show a comparable effect size since the M-L sway metric is likely bolstered by the application of GVS (canal cues have been theorized to result in a mostly net roll sensation12) as opposed to a post-spaceflight reinterpretation of vestibular cues. However, this study reveals peak-to-peak M-L sway during SOT 5 to be a more sensitive metric with our GVS waveform than peak-to-peak A-P sway or the vestibular index, and this metric could prove to be useful for assessing the effects of GVS on postural control in future studies.
Concerning the 2D hand–eye multidirectional tapping tasks, both the mean-time-to-press and the number-of-missed-presses metrics showed an increase with the application of GVS, demonstrating a decrease in task performance. Additionally, the balance metric also showed a small increase in head linear accelerations with the application of GVS. Together, these results indicate that subjects may be portioning their available attentional resources to maintain balance while performing the tapping task, resulting in decreased task performance. Further, when the target was appropriately pressed, the accuracy-of-successful-presses metric indicates that GVS did not induce a bias in where the target was pressed. Therefore, accuracy for successful presses is a limited metric for determining vestibular-dominated sensorimotor impairment within the scope of this task, and the number of missed presses appears to be a more operationally relevant metric.
While GVS is known to affect balance, the observation that even a fraction of the loss experienced by shuttle-era astronauts can lead to reduced hand–eye task performance when standing has implications for mission operations decisions. Current spaceflight displays (e.g., SpaceX Dragon 2) use touchscreens, which require both precision and accuracy from the crew during disoriented states. The ability to have assessments to characterize the deficits due to sensorimotor disturbances will support decisions about astronaut readiness for the mission tasks and can also inform interface design guidelines for spaceflight displays. Further efforts to specify what the readiness threshold should be for specific operations will need to be determined. However, the hand–eye performance metrics alone are not sufficient to characterize vestibular effects and should be coupled with balance measures and additional tasks, including those that require mobility.
Limitations of the current work include the small sample size, demographic background, and age range of the subjects. We acknowledge that the subjects are not fully representative of the age, education, capabilities, experience, and other qualities of astronaut populations for which the system is intended (i.e., subjects were younger on average; had mostly not yet acquired graduate degrees; and had not undergone astronaut basic, advanced, and mission-specific training procedures). Additionally, despite the wider field of view afforded by the Microsoft HoloLens 2 system over other market devices, the field of view was still considerably restricted compared to normal human vision. Although the target array was designed to fit within the headset’s field of view at an arm’s length, subjects were free to position themselves in front of the array as close as desired. The distance between the subject and the target array, coupled with the subject’s arm length, could influence the visibility of the array and the metrics obtained. To limit these effects, subjects were instructed to stand such that the view maximized the array, while subjects were still comfortably able to reach, and their chosen distance was marked to ensure consistency across blocks and trials. Finally, our significant findings pertaining to the 2D multidirectional tapping task excluded the most nonproficient subjects. This exclusion implies that proficiency in the task must be obtained in order for the task to be a useful assessment. Using the results of the proficiency determination analysis, it may be possible to determine proficiency in real time during training in future studies.
Beyond this effort, there exists a need to map assessment task performance decrements (either from “gold standard” assessment tasks; this study’s hand–eye task’s time to press, number of missed presses, and head linear accelerations; or alternate future assessment task metrics) to operational performance in spaceflight-relevant environments following a gravity transition (e.g., lunar and Martian environments following periods of prolonged exposure to microgravity). Ground-based paradigms such as NASA’s Active Response Gravity Offload System, coupled with operational task performance metrics, may provide valuable insight into how varied off-loading levels and associated modes of sensorimotor impairment relate to operational task performance in these environments.
The current study was an exploration into the sensitivity of the 2D hand–eye multidirectional tapping task to detect vestibular-dominated sensorimotor impairment. However, future experiments considering additional sensory channels (e.g., somatosensory) of sensorimotor impairment are necessary. Furthermore, hand–eye coordination performance under vestibular impairment during increasingly dynamic operational tasks remains unexplored in the current impairment paradigm. Reaction-based or 3D target-acquisition tasks are more representative of dynamic operational tasks (e.g., geology sampling), demanding active body postures that challenge hand–eye coordination and balance, and they may prove to be more sensitive in detecting operationally relevant impairment.
To this end, the development of a larger battery of tests in an AR application is underway. The Augmented Reality Operations Readiness Assessment (AURORA) seeks to assess different operational tasks27 and features various virtually administered neurovestibular and sensorimotor assessments chosen from clinical “gold standard” evaluations and tasks currently implemented by NASA for observing crew adaptation timelines pre- and postflight on earth. Building on the findings in this effort, AURORA leverages the IMU sensor and hand–eye tracking capabilities to evaluate task performance decrements as they relate to critical mission tasks. Future work may compare the sensitivity of the alternative hand–eye coordination tasks available in the AURORA suite, such as assessments focused on reaction-based random target acquisition without predefined selection sequences or 3D target acquisition tasks.
In conclusion, the significant results of the AR-based task experiment (increase in time to completion, number of missed presses, and head linear accelerations) indicate that the proposed 2D hand–eye multidirectional tapping task in AR is capable of detecting vestibular-dominated sensorimotor impairment. These results indicate that this hand–eye assessment using the AR headset may enable the collection of dependent variables for assessing crew sensorimotor impairment. Further, performance decrements while performing a 2D hand–eye multidirectional tapping task (time to completion and number of missed presses) indicate that vestibular-dominated sensorimotor impairment (on the level of around R + 0A) may hinder crews’ ability to acquire known target locations successfully and accurately while in a static standing posture (crucial for operational tasks such as managing umbilical panels) for some time following a gravity transition. However, further testing considering additional sensory channels (e.g., somatosensory) of sensorimotor impairment is necessary to fully connect these results to operational performance.
Galvanic vestibular stimulation (GVS) modifies the afferent signals innervating the vestibular organs, which is in turn centrally processed in determining a central state estimate of position in space.
A) SOT 1 and SOT 5 configurations are presented. Both SOT 1 and SOT 5 were conducted with and without Galvanic vestibular stimulation (GVS). B) An example subject’s inferred center-of-gravity sway trace in an Earth-horizontal plane is depicted both with and without GVS during SOT 5. C) The 2D hand–eye multidirectional tapping task protocol. Each block consisted of five trials. On the right, an AR view from the subjects’ perspective is displayed. The GVS electrodes remained on the subjects during all four blocks but was only turned on for the GVS block. D) The 2D hand–eye multidirectional tapping task pattern. Not all paths are shown. All 16 targets are acquired for each sequence.
Two performance clusters found during the training block. Clusters were determined using density-based spatial clustering of applications with noise. The proficient cluster (N = 16; blue) is normally distributed (the fitted normal distribution is shown with the matching blue curve). The nonproficient cluster (N = 4) is shown in red.
A) Peak-to-peak A-P sway compared across our 20 subjects, shown individually. Median scores across the three SOT 5 trials with and without galvanic vestibular stimulation (GVS) are shown (95% CI shown with blue bars). Shuttle-era R + 0 astronauts (N = 34)23 are compared as a population preflight and postflight (95% CI shown with red bars). B) Peak-to-peak M-L sway (95% CI shown with blue bars). C) Vestibular index with and without GVS (95% CI shown with blue bars). Asterisks denote significance level. Individual subject markers are consistent throughout this figure and Fig. 5, and horizontal jitter was applied to individuals.
A) Average times to press between the blocks with no Galvanic vestibular stimulation (no-GVS) and with GVS. The four subjects excluded from the statistical analysis (transparent blue bars) are highlighted in red and have been made transparent. B) The average distance pressed from the center of the target for successful presses, compared between the no-GVS and GVS blocks. C) Number of misses between the no-GVS and GVS blocks. D) The variation in index finger velocity between the no-GVS and GVS blocks. E) The total path length between the no-GVS and GVS blocks. F) The root mean square (RMS) of XY-plane acceleration compared between the no-GVS and GVS blocks. Population mean 95% confidence intervals are displayed with transparent blue bars.
Contributor Notes
