Leveraging Space-Flown Technologies to Deliver Healthcare with Holographic Physical Examinations
INTRODUCTION: Musculoskeletal injuries are one of the more common injuries in spaceflight. Physical assessment of an injury is essential for diagnosis and treatment. Unfortunately, when musculoskeletal injuries occur in space, the flight surgeon is limited to two-dimensional videoconferencing and, potentially, observations made by the crew medical officer. To address these limitations, we investigated the feasibility of performing physical examinations on a three-dimensional augmented reality projection using a mixed-reality headset, specifically evaluating a standard shoulder examination. METHODS: A simulated patient interaction was set up between Western University in London, Ontario, Canada, and Huntsville, AL, United States. The exam was performed by a medical student, and a healthy adult man volunteered to enable the physical exam. RESULTS: All parts of the standard shoulder physical examination according to the Bates Guide to the Physical Exam were performed with holoportation. Adaptation was required for the palpation and some special tests. DISCUSSION: All parts of the physical exam were able to be completed. The true to anatomical size of the holograms permitted improved inspection of the anatomy compared to traditional videoconferencing. Palpation was completed by instructing the patient to palpate themselves and comment on relevant findings asked by the examiner. Range of motion and special tests for specific pathologies were also able to be completed with some modifications due to the examiner not being present to provide resistance. Future work should aim to improve the graphics, physician communication, and haptic feedback during holoportation. Levschuk A, Whittal J, Trejos AL, Sirek A. Leveraging space-flown technologies to deliver healthcare with holographic physical examinations. Aerosp Med Hum Perform. 2024; 95(4):214–218.
The remote and extreme nature of space presents unique practical constraints for the provision of healthcare to astronauts. Traditionally, government astronauts have been held to strict health standards to decrease overall risk of medical events during spaceflight. Nevertheless, the hostile environment of space continues to result in medical events which are managed in a virtual fashion by terrestrial flight surgeons. In the near future, increased interest from the private spaceflight sector will see the private space economy market valuation grow to an estimated $1 trillion by 2040.1 With the anticipated increase in humans accessing space, the concomitant increase of potential comorbidities or medical events in low Earth orbit is anticipated to increase. Due to the perceived value of commercial space, individuals may expect terrestrial-quality healthcare as a common standard in low Earth orbit. Furthermore, government deep-space missions will increase the need for autonomous as well as remote-care capabilities.
The role of the flight surgeon during a space mission is the monitoring and consultant management of medical situations as required. Flight surgeons are traditionally terrestrially based and a model has evolved whereby designated crew medical officers are provided with additional training to assist the flight surgeon to provide care in orbit. While the crew medical officer may be a physician, this is not always the case. One limiting factor for the care provision by flight surgeons in the current paradigm is the lack of a complete clinical encounter, including the physical examination. Flight surgeons are constrained by the same limitations as terrestrial physicians reliant on telemedicine and/or videoconferences for clinical encounters. These technologies provide altered anatomical scale, the inability to touch the patient, and a lack of personal connection and separation, serving to attenuate the quality of physical exams flight surgeons and terrestrial physicians conduct remotely.
A potential solution to some of these limitations is the integration of augmented reality (AR), or virtual-reality (VR) headsets—or together called “mixed-reality” technologies. Mixed-reality technologies can blend modern AR communication technology with healthcare delivery hurdles; however, there has been little research into leveraging holographic encounters to facilitate better remote physical exams. Following the patient interview, the physical exam is the most important tool in a physician’s toolbox. Traditional videoconferencing distills the physical exam to inspection of the patient, while the anatomy of the patient being observed is not even on a correct anatomical scale. Mixed-reality holographic clinical encounters could help bridge the gap between virtual and in-person physical exams and help restore the quality of physical exams performed in a virtual setting.
A potential application of these technologies is to assess musculoskeletal (MSK) health virtually. MSK injuries are not uncommon in space and occur at a rate of roughly 0.021 per flight day for every astronaut.12 Muscle atrophy and deconditioning due to prolonged muscle unloading from low gravity during longer spaceflight contribute to joint injury risk and degradative joint changes.13 Thus, as spaceflight becomes more commonplace, the unloading effects on muscles and joints will become more prominent, and examining MSK health will be critical.
Since the first generation of mixed-reality commercial headset was released in 2016, there has been a variety of research efforts dedicated to applying AR, VR, and mixed-reality technology to current clinical methods. Researchers have used AR to display alphanumerical data, such as patient history, vitals, and lab results.2 This is advantageous for sharing data with multiple people who are wearing synchronized headsets and for accessing hands-free data when it is important to maintain sterility. Another clinical use for mixed reality is to stream or overlay live or previously captured images into the field of vision of the clinician. Common image streams include x-ray,3 endoscopy,4 ultrasound,5 and MRI.6 AR has proven instrumental in fostering collaboration among physicians by enabling remote medical experts to seamlessly connect and share the perspective of their counterparts. This capability is exemplified by its capacity to facilitate remote physicians in joining a collaborative session, granting them access to the viewpoint of their collaborating peers and thereby enhancing real-time, interactive medical consultations.7 AR has also been used for surgical navigation,8,9 intravenous injection guidance, and medical trainee education.10,11
Considering these benefits, the objective of this work was to investigate the feasibility of using novel holographic communication technology to conduct mixed-reality, holographic, remote physical exams in a terrestrial setting to set a foundation for expanding clinical care options in spaceflight. This technology was originally demonstrated during NASA missions and then in a bidirectional fashion during the Axiom-1 mission to the International Space Station in 2022. The Axiom-1 technical demonstration included a blinded evaluation by an Axiom flight surgeon performing a mock examination of a simulated medical condition on orbit to evaluate the benefits and drawbacks of the technology for clinical encounters. This manuscript describes the subsequent evaluation of bidirectional holographic communication at Western University in London, Ontario, Canada, for the purpose of enhanced virtual care delivery.
METHODS
A bidirectional holographic clinical evaluation was set up between a medical student in London, Ontario, Canada, and a healthy adult man in Huntsville, AL, United States. Informed consent was obtained from the simulated patient and medical examiner. All data was collected noninvasively and was a part of standard engineering testing that would otherwise be done during testing of HoloConnect software (Aexa Aerospace, Houston, TX, United States). No clinical scenario was used and both parties were aware that the examination was being performed on a normal upper extremity.
Both the examiner and the subject used a mixed-reality headset (HoloLens 2, Microsoft, Redmond, WA, United States) and 3D scanning camera (Kinetic, Microsoft) to create and reproduce holograms in their environments. HoloConnect software (Aexa Aerospace) was used to create and display the holograms and enable communication.
The HoloLens 2 is an untethered and self-contained mixed-reality headset. It is a wearable device that combines both virtual and augmented reality to allow users to interact with holographic objects and digital information in the physical world around them. The Kinect cameras on both ends of the bidirectional holoportation were used to capture the motions of the examiner and the simulated patient (Fig. 1). These cameras have optimal ranges between 1–3 m from the camera and within a 57° horizontal and 43° vertical field of vision. The Kinect LiDAR motion capture cameras, as well as the LiDAR cameras integrated into the HoloLens, permit real-time three-dimensional (3D) scanning and depth perception which allows both parties to interact with their own environment as well as the other party during the holoportation. The camera system allows both parties to move freely within their acceptable optimal motion capture zone, thereby permitting both ends to continue to interact with their own environment while engaging in holoportation.
Citation: Aerospace Medicine and Human Performance 95, 4; 10.3357/AMHP.6397.2024
The format of the physical examination followed the exam outlined in the Bates Guide to Physical Examination and History Taking and consisted of inspection of the joint, palpation, and moving of the joint. The exam was slightly modified due to the remote nature of the exam, but still consisted of inspection, palpation, and range of motion testing.
RESULTS
The virtual shoulder exam began with inspection of the shoulder as well as the joints above and below. Similar to traditional videoconferencing, the examiner was able to view all angles and anatomy of the shoulder. However, compared to the small-scale size of patients on computer screens in traditional videoconferencing, the holographic anatomy of the patient was true to anatomical size. The realistic anatomy conferred by the hologram provided the examiner with an increased ability to observe the anatomy; therefore, abnormalities such as swelling, erythema, atrophy, or deformations would have been more obvious if they had been present. Additionally, the realistic anatomy made appreciating muscle tone and bulk closer to an in-person encounter (Fig. 2). In a scenario where there is atrophy or an axillary nerve palsy, anatomical changes in the shoulder would have been obvious.
Citation: Aerospace Medicine and Human Performance 95, 4; 10.3357/AMHP.6397.2024
Palpation was performed following inspection. As physical palpation of the hologram was not possible, a clear set of directions for subject self-palpation were developed prior to the exam. In this manner, the examiner could clearly guide the subject through self-palpation. The subject was instructed to palpate with appropriately firm pressure using their second and third fingers. Anatomical landmarks such as the clavicle, coracoid process, acromion process, spine of the scapular, and biceps tendon were used to ensure that the patient was correctly palpating the full anatomy of the shoulder.
For individuals with no medical background, understanding anatomy and relevant nomenclature is not realistic. Thus, alternative physically descriptive names were used to describe anatomical landmarks. Imperfect communication combined with a lack of medical background could lead to errors in self-palpation. Furthermore, it was noted that a person who is experiencing, and is aware of, their own pain may be hesitant to appropriately palpate themselves and report unpleasant findings back to the examiner. Finally, the ability of an individual to palpate their own shoulder is largely dependent on the degree of mobility in their contralateral arm. The spine, medial border, and inferior border of the scapula could be particularly difficult for a mobility compromised person to palpate. Overall, the lack of true palpation was felt to be a shortcoming of this technology despite attempts to guide self-palpation of the subject.
Range of motion in all six degrees of freedom were completed successfully. The true anatomical size of the holograms made potential key findings, such as momentary subluxation and scapular dyskinesia, easy to assess (Fig. 3).
Citation: Aerospace Medicine and Human Performance 95, 4; 10.3357/AMHP.6397.2024
It was noted during the evaluation that the HoloConnect software was not as high-definition as popular videoconferencing platforms such as Zoom, Microsoft Teams, and WebEx. However, the granularity of the holograms was sufficient such that individual forearm muscles on a person could still be appreciated.
DISCUSSION
A common drawback of virtual health appointments is that telemedicine feels less personal and poses a threat to the gravity of the physician-patient relationship. This deficit may be the reason why virtual environments reduce rapport between physicians and patients and make alliance building more challenging.14 Holographic encounters can help overcome this problem by creating a tangible sense of presence between physicians and patients due to their ability to place a life-size 3D representation of the individual in anatomical size within the physical space that one is occupying.
This technology demonstrated effective inspection, aided by the life-sized hologram and despite the lower resolution of imagery compared to modern two-dimensional comparable technologies. Range of motion testing was also very effective with this technology. Drawbacks were noted with the subject-directed self-palpation. The lack of ability to feel the tissue and directly examine the individual was a major drawback to the holographic examination.
Special testing of individual shoulder anatomy is commonly done at the end of a physical exam based upon the general findings and to elicit a greater specificity of the presenting shoulder pathology. Nearly all of the special tests for the shoulder require the examiner to be physically present with the patient in order to provide resistance or physically manipulate the limb being examined. The Hawking’s test and Neer’s test for shoulder impingement syndrome, as well as the acromioclavicular joint scarf test, all require an examiner to manipulate the limb of the patient, and this cannot be performed in a virtual setting. Both Yerguson’s and Speed’s tests for bicep tendinopathy require the examiner to provide resistance and thus cannot be done in a virtual setting. A modified version of the Speed’s test could be done by having a patient hold a resistance creating device, such as an exercise band, in their hand and having them flex their arm while keeping their hand in a supinated position. The subscapularis liftoff, infraspinatus external rotation, and supraspinatus “empty cans” tests all require the examiner to provide resistance to the patient’s movement. Modified versions of these tests could be done by having the patient push against an unmovable wall during the subscapularis lift off test and the infraspinatus external rotation test, and having the patient hold a resistance device in their hand during the supraspinatus empty can test. Finally, the glenohumeral instability test normally requires the examiner to provoke the joint into a feeling of instability; however, some researchers have suggested that this test can be done virtually by having the patient place their arm in a “throwing position” to evoke a feeling of instability.15 However, this may not be recommended during spaceflight because of the possibility of shoulder subluxation or dislocation. Unfortunately, in addition to requiring special technology, completion of these tests would require specialized equipment, further limiting the ability of the technology for virtual care delivery.
In this technology evaluation, the bidirectional 3D AR communication was shown to facilitate virtual physical MSK exams. Compared to traditional videoconferencing, augmented reality communication could confer advantages that improve the quality of physical exams, as well as improve the physician-patient relationship in a virtual setting. Moreover, the directional 3D augmented reality communication demonstrated here could feasibly be used during spaceflight because the technical specifications of the technologies used here are all available in space.
The authors note that improving the quality of graphics would significantly close the gap between in-person and holoportation visits. Additionally, integration of haptic feedback into holoportation would enhance its realism and give physicians the ability to palpate patients remotely.
Many remote areas in the world do not have access to specialty physicians, but holoportation could offer the opportunity for remote areas to access consultation from physicians in a more wholesome manner. Holographic communication remains an exciting and potentially transformative tool poised to redefine virtual medicine by seamlessly transcending distance, enhancing collaboration, and enabling comprehensive, real-time healthcare delivery to astronauts and people on Earth, in the most remote and challenging settings.
Materials setup. Both the medical examiner and the simulated patient used a Kinect camera to record their motion. Video streams were connected to Holoconnect software on a local PC. HoloLens 2 headsets running Holoconnect software worn by both people were connected to the same Holoconnect server. Live video from the other’s Kinect camera was displayed as a life size 3D augmented reality projection in the other’s environment.
Holographic inspection. Holoportation image allows the forearm muscle definition to be appreciated by the examiner.
Holographic range of motion. Examiner guiding participant through flexion range of motion testing.
Contributor Notes
