Augmented Reality

Index:

Augmented Reality (AR) superimposes 3D virtual objects to physical objects in real space. In other words, it can provide real-time virtual information directly in our environment. This tool has been lately adopted in many medical areas with exciting benefits. In this group, we also believe in the potential of AR, and so, we have developed several AR applications applied to the clinical field. Our research has been done in collaboration with Hospital Gregorio Marañón and combines this technology with 3D printing.

How does it work? We create 3D virtual models of patients’ internal structures using their medical images. On the other hand, we design patient-specific 3D printed surgical guides that can fit into a specific region of the patient’s anatomy. Attached to these guides, we include a 3D printed reference that the AR system can recognize. As a result, we can project the patients’ inside over themselves to create the illusion that we are looking through their skin. With that, we attempt to facilitate surgical planning, assist during patient communication, and enhance advanced surgeries.

HoloLens 2

2023 – HoloLens 2 + 3D Slicer

At Universidad Carlos III de Madrid, we are mainly focused on Microsoft HoloLens 2. To date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device. Our most recent project presents a novel connection approach between 3D Slicer and Microsoft HoloLens 2 using OpenIGTLink. This project has been developed in collaboration with Perk Lab from Queen’s University. The solution is implemented in a 3 elements system: A Microsoft HoloLens 2 headset, the Unity software, and the 3D Slicer platform.

Specifically, we developed an application that transfers geometrical transform and image messages between the platforms. It sends the position of a plane from HoloLens 2 to 3D Slicer. The computer software reslices the CT volume of the patient at this specific pose and sends it back to the AR glasses. This interaction is performed in real-time, so that the user can directly visualize CT reslices of a patient overlayed to the real world. Below you can see a demonstration for pedicle screw placement planning.

All our work is publicly available in this GitHub repository. This repository contains all the resources and code needed to replicate our work on a new computer.

  • Pose-Díez-de-la-Lastra A, Ungi T, Morton D, Fichtinger G, Pascau J. Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning. Int J CARS (2023). [doi] [pdf, CC BY License] – Impact Factor: 3.421 (Q2)

2022 – Orthopedic oncological surgeries

We wanted to go a step forward from the previously used Smartphones and HoloLens 1. Our first task was to evaluate their tracking accuracy and ergonomics compared to HoloLens 1 during experimental and surgical scenarios. Check our free-access paper below to find out our results.

AR visualization on Microsoft HoloLens 2 from surgeon’s perspective during a surgical intervention.
The top left corner shows an external view of the surgical field
  • A. Pose-Díez-de-la-Lastra, R. Moreta-Martinez, M. García-Sevilla, D. García-Mato, J. A. Calvo, L. Mediavilla-Santos, R. Pérez-Mañanes, F. von Haxthausen, and J. Pascau. HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions. Sensors 2022, 22, 4915. [doi] [Open Access under the Creative Commons Attribution-NonCommercial-NoDerivs License]

In addition, we have analyzed the HoloLens 2’s tracking performance using two different types of AR markers: a pattern-based and a spheres-based marker. The first recognition system is based on pattern identification using Vuforia. This system uses computer vision algorithms to identify visual patterns in an RGB image acquired with the HoloLens 2’s camera. The second system uses HMD’s RGB and depth camera to detect another AR marker made of three retro-reflective spheres. We tested both strategies in an experimental scenario and during real surgery. Our results were presented at Computer Assisted Radiology and Surgery (CARS) congress, celebrated in Tokyo during June 2022.

(a) Pattern-based, and (b) spheres-based augmented reality markers.
  • A. Pose-Díez-de-la-Lastra, R. Moreta-Martinez, F. con Haxthausen, M. García-Sevilla, L. Hernández-Álvarez, J. A. Calvo, R. Pérez-Mañanes, and J. Pascau. Analysis of tracking performance with two different HoloLens 2 augmented reality markers for orthopedic oncological surgeries. In: CARS 2022 – Computer Assisted Radiology and Surgery Proceedings of the 36th International Congress and Exhibition, Tokyo, Japan, June 7-11, 2022. Int J CARS 17, 1-147 (2022). [doi]

HoloLens 1

2018 – Orthopedic oncological surgeries

Augmented reality can be an interesting technology for clinical scenarios as an alternative to conventional surgical navigation. However, the registration between augmented data and real-world spaces is a limiting factor. In this work, we propose a method based on desktop 3D printing to create patient-specific tools containing a visual pattern that enables automatic registration. This specific tool fits the patient only in the location it was designed for, avoiding placement errors. This solution has been developed as a software application running on Microsoft HoloLens.

The workflow was validated during the surgical intervention of a patient presenting an extraosseous Ewing’s sarcoma. The application allowed physicians to visualize the skin, bone, and tumor location overlaid on the patient.

Smartphones and Tablets

2024 – Microtia reconstruction

In addition to orthopedic oncology, the biomedical engineers from the IGT group have also joined efforts with Hospital General Universitario Gregorio Marañón to treat microtia. This is an congenital malformation in which the ears develop incorrectly, resulting in tiny or even non-existent ears. This condition, which affects 1 in every 5,000 births, can not only compromise the patient’s hearing ability but also interfere with everyday situations such as wearing glasses.

Patient presenting left hemifacial microsomia with a lobular type microtia. (A) Frontal view. (B) Right side view. (C) Left side view showing microtia

The affected ear can be reconstructed during surgery using the patient’s own costal cartilage. Traditionally, the contralateral healthy ear has been used as a template to trace the ideal shape of the new ear in the extracted cartilage, while anatomical measurements usually guide its placement on the patient’s head to ensure symmetrical outcomes. However, this technique is time-consuming, subjective, and highly dependent on the surgeon’s expertise, making it susceptible to shape errors and misplacement.

We present an innovative clinical workflow that combines 3D printing and augmented reality (AR) to increase the objectivity and reproducibility of these procedures. Specifically, we introduce patient-specific 3D cutting templates and remodeling molds to carve and construct the cartilaginous framework that will form the new ear. Moreover, we developed an in-house AR application compatible with any commercial Android tablet. It precisely guides the positioning of the new ear during surgery, ensuring symmetrical alignment with the healthy one and avoiding time-consuming intraoperative linear or angular measurements. Our solution was evaluated in one case, first with controlled experiments in a simulation scenario and finally during surgery.

Click on the image above to see a video of the whole process available at: https://www.youtube.com/watch?v=V2iKB0gPfJY&t=33s&ab_channel=IGTResearchGroupUC3M
  • Alberto Díez-Montiel, Alicia Pose-Díez-de-la-Lastra, Alba González-Álvarez, José-Ignacio Salmerón, Javier Pascau, Santiago Ochandiano. Tablet-based Augmented reality and 3D printed templates in fully guided Microtia Reconstruction: a clinical workflow. 3D Printing in Medicine 10, 17 (2024). [doi] [pdf, Open Access under the Creative Commons Attribution 4.0 International License]

2020 – DIY app tutorial

At Universidad Carlos III de Madrid and Hospital General Universitario Gregorio Marañón, we have implemented a protocol to develop your own smartphone app combining augmented reality and 3D printing for its use in the medical field. The method describes detailed steps to go from the patient medical image to the development of a smartphone app using free and multi-platform software. This protocol is expected to accelerate the adoption of AR and 3DP technologies by medical professionals. No prior knowledge is required.

Check out the protocol following this link: https://www.jove.com/v/60618/ (Journal of Visualized Experiments)

It will take you to a step-by-step guide, including open access to all the material.

Workflow to go from the medical image to an Augmented Reality smartphone application

In the following video, we show you some of its applications:

2021 – Orthopedic oncological surgeries

The standard treatment of bone and soft tissue tumors includes complete surgical resection. This procedure still depends on the surgeon’s previous experience and subjective judgment to achieve complete tumor removal, ensuring a safety margin of healthy tissue. For this reason, it is essential to efficiently plan the surgical approach preoperatively to improve surgical outcomes, leave enough surgical margin, and reduce the risk of local recurrence or metastasis

Proposed step-by-step orthopedics oncology medical workflow.

In this project, we propose a new surgical framework that combines augmented reality and 3D printing for their integration in orthopedic oncological surgeries. It includes 3D-printed patient-specific models and tools and a novel smartphone-based AR application. This app displays the internal anatomical structures of the patient in real-time. The system is part of a new surgical workflow in which both technologies assist in preoperative planning, patient communication, and surgical intervention.

Integration of the augmented reality system at each step of the medical workflow. (a,b) A physician using ARHealth during surgical planning of a patient; (c,d) Medical staff explaining a patient her condition using ARHealth; (e) One physician using ARHealth during the surgical intervention of one patient after the surgical guide was placed on the patient, and other surgeon delimiting surgical margin while looking at the AR-display. (b,d,f) Smartphone visualization at the same moment of (a,c,e), respectively.
  • R. Moreta-Martinez, A. Pose-Díez-de-la-Lastra, J. A. Calvo-Haro, L. Mediavilla-Santos, R. Pérez-Mañanes, and J. Pascau. Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation. Sensors, 21(4), 1370 (2021). [doi] [pdf, Open Access under the Creative Commons Attribution 4.0 International License]

Other Applications

2021 – AR in Craniosynostosis

Link to section

Related publications

  • Alberto Díez-Montiel, Alicia Pose-Díez-de-la-Lastra, Alba González-Álvarez, José-Ignacio Salmerón, Javier Pascau, Santiago Ochandiano. Tablet-based Augmented reality and 3D printed templates in fully guided Microtia Reconstruction: a clinical workflow. 3D Printing in Medicine 10, 17 (2024). [doi] [pdf, Open Access under the Creative Commons Attribution 4.0 International License]– Impact Factor: 3.7 (Q1)
  • Pose-Díez-de-la-Lastra A, Ungi T, Morton D, Fichtinger G, Pascau J. Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning. Int J CARS (2023). [doi] [pdf, CC BY License] – Impact Factor: 3.421 (Q2)
  • Pose-Díez-de-la-Lastra A, Moreta-Martinez R, García-Sevilla M, García-Mato D, Calvo-Haro JA, Mediavilla-Santos L, Pérez-Mañanes R, von Haxthausen F, and Pascau J. HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions. Sensors, 22(13), 4915 (2022). [doi] [Open Access under the Creative Commons Attribution 4.0 International License] – Impact Factor: 3.576 (Q1).
  • R. Moreta-Martinez, D. García-Mato, M. García-Sevilla, R. Pérez-Mañanes, J. A. Calvo-Haro, J. Pascau. Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone. J. Vis. Exp., 155, e60618 (2020). [doi] [UC3M Research Portal] [pdf, Open Access under the Creative Commons Attribution-NonCommercial-NoDerivs License]
  • R. Moreta-Martinez, A. Pose-Díez-de-la-Lastra, J. A. Calvo-Haro, L. Mediavilla-Santos, R. Pérez-Mañanes, and J. Pascau. Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation. Sensors, 21(4), 1370 (2021). [doi] [pdf, Open Access under the Creative Commons Attribution 4.0 International License]
  • R. Moreta-Martinez, D. García-Mato, M. García-Sevilla, R. Pérez-Mañanes, J. A. Calvo, and J. Pascau. Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference. Healthcare Technology Letters, 1–5 (2018). [doi] [UC3M Research Portal] [pdf, Open Access under the Creative Commons Attribution-NonCommercial-NoDerivs License]
  • D. García-Mato, R. Moreta-Martinez, M. García-Sevilla, S. Ochandiano, R. García-Leal, R. Pérez-Mañanes, J. A. Calvo-Haro, J.I. Salmerón, J. Pascau. Augmented reality visualization for craniosynostosis surgery. Comput. Methods Biomech. Biomed. Eng. Imaging Vis., 1–8 (2020). [doi]