Optical and hyperspectral image analysis for image-guided surgery

For interventional imaging and surgical navigation, it is important that the approach allows for accurate patient tracking to compensate for patient movements particularly during delicate procedures (such as spinal fixation surgeries), and that the approach provides enhanced vision to give the surgeon assistance and precise information about the internal anatomy, as is needed when operating on tumors. For her research, Francesca Manni investigated two techniques to improve tracking and visual assistance for surgeons.

Patient localization and guidance

First, Manni considered the use of hyperspectral imaging (HSI) to aid in patient localization and monitoring for spinal surgery. Second, the researcher then looked at hyperspectral tissue characterization as an approach towards improving surgeon guidance during tissue resection in surgical oncology.

The problem of using imaging techniques for surgical guidance are twofold. For patient positioning, accuracy relies on continuous tracking of the patient, without loss of this accuracy throughout a procedure. If positioning is done with markers or dynamic reference frames (DRF), both markers and frames can be dislodged or obscured during the surgical procedure, resulting in loss of navigational feedback and accuracy.

In addition, and for cases with internal navigation for oncology surgery, surgical navigation is crucial for tissue characterization. During tumor resection, the success of a surgical tumor removal relies on the detection of malignant tumor boundaries. Manni concentrated on improving the surgical guidance for the two medical use cases. First, she turned to optical and hyperspectral imaging to achieve accurate spinal surgery, and second, she looked at the use of HSI for tumor identification for advancing oncology surgery.

Spinal surgery cases

First, with optical imaging, spine features are detected from open spinal surgery cases, in an augmented-reality navigation system. Camera images are used to identify and match spinal landmarks in different camera views, overcoming the use of reference markers for patient tracking.

Focusing on minimally invasive surgery (MIS), Manni investigated anatomical landmarks on the skin of the patients. A new markerless framework is proposed which is based on skin feature recognition and localization using optical cameras.

The spine features detected by the cameras reach an overall 3D localization error lower than 0.5 mm on 23 patients, while the skin features were localized with an error less than 0.3 mm in 8 clinical cases.

Next, Manni used HSI to augment the detected skin features performing a human subject study with a novel snapshot hyperspectral (HS) camera prototype. Convolutional neural networks (CNNs) were applied to learn and detect skin features, which were localized with an average error of only 0.25 mm with respect to the ground truth based on optical markers. The resulting accuracy is clinically acceptable, and well in line with the state-of-the-art robotic systems.

Image-guided surgery

Second, Manni studied the potential of HSI for tissue characterization during image-guided surgery. HSI is a reflectance-based imaging modality that captures the diffuse reflectance spectra across a wide spectral range in the spectrum. It’s also a non-ionizing imaging technique beyond the visible spectrum and can identify the tumor tissue to expedite the clinical workflow for tumor resection.

Manni proposes the use machine learning and deep learning frameworks, initially on colon and tongue cancer datasets, for brain tumor classification by extracting both spectral and spatial information. Using 12 in-vivo HS acquisitions, the system reaches an overall accuracy of 80% in classifying tumor, healthy tissue, and blood vessels, all together, outperforming state-of-the art approaches.

Replacements for marker-based approaches

The contributions from Manni’s research offer novel solutions for replacing marker-based approaches for markerless tracking using optical and HSI analysis.

She demonstrates the potential of HSI for tumor classification, providing an intra-operative feedback to the surgeon for objective assessment of cancer. Results show that HSI could be an important technique for surgical guidance in complex interventions to improve the surgical outcome.

Finally, with the advance of robotics in surgery, camera-based and HSI-based technologies could be employed for surgical guidance with multimodal imaging techniques, as a radiation-free application for robot-assisted surgery.

Title of PhD thesis: Optical and hyperspectral image analysis for image-guided surgery. Supervisors:  Peter de With, Fons van der Sommen, and Sveta Zinger.