National Biocomputation Center

A Case Study Using the Virtual Environment for Resonstructive Surgery

KEVIN MONTGOMERY PhD, Michael Stephanides MD, Stephen Schendel MD, Muriel Ross PhD

National Biocomputation Center
Stanford University
Stanford, CA 94305

NASA Ames Research Center
Moffett Field, CA 94035

ABSTRACT_______________

This paper details the use of a Virtual Environment for Reconstructive Surgery (VERS) in the case of a 17 year-old boy with a severe facial defect arising from the removal of a soft-tissue tumor. Computed tomography (CT) scans were taken of the patient, the data were segmented, a mesh was generated, and this patient-specific mesh was used in a virtual environment by the surgeons for preoperative visualization of the defect, planning of the surgery, and production of a custom surgical template to aid in repairing the defect. This paper details the case of this patient, provides a background on the virtual environment technology used, discusses the difficulties encountered, and describes the lessons learned.

Keywords: computer applications, life and medical sciences, surgical planning.

INTRODUCTION___________

The Virtual Environment for Reconstructive Surgery (VERS) project was started three years ago between the NASA Ames Biocomputation Center and the Stanford University Department of Reconstructive Surgery. Its initial aims were to use the 3D reconstruction and visualization technologies that were developed for space-related research toward providing surgeons with visualization capabilities to aid in preoperative planning. Since then, the scope of the work, and our organizations, have expanded substantially.

NASA and Stanford have recently founded a National Biocomputation Center to expand this work to adapt and deploy many of NASA's technologies in the broader area of medicine. The new Center ties together many different groups throughout NASA, with many groups within Stanford (plastic surgery, neurosurgery, education, radiology, computer science, engineering) and external organizations to push the state of the art in computation and visualization to revolutionize medicine.

The VERS project is only one of several research areas currently under development and it has since expanded its scope to allow full visualization and interaction between the surgeon and the reconstructed data of their patient. The result is a complete surgical planning station that allows the surgeon to visualize their patient's data, extract quantitative information (such as distances and angles) directly from the dataset, to actually simulate the surgical procedure for training purposes, and to even generate patient-specific custom templates that can be used in the surgery itself.

This paper will briefly discuss the technologies behind this project, detail a case study for which this system proved invaluable, and explore the lessons learned in the production and use of a virtual environment for reconstructive surgery.

CASE BACKGROUND______

This case involves a 17 year-old hispanic boy. At age 9 he had a fast-growing soft tissue tumor removed from under his left eye. Because this was an advanced and very aggressive tumor, the surgeon removed all bones and much of the soft tissue on the left side of the boy's face (including the eye) in order to be certain to fully remove the tumor (see figure 1a-b).

Now, at age 17, surgeons at the Department of Functional Restoration and Division of Plastic and Reconstructive Surgery are planning the reconstruction of his face. Because this is a particularly unusual and difficult case, we sought to use advanced visualization to aid in the preparation of this surgery.

METHODS________________

Reconstruction
First, a computed tomography (CT) scan of the boy was performed. This data was then transferred from the Radiology Department to our SGI Onyx InfiniteReality server. Once there, the data are translated (if necessary) from their vendor-specific format into the standard DICOM format.

Next, in-house 3D reconstruction software reads the successive DICOM slice images, allows viewing of the raw data, performs segmentation using morphological techniques and connected components analysis, generates a mesh using a Marching Cubes algorithm, allows for automated and semiautomated artifact removal, and permits mesh reduction through a number of techniques. The resulting high resolution meshes are then used for preoperative visualization, where great detail is required (see figures 1a-d).

There were 2.5 million polygons in the skull model and 1.5 million polygons comprising the face. For preoperative visualization, we typically view these images in stereo, so we require a minimum of 8 million polygons per frame. With currently available technology, this much information could clearly not be rendered in real-time.

Therefore, the meshes are reduced for interactive visualization purposes in the VERS environment. In the case of this dataset, both the skull and face were reduced to roughly 500K polygons for interactive visualization. Our mesh reduction techniques preserve as much detail as possible, while reducing the number of triangles required to represent the surface. However, this was the minimum size that was judged by the surgeons that still provided sufficient detail.

Virtual Environment for Reconstructive Surgery
The Virtual Environment for Reconstructive Surgery consists of the following components:

Silicon Graphics Onyx InfiniteReality workstation
FakeSpace Immersive Workbench display system
Polhemus FasTrak stylus
StereoGraphics CrystalEyes stereo glasses
Sense8 WorldToolKit

The WorldToolKit application reads in the given mesh files and allows the user to visualize, measure, interact, and manipulate the data of their patient. A number of "virtual tools" are implemented:

Selection/moving tool- allows the user to grab and move/rotate an object
Marker tool- allows the used to lay down markers on the surface of an object
Lighting tool- a "spotlight" that allows more precise localization of lighting

A number of operations are also available. There are operations for performance characterization and scene graph display, manipulating object attributes (color, transparency), turning on/off display of objects, setting standard viewpoints, rendering modes (wireframe, solid), saving the scenegraph in VRML, dumping a screen image, deleting all markers, and on-line help. In addition, a number of operations use the locations of the markers: the measure operation measures and displays the distances (linear and surface) and angles between each of the markers; the cut operation subdivides the mesh between to the markers; the reflect operation allows the user to make a duplicate, but reversed object, etc.

Difficulties and Solutions
The major difficulties identified in producing such an environment lie in the desire to maintain a high fidelity of the dataset, but to provide a reasonable frame rate for the interactive virtual environment. Note that this is a very different paradigm than occurs in most virtual environment applications. In many application areas (such as architectural walkthrough), many objects may be culled because they are currently out of view of the user. In these cases, a scene graph model is of great benefit to decrease rendering to only those polygons that may be visible in the scene. Also, a bounding box test can be used for aiding in visibility or selection testing and each object is relatively small (under a thousand polygons). In addition, maintaining multiple levels of detail of an object can further decrease rendering requirements. These techniques can allow the elimination of the consideration of many objects in the scene and further increase display and selection speed.

In a surgical virtual environment application [1-6,8-14], there is typically one, very high resolution object to be viewed. All of the object is in view, all of the time. Even at 500K polygons and with a high-end graphics supercomputer providing a theoretical peak of 10M polygons/second, we have a theoretical maximum framerate of 20 fps. Our observed performance of 4 fps (without display-list objects) or 8 fps (with display-list objects) was considerably less.

A number of techniques were employed to ameliorate this situation. First, the original meshes of millions of polygons need to be reduced dramatically. Curvature-based (chord height) methods (which reduce the number of polygons in areas of low curvature) initially appeared straightforward, but these techniques, when applied liberally, produced large faceted (flat) areas, particularly on the forehead. By not constraining the size of the conjoined regions, such areas of low curvature were over decimated. Employing polygon size metrics (to eliminate small polygons because they contribute little to the overall visualization) worked well and solved the above shortcomings. However, these techniques, when applied liberally, would tend to erode away sharp edges of the model (such as near the sutures of the skull), since small polygons are often present along such sharp edges. Finally, a technique that allowed user control of the weighting of these two functions allowed us to decrease polygon count, but preserve sufficient fidelity. This is clearly an area that warrants more research.

Second, once the original meshes have been decreased in size as much as the surgeons deemed clinically accurate, the issue of how to render these meshes efficiently becomes important. In our case, the large geometry was split up into smaller "slabs". In this way, the slab comprising the rear of the skull could be viewed in lower resolution using a level-of-detail mechanism. Note that a rear slab can not be completely culled from the scene because there may exist holes in the slabs closer to the user (through the eyes, for instance).

In addition, this slab-based approach also dramatically improved prebuild time (time to optimize the geometry for rendering and create display list objects- an O(n^2) operation) from 20 minutes to 2 minutes. When preprocessing required 20 minutes, the system was very difficult to use and to test due to this time delay. After the reduction in time, preprocessing, and hence its great rendering advantages, could be realized.

Moreover, this approach allowed for an optimized selection mechanism. When laying down markers, the marker tool casts a ray toward the geometry and displays a surface-hugging cursor at the location that it is hitting. By subdividing the geometry into these slabs, a quick bounding box test could be used to determine which slab the ray was hitting, before the more expensive determination of which polygon was being hit was attempted.

A related difficulty arose in the implementation of certain of the operations listed above. When geometries are small (a thousand polygons), a frame rate can be high enough to allow for interactive, simple polygon deletion for cutting (subdividing) a mesh. While this technique provides a less than optimal cut (very ragged cut lines based on the geometry), it is often used and is trivial to implement. A further refinement is to actually calculate the real location of the cut and to subdivide cut polygons on the fly. Again, this technique works well for small geometries, but does not scale well and does not lend itself to use with lower frame rates.

For these reasons, we developed the marker-based cutting method outlined above. By interactively laying down markers, the user can specify the locations of the endpoints of the cut and the more computationally complex cutting algorithm can be invoked after all cut endpoints are specified. This algorithm can take as much as a few seconds with no impact to the user. Moreover, the surgeons found that this mode of interaction fit their paradigm of cutting from point to point well also and they were more amenable to display lag when positioning a marker than they were during interactive mesh subdivision. This method of deferring the computationally intense tasks until after all interactive parameters are provided and the user is willing to wait is a technique that worked well and we intend to pursue further.

CASE STUDY APPLICATION

For this case, the surgeons first visualized the high-resolution data of the patient and produced color prints from various views. Next, the VERS system was used to interact with the data. The surgeon could use the system to quickly interact with the meshes representing the skull and the soft tissue (face). When concentrating on the skull, the face object can be left unrendered. The skull could be moved closer into a clipping plane to allow viewing inside the skull for interior structural anomalies. Markers can be laid down on the surface of the skull to measure distances and angles in order to compare the intact side of the face with the affected side. They also could use the cut operator to cut the bone on the intact side of the face, use the reflect operation to produce a mirror duplicate, and examine the fit of this new piece of bone into the area of the defect. If correct, the resulting surgical template could be written out as a VRML file for later use.

In the case of this patient, the intact side of the face was reflected over the affected side and a template to fix the defect was produced (figure 1e). While this did make the assumption that lateral symmetry could be used as a basis for reconstruction, this was deemed clinically acceptable. This template was then subdivided into more planar subpieces. Then a CT model of the boy's hip was generated and, within the environment, the pieces of the template could be moved within the model of the hip to find the location of the best curvature match. Then a paper template was produced and taken into the operating room to allow the surgeon to harvest the bone directly from the hip. Since they had planned it out ahead of time, they could be able to harvest the bone, wire it together as planned, and repair the defect in significantly less time than would otherwise be required. Also, the goal was to repair the defect correctly the first time, without requiring the usual successive procedures for refinement. Both of these benefits also decrease the risk to the patient due to long-term exposure to anesthesia and risk of infection.

SUMMARY_______________

We have created a system which integrates 3D reconstruction, visualization, quantification, and manipulation of multimodal patient data for the purpose of surgical planning. This system was found to be instrumental in the preparation and correction of a severe craniofacial defect and was well received by the surgical community. While more research is warranted in segmentation, mesh reduction, high-performance visualization, user interfaces, and other areas, this tool is a first step and will continue to be used clinically and enhanced to provide the surgeon with all the capabilities required to ensure a superior outcome.

ACKNOWLEDGEMENTS_

This work was partially supported by the NASA Ames Research Center, under grants to Muriel Ross. The authors would like to acknowledge and thank her for her support of the National Biocomputation Center and continued support of this work.

REFERENCES____________

Vannier and J Marsh and J Warren. Three Dimensional Computer Graphics for Craniofacial surgical planning and Evaluation. SIGGRAPH 83 Proceedings, v(17)3, July 1983.
J. Brewster and S. S. Trivedi and H. K. Tuy and J. K. Udupa, Interactive surgical planning, IEEE Computer Graphics and Applications, v(4)3, p.31-40, March 1984.
Jayaram K. Udupa and Dewey Odhner, Display of medical objects and their interactive manipulation, Proceedings of Graphics Interface '89, p40-46, June 1989.
Steven Pieper and Joseph Rosen and David Zeltzer, Interactive graphics for plastic surgery: task-level analysis and implementation, Computer Graphics (1992 Symposium on Interactive 3D Graphics), v(25)2, p127-134, March 1992.
M. Koch and M. H. Gross and F. R. Carls and D. F von Buren and G. Fankhauser and Y. Parish, Simulating Facial Surgery Using Finite Element Methods, SIGGRAPH 96 Conference Proceedings, ACM SIGGRAPH, p421-428, August 1996.
Vikas V. Patel and Michael W. Vannier and Jeffrey L. Marsh and Lun-Jou Lo, Assessing Craniofacial Surgical Simulation, IEEE Computer Graphics and Applications, v(16)1, p46--54, January 1996.
Montgomery, VERS: a virtual environment for reconstructive surgery planning, Proc. SPIE Vol. 3012, Stereoscopic Displays and Virtual Reality Systems IV, Scott S. Fisher; John O. Merritt; Mark T. Bolas; Eds. p. 487-492, May 1997.
Kay, Paul A.; Robb, Richard A.; King, Bernard F.; Myers, R. P.; Camp, Jon J. Surgical planning for radical prostatectomies using three-dimensional visualization and a virtual reality display system Proc. SPIE Vol. 2431, p. 119-125, Medical Imaging 1995: Image Display, Yongmin Kim; Ed. , April 1995.
Kameyama, Ken-Ichi; Ohtomi, Koichi; Ohhashi, Akinami; Iseki, Hiroshi; Kobayashi, Naotoshi; Takakura, Kintomo Virtual surgical operation system using volume scanning display Proc. SPIE Vol. 2164, , Medical Imaging 1994: Image Capture, Formatting, and Display, Yongmin Kim; Ed. p. 146-154, May 1994.
Sader, Robert; Zeilhofer, Hans-Florian; Horch, Hans-Henning Computer simulation of orthognathic surgery with video imaging Proc. SPIE Vol. 2178, Visual Data Exploration and Analysis, Robert J. Moorhead; Deborah E. Silver; Samuel P. Uselton; Eds. p. 186-196, April 1994.
Hill, Derek L.; Green, S. E.; Crossman, J. E.; Hawkes, David J.; Robinson, Glynn P.; Ruff, C. F.; Cox, T. C.; Strong, A. J.; Gleeson, M. J. Visualization of multimodal images for the planning of skull base surgery Proc. SPIE Vol. 1808, Visualization in Biomedical Computing '92, Richard A. Robb; Ed., p.564-573, September 1992.
Kikinis, Ron; Cline, Harvey E.; Altobelli, David; Halle, Michael W.; Lorensen, William E.; Jolesz, Ferenc A. Interactive visualization and manipulation of 3-D reconstructions for the planning of surgical procedures Proc. SPIE Vol. 1808, Visualization in Biomedical Computing '92, Richard A. Robb; Ed. p. 559-563, September 1992.
Linney, Alfred D.; Moss, James P.; Richards, Robin; Mosse, C. A.; Grindrod, S. R.; Coombes, Anne M. Use of a 3-D visualization system in the planning and evaluation of facial surgery Proc. SPIE Vol. 1380, Biostereometric Technology and Applications, Robin E. Herron; Ed., p190-199, April 1991
Taylor, S. Lavallee, G. Burdea, R. Moesges, ed. Computer-Integrated Surgery , MIT Press, September 1995.

BACK