DEVELOPMENT AND APPLICATION OF A
VIRTUAL ENVIRONMENT FOR RECONSTRUCTIVE SURGERY
Kevin Montgomery PhD, Michael Stephanides MD,
Stephen Schendel MD DDS
National Biocomputation
Center
Stanford University
Submitted to Computer-Aided Surgery
March 1, 2000
Keywords:
3d reconstruction,
visualization, surgical planning
Running Title:
Virtual Environment Surgical
Planning
Address correspondence to:
Kevin Montgomery
National Biocomputation
Center
701A Welch Road, Suite 1128
Stanford, CA 94305
Phone: 650.498.6978
Fax: 650.498.5626
Email:
kevin@biocomp.stanford.edu
ABSTRACT
Objective:
This paper details the development and application of a
Virtual Environment for Reconstructive Surgery (VERS). It addresses the
technical and user-interface challenges in developing such a system and the
lessons learned during application of the system in the case of a 17 year-old
boy with a severe facial defect arising from the removal of a soft-tissue
sarcoma.
Materials and
Methods:
Computed tomography (CT) scans were segmented into bone and
soft-tissue classifications using traditional and novel algorithms, a surface
mesh was generated, and imaging artifacts were removed, yielding a mesh
suitable for visualization. This patient-specific mesh was then used in a
virtual environment by the surgeons for preoperative visualization of the
defect, planning of the surgery, and production of a custom surgical template
to aid in repairing the defect.
Results:
This system was successfully used to plan the surgery of the
patient and to produce a custom, patient-specific template that was used to
harvest bone from a donor site in order to reconstruct the defect.
Conclusion:
Despite technical challenges, virtual-environment surgical
planning is useful as a clinical tool for preoperative visualization,
cephalometric analysis, and surgical intervention. It can provide a more
precise surgical result than would otherwise be realized using traditional
methods.
Key Links:
http://biocomp.stanford.edu
INTRODUCTION
Computer-based surgical planning has been investigated by
many researchers over the past decade. The promise of the technology is to
provide better surgical results with fewer procedures, decreased time in the
operating room, lower risk to the patient (increased precision of technique,
decreased infection risk), and a lower resulting cost.
The desire to obtain these goals has led to a large number
of research projects and clinical applications in this area. Routine use of
surgical planning is still in its infancy, although the application of
computer-based cephalometric measurement and surgical planning is clearly of
great interest to surgeons.
Vannier, Marsh et al [Vannier96] at Washington University in
St Louis have done some of the most far-reaching and consistent work in this
field. In addition, the orthopedic surgery group [Sutherland94] has also done
planning in their area. Also in the United States, surgeons at the University
of Texas and Baylor University [Byrd92] have worked on rhinoplasty planning, Johns
Hopkins has done orthopedic planning [Chao93], and the excellent orthopedic
group at UCLA [Gautsch93] continue to work in this area as well.
Internationally, clinical and engineering groups at the
University of Heidelburg [Hassfeld98, Pokrandt96] have done exceptional work in
applications of computing and visualization in surgical planning. In addition, the University hospitals at
Leuven Belgium [Lamoral90,Verstreken96] have done good work in quantification for
maxillar implants.
Researchers at the University College in London [Moss88,
Perry98, Ayoub96] have done work in maxillofacial surgery prediction and
craniofacial reconstruction. Maxillofacial implant optimization has been
researched by the University Hospital Groningen, Netherlands [Rozema92] and
other craniofacial visualization has been done at Utrecht University Hospital
[Zonneveld89].
In the case of our research, the Virtual Environment for
Reconstructive Surgery (VERS) project [Montgomery97,98ab] was started between
the NASA Ames Biocomputation Center and the Stanford University Department of
Reconstructive Surgery. Its initial aims were to apply the 3D reconstruction
and visualization technologies developed for space-related research toward
providing surgeons with visualization capabilities to aid in preoperative
planning.
The VERS project has since expanded its scope to allow full
visualization and interaction between the surgeon and the reconstructed data of
their patient [Montgomery99abc]. The result is a complete surgical planning station
that allows the surgeon to visualize their patient's data, extract quantitative
information (such as distances and angles) directly from their dataset, to
actually simulate the surgical procedure for training purposes, and to even
generate patient-specific templates that can be used in the surgery.
This paper will briefly discuss the technologies behind this
project, detail a case study for which this system proved invaluable, and
explore the lessons learned in the production and use of a virtual environment
for reconstructive surgery.
METHODS
The system described below consists of four steps. First, a
Computed Tomography (CT) scan of the patient was performed. Second, this data
was used to segment the bone and soft-tissue of the patient. Third, surface
meshes of these tissues were produced from the segmented data and artifacts
were removed. Finally, these surface meshes were used for visualization,
quantification, and interaction in the process of surgical planning.
Reconstruction
A CT scan of the patient was performed using a GE 9800-class
device (General Electric Medical Systems, Waukesha, WI). This data was then
transferred from the Stanford Radiology Department to our Onyx InfiniteReality
server (Silicon Graphics Inc, Mountain View, CA). Once there, the data was
translated (as necessary) from vendor-specific formats into the standard DICOM
(ACR/NEMA) format.
In processing the data for a patient, our software would
create an intermediate, internal, block-file format of the data. The DICOM image
files would be examined, and split into separate directories based on image
series and other parameters. Once there, the script would extract the image
data from each DICOM image file and build a single, volumetric block file of
the imaging data, as well as a separate file comprising all relevant DICOM
header information. In this way, rather than parsing each DICOM image file when
reading the data into our reconstruction software (~5 minutes of processing),
we could instead read the header, then do a very fast block read of the data
into memory (~2 seconds).
This block data was read into 3D reconstruction software
that had been produced in-house. This software reads and allows us to view the
raw CT data, and provides for automated or semiautomated segmentation by voxel
classification. In the case of segmenting bone and soft-tissue, a simple
thresholding algorithm based on radiographic density (Hounsfeld value) was
sufficient for initial segmentation.
However, the resulting segmentation also contains many small
features that are not desired in the final reconstruction. These features are
due to imaging artifacts, small cavities within the structure of interest, and
other anomalies. Therefore, a connected-components algorithm was used to
segment out connected voxels into regions. The volume of these regions was then
used to select only the major anatomical features, while other regions were
omitted from selection. The resulting data provided a volume of segmented data
with high specificity.
From this segmented volumetric dataset, a mesh was generated
for both the bone and soft-tissue using a Marching Cubes algorithm
[Lorensen87]. A number of heuristics were then applied to ensure that the
resulting mesh was sufficient for visualization. The resulting high-resolution
meshes are then used for preoperative visualization, where great detail is
required.
Virtual Environment for Reconstructive Surgery
The Virtual Environment for Reconstructive Surgery consists
of the following components:
- Silicon Graphics Onyx
InfiniteReality graphics workstation
- FakeSpace Immersive Workbench
display system
- Polhemus FasTrak stylus
- StereoGraphics CrystalEyes stereo
glasses
- Sense8 WorldToolKit
The WorldToolKit application reads in the given mesh files
and allows the user to visualize, measure, interact, and manipulate the data of
their patient. This application handles
the interaction between the FasTrak stylus 6-degree of freedom electromagnetic
tracking/input device manipulated by the user, applies the position and orientation
data to a virtual tool in the environment, and provides for real-time stereo
display using the CrystalEyes stereo glasses on the Immersive Workbench.
A number of "virtual tools" were implemented,
including:
- Selection/moving tool- allows the
user to grab and move/rotate an object
- Marker tool- allows the used to
lay down markers on the surface of an object
- Lighting tool- a
"spotlight" that allows more precise localization of lighting
Using these tools, the surgeon can apply the selection/moving
tool to pick up the skull to rotate it into a new position. By applying the
same mechanism, the user could reach into the virtual toolbox to select and
then manipulate a new virtual tool. Further, the surgeon can lay down markers
on the patient anatomy and have the system compute 3D cephalometric
measurements from the marker positions.
Finally, other virtual objects, such as spotlights, could be created and
moved about to aid in visualization as well.
A number of operations were also available. There were
operations for technical feedback such as performance characterization and
scene graph display, as well as operations to modify the presentation of
objects, such as manipulating object attributes (color, transparency) and
turning on/off display of objects. Visualization operations included setting
standard viewpoints, rendering modes (wireframe, solid), saving the scenegraph
in VRML, and dumping a screen image for later use.
In addition, a number of operations use the locations of the
markers. The measure operation measures and displays the distances (linear and
surface) and angles between each of the markers. The cut operation subdivides
the mesh between to the markers. The reflect operation allows the user to make
a duplicate, but reversed object, etc. This marker-based method for interaction
proved invaluable in providing quick feedback and interaction with the user.
This system allowed for the visualization, quantification,
and, most importantly, interaction with the patient’s data.
RESULTS
This case involves a 17 year-old Hispanic boy [Figure 1]. At
age 9 his maxillofacial surgeon found a fast-growing soft tissue tumor located
under the boy's left eye. Because it appeared to be an advanced and very
aggressive tumor, complete resection was indicated, which required removal of
all bone and significant soft tissue on the left side of the boy's face. This
aggressive approach was important, as the boy presented cancer-free at age 17,
yet required advanced reconstructive surgery. Because this is a particularly
unusual and difficult case, we sought to use advanced visualization to aid in
the preparation of this surgery.
The boy underwent a CT scan, and his data was processed as
outlined above- segmentation, mesh generation, artifact removal, and preparation
for visualization. There were 1.5 million polygons comprising the face [Figure
2] and 2.5 million polygons in the skull model [Figure 3]. For preoperative
visualization, we typically view these images in stereo, so we require a
minimum of 8 million polygons per frame. With technology available at the time,
this much information could not be rendered in real-time.
Therefore, the meshes were reduced for interactive
visualization purposes in the VERS environment. In the case of this dataset,
both the skull and face were reduced to roughly 500,000 polygons for
interactive visualization. Our mesh reduction techniques preserve as much
detail as possible, while reducing the number of triangles required to
represent the surface. However, this was the minimum size that was judged by
the surgeons that still provided sufficient detail.
For this case, the surgeons first visualized the
high-resolution data of the patient and produced color prints from various
views. Next, the VERS system was used to allow the surgeon to interact with the
meshes representing the skull and the soft tissue (face). When concentrating on
the skull, the rendering of the face can be turned off. The skull could be
moved closer into a clipping plane to allow viewing inside the skull for interior
structural anomalies. Users could lay down markers on the surface of the skull
to measure distances and angles to compare the intact side of the face with the
affected side. They also could use the cut operator to cut the bone on the
intact side of the face, use the reflect operation to produce a mirror
duplicate, and examine the fit of this new piece of bone into the area of the
defect. If correct, the resulting surgical template could be written out as a
VRML file for later use.
In the case of this patient, the intact side of the face was
reflected over the affected side [Figures 4-5] and a template to fix the defect
was produced [Figures 6]. This template was then subdivided into more-or-less
planar subpieces [Figure 7] using a plane-based cutting tool. Then a CT model
of the boy's hip was generated and, within the environment, the pieces of the
template could be moved within the model of the hip to find the location of the
best curvature match [Figures 8-10]. Then a 2D paper template was produced,
traced onto radiological film to withstand sterilization, and taken into the
operating room to allow the surgeon to harvest the bone directly from the hip.
The film-based template could then be laid onto the donor site (inner table of
the iliac crest) so that donor bone of the right size and shape could be
delaminated from the pelvis, while preserving its blood supply (deep circumflex
iliac artery and vein). While this patient did undergo reconstruction of the
bony defect and soft tissues, unfortunately a post-operative image is not
currently available due to patient non-compliance with longer-term
postoperative care.
By using surgical templates, the surgeons could repair the
defect in less time than would otherwise be required. Also, they could repair the
defect correctly the first time, without requiring the usual successive
procedures for refinement. Both of these benefits also decrease the risk to the
patient due to long-term exposure to anesthesia and risk of infection.
DISCUSSION
The major difficulties identified in producing such an
environment lie in the desire to maintain high fidelity of the dataset, but to
provide a reasonable frame rate for the interactive virtual environment. This
is a very different paradigm than occurs in most virtual environment
applications. In many application areas, such as architectural walkthrough,
many objects may be culled because they are currently out of view of the user.
In these cases, a scene graph model is of great benefit to decrease rendering
to only those polygons that may be visible in the scene. Also, a bounding box
test can be used for aiding in visibility or selection testing. In addition,
each object is typically rather small (under a thousand polygons). Finally,
maintaining multiple levels of detail of an object can further decrease
rendering requirements. These techniques can allow the elimination of the
consideration of many objects in the scene and further increase display and
selection speed.
In a surgical application, there is typically one, very
high-resolution object to be viewed. All of the object is in view, all of the
time. Even at 500,000 polygons and with a high-end graphics supercomputer
providing a theoretical peak of 10,000,000 polygons/second, we have a
theoretical maximum framerate of 20 fps. Our observed performance of 4 fps
(without display-list objects) or 8 fps (with display-list objects) was
considerably less.
A number of techniques were employed to ameliorate this
situation. First, the original meshes of millions of polygons need to be
reduced dramatically. Curvature-based (chord height) methods (reduce the number
of polygons in areas of low curvature) initially appeared straightforward, but
these techniques, when applied liberally, produced large faceted (flat) areas,
particularly on the forehead. By not constraining the size of the conjoined
regions, such areas of low curvature were over decimated. Employing polygon
size metrics (eliminate small polygons because they contribute little to the
overall visualization) worked well and solved the above problems. However,
these techniques, when applied liberally, would tend to erode away sharp edges
of the model (such as near the sutures of the skull), since small polygons are
often present along such sharp edges. Finally, a technique which allowed user
control of the weighting of these two functions allowed us to decrease
requirements, but preserve sufficient fidelity. This is clearly an area that
warrants further research.
Second, once the original meshes have been decreased in size
as much as the surgeons deemed clinically accurate, the issue of how to render
these meshes efficiently becomes important. In our case, the large geometry was
split up into smaller "slabs". Therefore, the slab comprising the
rear of the skull could be viewed in lower resolution using a level-of-detail
mechanism. Note that a rear slab can not be completely culled from the scene
because there may exist holes in the slabs closer to the user (through the
eyes, for instance).
This slab-based approach also dramatically improved prebuild
time (time to optimize the geometry for rendering and create display list
objects- an O(n2) operation) from 20 minutes to 2 minutes. When
preprocessing required 20 minutes, the system was very difficult to use and to
test due to this time delay. After the reduction in time, preprocessing, and
hence its great rendering advantages, could be realized.
In addition, this approach allowed for an optimized
selection mechanism. When laying down markers, the marker tool casts a ray
toward the geometry and displays a surface-hugging cursor at the location that
it is hitting. By subdividing the geometry into these slabs, a quick bounding
box test could be used to determine which slab the ray was hitting, before the
more expensive determination of which polygon was being hit.
A related difficulty arose in the implementation of certain
of the operations listed above. When geometries are small (a thousand
polygons), a frame rate can be high enough to allow for interactive, simple
polygon deletion for cutting (subdividing) a mesh. While this technique
provides a less than optimal cut (very ragged cut lines based on the geometry),
it is often used and is trivial to implement. A further refinement is to
actually calculate the real location of the cut and to subdivide cut polygons
on the fly. Again, this technique works well for small geometries, but does not
scale well and does not lend itself to use with lower frame rates.
For these reasons, we developed the marker-based cutting
method outlined above. By interactively laying down markers, the user can
specify the locations of the endpoints of the cut and the more computationally
complex cutting algorithm can be invoked after all cut endpoints are specified.
This algorithm can take as much as a few seconds with little impact to the
user. Moreover, the surgeons found that this mode of interaction fit their
paradigm of cutting from point to point well also. This method of deferring the
computationally intense tasks until after all interactive parameters are
provided and the user is willing to wait is a technique that worked well in
practice.
While the system described was usable in this case, more
recent work has focused on clinical applications for mandibular reconstruction,
wrist fracture diagnosis and treatment planning, and soft-tissue defect
planning. As in the case presented here, we have continued to find great
utility in surgical templates to increase the precision and decrease the risk
in craniofacial surgery. To support these applications, soft-tissue and
rigid-body kinematic modeling software has been developed and will form the
basis for true preoperative prediction of postoperative outcome. However, the
use of volumetric decomposition methods for real-time interaction as described
above will continue to make optimal use of limited computing resources.
ACKNOWLEDGEMENTS
This work was supported by the NASA Ames Research Center,
under grants to Stephen Schendel and Muriel Ross. The authors would like to
acknowledge and thank NASA for their continuing support of the National
Biocomputation Center.
The authors also would like to thank Silicon Graphics
Incorporated for their technical assistance during this project, Sense8
corporation for providing outstanding support and services related to this work,
and FakeSpace Incorporated for their feedback on virtual-model display
visualization research.
REFERENCES
[Ayoub96] Ayoub, A.F., Wray, D.,
Moos, K.F., Siebert, P., Jin, J., Niblett, T.B., Urquhart, C. and Mowforth,
P. Three-dimensional modeling for
modern diagnosis and planning in maxillofacial surgery. Int. J. Adult Orthod. Orthognath. Surg.
11:225-233 (1996).
[Bechtold93] Bechtold, J.E. and
Powless, S.H. The application of
computer graphics in foot and ankle surgical planning and reconstruction. Clin. Podiatr. Med. Surg.10: 551-562 (1993).
[Byrd93] Byrd, H. S. and Hobar,
P.C. Rhinoplasty: a practical guide for surgical
planning. Plast. Reconstr. Surg.
91:642-654 (1993).
[Chao95] Chao, E.Y.S. and Sim,
F.H. Computer-aided preoperative planning
in knee osteotomy. Iowa Orthop. J. 15:4-18 (1995).
[Gautsch94] Gautsch, T.L.,
Johnson, E.E. and Seeger, L.L. True
three dimensional stereographic display of 3D reconstructed CT scans of the
pelvis and acetabulum. Clin. Orthop.
305:138-151 (1994).
[Gerbo97] Gerbo, L.R., Poulton,
D.R., Covell, D.A. and Russell, C.A. A
comparison of a computer-based orthognathic surgery prediction system to
postsurgical results. Int. J. Adult
Orthod Orthognath. Surg. 12:55-63 (1997).
[Hassfeld98] Hassfeld, S., Zöller,
J., Albert, F.K., Wirtz, C.R., Knauth, M. and Mühling, J. Preoperative planning and intraoperative
navigation in skull base surgery. J.
Cranio-Maxillofac. Surg. 26:220-225 (1998).
[Lamoral90] Lamoral, Y.,
Quirynen, M., Peene, P., Vanneste, F., Lemahieu, S.F., Baert, A.L. and van
Steenberghe, D. Computer tomography in
the preoperative planning of oral endo-osseous implant surgery. Rofo Fortschr Geb Rontgenstr Neuen Bildgeb
Verfahr 153:505-509 (1990).
[Linney93] Linney, A.D., Tan,
A.C., Richards, R., Gardener, J., Grindrod, S. and Moss, J.P. Three-dimensional visualization of data on
human anatomy: diagnosis and surgical
planning. J. Audiov. Media Med. 16:4-10
(1993).
[Lorensen87] Lorensen, W.E. and
Cline, H.E., "Marching Cubes: a high resolution 3D surface reconstruction
algorithm," Computer Graphics, Vol. 21, No. 4, pp 163-169 (Proc. of
SIGGRAPH), 1987. Watt, Alan, and Watt, Mark, Advanced Animation and Rendering
Techniques, Addison-Wesley, 1992.
[Lotwin93] Lotwin, R.S. and
Baerg, R.H. Orthographics. A two-dimensional surgical planning
system. Clin. Podiatr. Med. Surg. 10:
539-549 (1993).
[Montgomery99a]
Kevin Montgomery PhD, Michael Stephanides MD, Stephen Schendel MD, Muriel Ross
PhD; "Preoperative Planning of a Craniofacial Defect Using a Virtual
Environment", Medicine Meets Virtual Reality, January 1999.
[Montgomery99b] Kevin Montgomery PhD, Michael Stephanides
MD, Stephen Schendel MD, Muriel Ross PhD "Virtual Surgical Tools for
Planning Craniofacial Surgery", Medicine Meets Virtual Reality, January
1999.
[Montgomery99c] Kevin Montgomery Ph.D., Michael Stephanides
MD, Joel Brown BS, Jean-Claude Latombe Ph.D., Stephen Schendel MD, DDS;
"Virtual Reality in Microsurgery Training", Medicine Meets Virtual
Reality, January 1999.
[Montgomery98a] Montgomery K, Stephanides M, Schendel S,
Ross M; "A Case Study Using the Virtual Environment for Reconstructive
Surgery", IEEE Visualization `98, Research Triangle Park, NC, October,
1998.
[Montgomery98b] Montgomery, K; Stephanides, M; Cheng, R;
Schendel, S; Ross, M; "Clinical Application of VR for Surgery", SPIE
Electronic Imaging- Engineering Reality of Virtual Reality, San Jose, CA,
January 1998.
[Montgomery97] Montgomery, K, "VERS- A virtual
environment for reconstructive surgery planning", Engineering Reality of
Virtual Reality, SPIE Electronic Imaging, San Jose, CA, February 1997.
[Moss88] Moss, J.P., Grindrod, S.R., Linney, A.D., Arridge, S.R.
and James, D. A computer system for the
interactive planning and prediction of maxillofacial surgery. Am. J. Orthod. Dentofac. Orthop. 94:469-475
(1988).
[Perry98] Perry, M., Banks, P.,
Richards, R., Friedman, E.P. and Shaw, P.
The use of computer-generated three-dimensional models in orbital
reconstruction. Br. J. Oral Maxillofac.
Surg. 36: 275-284 (1998).
[Pokrandt96] Pokrandt, P.,
Bohner, P., Knolle, J.P., Hassfeld, S. and Raczkowsky, J. A communication system supporting
simultaneous planning and execution in cranio-maxillo-facial surgery. J. Telemed. Telecare 2:3-7 (1996).
[Rozema92] Rozema, F.R., Otten,
E., Bos, R.R.M., Boering, G. and van Willigen, J.D. Computer-aided optimization of choice and positioning of bone
plates and screws used for internal fixation of mandibular fractures. Int. J. Oral Maxillofac. Surg. 21: 373-377
(1992).
[Stephanides99b] Stephanides M,
Montgomery K, Bowen V, Hentz V, Schendel S; "Computerized Diagnostic and
Surgical Planning Tools for the Surgery of the Wrist", Medicine Meets
Virtual Reality, January 1999.
[Sutherland94] Sutherland, C.J.,
Bresina, S.J. and Gayou, D.E. Use of
general purpose mechanical computer assisted engineering sofware in orthopaedic
surgical planning: advantages and
limitations. Comput. Med. Imaging
Graph. 18:435-442 (1994).
[Vannier96] Vannier, M.W. and
Marsh, J.L. Three-dimensional imaging,
surgical planning, and image-guided therapy.
Radiol. Clin. North Am. 34:
545-563 (1996).
[Verstreken96] Verstreken, K.,
Van Cleynenbreugel, J., Marchal, G., Naert, I., Suetens, P. and van
Steenberghe, D. Computer-assisted
planning of oral implant surgery: A
three-dimensional approach. Int. J.
Oral Maxillofac. Implants 11:806-810 (1996).
[Yune93] Yune, H.Y. Two-dimensional-three-dimensional
reconstruction computed tomography techniques.
Dent. Clin. North Am. 37:613-626 (1993).
[Zonneveld89] Zonneveld, F.W.,
Lobregt, S., van der Meulen, J.C.H., and Vaandrager, J.M. Three-dimensional imaging in craniofacial
surgery. World J. Surg. 13: 328-342
(1989).
FIGURE CAPTIONS
Figure 1. 17 year-old hispanic male with severe facial
defect due to aggressive resection of sarcoma.
Figure 2. Surface reconstruction of soft-tissue rendered
with transparency (alpha=0.5) over skull model.
Figure 3. Automatically-generated 3D reconstruction computer
model showing skull and extent of defect.
Figure 4: Computer-generated template in place on computer
model of skull.
Figure 5. Soft-tissue overlay onto projected result with
original skull structure and computer-generated template.
Figure 6. 3D template piece of bone in isolation.
Figure 7. Use of a planar cutting tool to break 3D template
into relatively planar subpieces.
Figure 8. 3D template subpieces within virtual environment,
with computer model of the patient’s hip.
Figure 9. Use of the system to move subpieces within hip
model to match curvature.
Figure 10. Extraction of template from hip model.
Figure 1 Figure
2
Figure 3 Figure
4
