Volume Graphics:The road to interactive medical imaging?

by Amere Oakman ISE II
for SURPRISE 96


Introduction - The need for visualisation
What are volume graphics?
A brief history of computer graphics
Surface graphics and volume graphics

Surface graphics (surface rendering)
Volume graphics (volume rendering)
Surface graphics vs. volume graphics
Parallel rendering
Medical Applications
Conclusions and future directions
References

Introduction - The need for visualisation

Data is what most sciences and engineering disciplines are all about. Once theories have been formulated, hypotheses composed, and designs drawn up, the real test is the empirical one. Does what actually happen match what was predicted/intended? This is the reason that analysing data is so important.

Often, the clearest and most intuitive mode of analysis is the visual one. This is certainly true of medical images. A large fraction of medical information (not counting verbal diagnosis of course) is collected in the form of numerical data (mostly digital) and pictorial data (e.g. x-rays). Usually there is a large amount of data and/or it is not very clear. Most importantly, the data is nearly always supposed to be a representation of volumetric data; that is, data that depicts a three-dimensional (3D) object. For example, an x-ray is pictorial information that is in two-dimensional (2D) form, yet it is derived from a 3D object - namely, a person. In losing a dimension there is inherently some loss of information, that of a spatial nature. Another example is that of data on the size, shape and position of a tumor. 2D information is useful, but 3D information, viewable in a realsitic manner is obviously more so.

Having established the need for visualisation of medical data, we will now take a look at the way in which an emerging trend in computer graphics is at last making that visualisation more possible than ever before.

Back to top


What are volume graphics?

Volume graphics is the subfield of computer graphics that is concerned with representing environments in terms of the 3D volume spaces that they occupy, and with rendering (visualising) such environments. "So what?", one could say, "Isn't that the natural way to represent volumes?". It is the natural way humans think of volumes, but up until a few years ago it is not how computers 'thought' of 3D data.

Back to top


A brief history of computer graphics

Early computer graphics (1960's - 1970's) were based on devices that drew vectors (straight lines) and that was all. Objects in the scene were stored as geometric data (vertices of a cube for example) in a display list. Manipulation of the objects (e.g. scaling or animation) was achieved through redrawing all of the objects in their transformed appearance - refreshing (redrawing) the screen. This approach was good in that it allowed excellent control over the objects in the view. The realism of such pictures was, however, not very high. The surfaces of the objects were very hard to render using vectors and, as a result, the graphics were wire-frame.

The alternative to vector graphics were raster graphics. A raster (or screen buffer) is a 2D array representing the screen. Each element in the raster represents one pixel (screen element) on the display. The pixel can be either on or off in a monochrome display or can take any combination of RGB (Red, Green, Blue) values in a colour display. Since each pixel can be controlled, the capability to show realistic surfaces is provided.

The explanation for the use of vector techniques at all is an economic and technological one. To hold such an amount of data as in a screen buffer requires a large amount of memory. To process such an amount of data as in a screen buffer requires a large amount of processing power to get an acceptable refresh speed. These two highly important drawbacks to raster graphics prevented the method from being widely used until the late 1970's, by which time the expense involved had reduced enough and the technology required had improved enough to make it feasible.

In order to viualise 3D volumetric data using the raster technique, vector-based object ideas have been taken up to produce surface graphics methods. The earliest of these methods extracted the surfaces of the objects from their binary data and then rendered them using polygons - most often triangles.

Back to top


Surface graphics and volume graphics

The visualisation techniques of the present are split into two distinct families - one established method with a proven ability and another less generally accepted system.

Both approaches use a new element that comes closer than ever before to the way in which volumes are thought of by humans - the voxel.


A voxel

As suggested by the name, a voxel is a companion to the pixel. A pixel is an element that represents a unit of area, a voxel is an element that represents a unit of volume. In the two procedures (surface and volume graphics) the data is voxelised, that is, converted into voxels by various interpolation algorithms.

Surface graphics (surface rendering)

Surface graphics techniques then select the voxels that appear as surfaces to a viewer and project them (after suitable transformation to account for angle of view and other such criterion) onto the 2D pixelated screen. Specialised hardware and software have been developed to cope with the processing requirements of the scheme - termed geometry engines.

Volume graphics (volume rendering)

In volume graphics, no extraction of surfaces is required. A 3D volume buffer, analogous to the 2D screen buffer, is used that stores the voxels as they are. The final image is directly rendered from the volume buffer by a variety of algorithms - ray-casting being the one that has emerged (in an assortment of different variations) as the method of choice.


Ray-casting

Ray-casting is a technique whereby hypothetical light rays are sent from each pixel on the screen towards the volume and either travel through and are absorbed somewhat or are reflected. The values associated with the voxels determine what happens to each ray and therefore what image is finally displayed.

Surface graphics vs. volume graphics

There are some important advantages that volume graphics have over surface-based techniques, and also some disadvantages.

The most important advantage that volume graphics provide is that of insensitivity to environment and object complexity. With surface graphics, the number and elaborateness of the objects in the scene affects the time taken to render the scene since transformations of the objects on the display list are affected by their size and complexity. In volume graphics, the scene is already converted into a finite-size volume buffer. This advantage is equivalent to the advantage that raster graphics have over vector graphics in terms of screen refresh time.

Another advantage is the fact that viewpoint of the observer is not pertinent. In surface graphics, the data must be converted into surfaces after every change in viewpoint. In volume graphics, all viewpoints are pre-computed by the very nature of the stored data. This means that any viewpoint-independent characteristic of the voxel (e.g. density) can be stored along with it in the volume buffer. This cannot be done with surface graphics since the surfaces are polygonised and are not represented by the actual voxels themselves.

Perhaps the most important advantage relevant to medical applications is that the interior of the objects are accessible to the viewer.


Cuts made in real-time with InViVo Software
(see InViVo reference[10])

With surface graphics the interior information is lost - again analogous to the advantage of raster graphics over vector graphics in terms of surface representation.

One more improvement is the extra closeness to the original data - 3D volume data should be represented as a volume as in the volume buffer.

Disadvantages of volume graphics are such issues as the loss of the continuous form of surface graphics. The discrete nature of the direct voxel representation in volume graphics, as contrasted with the stored geometric representations of the objects in surface graphics, means that there is a loss of accuracy in the information. Relating this to medicine, such operations as calculating the volume of a tumor would be less accurate (but more simple) under volume graphics.

Another, perhaps more obtrusive obstacle, is the cost and technology involved in volume graphics. Typical volume data sets have volume buffers of the size 512x512x512. This is over 100 million voxels. With just 1 byte per voxel this requires over 128Mbytes. This will probably not be too much of a problem however, as memory prices continue to fall.

Processing power required to handle the volume buffer are more prominent. At present the fastest rendering time for a volume buffer of size 256x256x256 (on a single processor system) is approximately one second (conducted on an R4000 Indigo). If acceptable frame rates (10-15 frames per second minimum; 30 frames per second ideally) for real-time animation purposes (copious applications in medicine, e.g. real-time acquisition and viewing of ultrasound data) are to be achieved, there needs to be a thirty-fold improvement in speed over current technology.

If not for this last point, it is apparent that there are enormous advantages to be gained, in terms of medical imaging, through the adoption of volume graphics.

The seemingly insurmountable problem of processor inadequacy is being addressed through the use of parallel and massively parallel computers.

Back to top


Parallel rendering

Parallel computers, consisting of tens of processors, have been used in the past few years in an effort to improve the rendering speeds of volume rendering algorithms. Attempts have also been made using massively parallel machines (thousands of processors), but the way forward should be to use the fewest number of processors as possible as this is closer to the desired goal of an affordable system for mass-acceptance and use.

To date, the fastest rendering speeds have, perhaps surprisingly, been registered by parallel (as opposed to massively parallel) computers. The crux of the matter is - as it has, and continues to be, in many engineering applications - the trade-off between speed and cost. Frame rates, for 128x128x109 volume buffer renderings, of 31Hz have been reported [Lacroute,2] from a 16-processor machine (SGI Challenge) using the latest ray-tracing technique of the shear-warp algorithm (too in-depth to be discussed here, but see [Lacroute] for reference). The same algorithm produces 13Hz for a 256x256x225 buffer on the same machine. It must be noted, however, that the pre- computation time (essentially voxelisation of the data) was 65 seconds (on a single processor). Thus, there is still a long way to go before real-time data-acquisition is reachable.

Back to top


Medical applications

These advances would not be much of a benefit if they had no clinical application. However, they are of great use in such areas as clinical descision-making of all kinds.

3D displays of CT and MR (see [Krestel,7]) data give a better view of many types of injury and disease than do conventional 2D scans. In particular, it can be stated that the more complex the area to be evaluated the more helpful 3D images can be. Additionally, one can say that the more treatment options there are available the more any extra piece of information can help in the arrival at the correct solution. With some types of disorders, e.g. deformities of the bone, 3D visualisation will be extremely useful in the planning of possible surgical correction. Trauma injuries, such as fractures, are sometimes only truly appreciated when looking at 3D images.

Head-mounted display units that combine video and computer images (allowing doctors to see 'through' a patient's skin) is also an area of research that could benefit from improved visualisation of data, e.g ultrasound data.


Doctor viewing patient's interior [13]

Along with topics like anatomical atlases and plastic surgery planning these techniques of visualisation will aid physicians in all areas of treatment from diagnosis to surgery.

Back to top


Conclusions and future directions

One thing has yet to be mentioned in association with this topic. That is the fact that the process of determining what is to be viewable as surfaces and what is not to be viewable, along with deciding on such aspects of colours and opacity of voxels is not yet an automated process.

At present, a user must observe the 3D data and assign these attributes in an iterative process that requires multiple renderings of the image until the user is satisfied. There is no algorithm for processing the data and deciding what it is and therefore how it is to be represented. Thus the environment must be known in advance, there is no method within the grasp of current thought that could classify (segment) arbitrary medical scenes. This must become an active area of research if visualisation is to yeild its full potential.

Having said that, improved rendering times are making the interactive segmentation (as it is called) by the user perfectly usable and acceptable until general methods of segmentation and classification are found.

As we have seen, the emerging trend in computer graphics of volume rendering is affecting the way we will be able to visualise medical data as never before. 3D views will be interactive in that they will allow rotations, interior views, zoom-ins and a whole host of other functions all in real-time, giving doctors of medicine more information than ever previously available.

Back to top


References

  1. "Voxels:Data in 3D", Byte, May 1992, pp.177-182.
  2. "Real-Time Volume Rendering on Shared Memory Multiprocessors Using the Shear-Warp Factorization",
    Lacroute, P.; 1995 Parallel Rendering Symposium.
  3. "Surface And Volume Rendering Techniques To Display 3-D Data", Barillot, C. ;
    IEEE Engineering In Medicine And Biology, Vol.11, No.3, March 1993.
  4. "Volumetric Rendering of Computed Tomography Data:Principles and Techniques ", Ney, D.R., Fishman E.K., Magid D.;
    IEEE Computer Graphics & Applications, Vol.10, No.3, March 1990.
  5. "Volume Graphics", Kaufman A.; IEEE Computer, Vol.26, No.7, July 1993.
  6. "Research Issues In Volume Visualization", Kaufman A., Hohne K.H., Rosenblum L., Schroeder P.;
    IEEE Computer Graphics & Applications, Vol.14, No.3, March 1994.
  7. "Imaging Systems For Medical Diagnostics", Krestel E.; pp.138-142.
  8. "Visualization of 3D Ultrasound Data", Nelson T.R., Elvins T.T.;
    IEEE Computer Graphics & Applications, Vol.13, No.11, November 1993.
  9. Image Processing Group, UMDS
  10. InViVo - Homepage
  11. Computer Vision Home Page
  12. Reconstruction Home Page
  13. Graphics Groups, University of North Carolina
  14. Y. Touboul, Institut Francais du Petrole, Paris, France.
  15. Gregg Leichtman, Ph.D; Director: Computer Integration Laboratory, Thomas Jefferson University, Philadelphia, USA.

Back to top


Back to main page