Interactive Three – Dimensional Model Reconstruction of Ultrasonography

This paper discusses interactive 3D reconstruction of ultrasonography model. The proposed scheme can be divided into three modules, consisting of Image Segmentation, 3D Model Reconstruction and Interactive Model Editing.More specifically, a set of 2D radial scans acquired from ultrasonography were segmented using Active Contour, resulting in an unorganized set of 3D points delineating an object of interest on multi-planar images. These points were subsequently fitted with a 3D surface model by an expansion of Spherical Harmonics. As accurately segmenting ultrasound images with low SNR can be challenging, this fitted model will be later interactively adjusted to be more properly coincided with underlying structure subject to expert’s opinion. The contribution of this paper is therefore the algorithm and the organizing of related data structure which lend to responsive rendition of the parametric surface being edited.


Introduction
Image processing has become a primary step toward various computational vision schemes (1) . Most prominent ones include image enhancement, noise filtering and object segmentation for subsequent data analyses, e.g., size and shape estimation as well as motion detection (2) . Medical Imaging has also enjoyed these advancements. Not until recently, various imaging modalities such as the Computed Tomography (CT) and Ultrasonography had been primarily used as supplement data in visual diagnosis. Representation was merely in 2D grayscale image array, despite its full 3D modal capability. Modern visualization techniques such as surface reconstruction has made it possible for cutting edge scanners to produce 3D rendering of anatomical structures, which however remains inaccessible to less equipped clinic or local healthcare providers. This factor has thus impeded otherwise efficient communication between physicians and their patients regarding the understanding of the symptom, diagnosis and treatments. In particular for ultrasonography, for instance, real-time 3D visualization offers efficient and early prognostic determinant of the fetal anomalies in vivo.
Existing built-in software nonetheless, mostly render the 3D structure by using voxel-based techniques, e.g., ray casting, etc. Despite its visual appearance, these non-photo realistic rendering limits their applications within the scope of visual assessment. Quantitative evaluation, on the other hand, would require yet an additional set of tools for drawing the geometrical overlaid, whose dimensions (length, area and volumes) may be calculated, to determine the degree of those anomalies. Explicit geometrical model reconstruction from various imaging modalities has attracted considerable interests in the literatures. Geometrical augmentation was proposed in a Virtual Reality scheme for surgical planning (3) and other medical applications (4) , such as interactive colonoscopy system (5) . More recently, physical modeling was also incorporated into 3D synthetic object to improve realistic sense during medical training. Specifically, haptic force was produced by calculating related attributes upon surgical instrument touching the target object (6)  Motivated by this scenario, this paper presents a method which assists interactive surface reconstruction of 3D model from ultrasonographic images. This framework was based on two well established algorithms, i.e., Active Contour and Spherical Harmonics for initial segmentation and 3D surface reconstruction, respectively. Subsequent interactive model editing was permitted by properly organizing relevant data structure so as to optimize local surface modification. The resultant parametric surface model can readily serve as a source for subsequent quantification should require. With interactive editing, any mistake or inaccurate segmentation resulted from compromised image quality, can be amended by medical expert. In the experiment, the proposed scheme was validated against both the synthetic models (a sphere and a pumpkin) and a real fetal image, whereby accuracy analysis and time complexity was reported.

Image segmentation
Segmentation is a method by which the object of interest is extracted from the image. The method is a primary step toward various medical image analysis and therapeutic intervention schemes, such as left ventricular analysis and brain and cerebral mapping, using MRI, vertebral extraction from CT and fetal diagnosis and therapy by means of an ultrasound sequence. Generally, a single image acquisition contains not only the anatomical structure of interest but also its peripherals, i.e., connective tissue, bone and organs. Furthermore, there are also noise and artifacts present at various degrees, depending on acquiring modality. Medical image segmentation can be divided in to two categories, pixel and parameter based, depending on the objective of the study. This paper focuses primarily on the later, whose main advantage is its ability to offer analytical parameters that can readily be used to derive quantitative attributes, e.g., length, areas, and volumes. Active Contour: Snake (6) is a simple yet reliable scheme, widely adopted in the literatures. (5) The method extracts boundary of an object based on the minimization of energy, resulted from image forces (external) and curve constraints (internal). (7) It also allows user intervention via adjustable weighting parameters to suite problem at hand. In addition, moving objects can be tracked from an image sequence by incorporating temporal constraints. Detailed treatments on its variants can be found in (8,9) and references therein.

Surface Reconstruction
3D surface model of an object can be reconstructed from segmented entities. Marching cubes (10) , for instance, build a network of triangle mesh whose each facet is extracted from 8-connected pixels using tri-linear interpolation. Since the model contains a set of vertex points, that may be too scattering and noisy to be analyzed. The spherical harmonic function can thus be applied to regularize the mesh entity while retaining its geometry (11,12) . Model reconstruction begins with converting each point on a genus-zero surface into corresponding spherical coordinates. Fourier transform is then applied to expand those points with harmonic series on a unit circle. The resultant surface can be obtained from uniformly sampled polar and azimuth angles on the sphere. Different model resolutions can thus be obtained by varying sampling intervals. Several investigations have been carried out based on the idea. Filtering the model based on generic signal processing was proposed in (13) , where a 2-manifold mesh was considered as a vector function, i.e., signal, which was then decomposed into a spherical domain by harmonic transforms. Most traditional filtering technique consisting of kernel convolution can then be applied either spatially or in the frequency domain. More specifically, low-, band-or high-pass filters are used to suppress high frequency noise, irregularities or enhance some desired information. Upon building an optimal filter, (14) proposed a sampling scheme, which offers accurate computation on a harmonic and its inverse of a band limit signal. In their work, surface with L 2 samples has a band limitation of L in the harmonic space. Based on the Driscoll and Healy theory, the sampling along longitude axis was taken over by corresponding iso-latitude rings, resulting in variable separation in Spherical Harmonic calculation. Computational complexity and accuracy were validated on the real dataset. Their solution was suitable to models whose band was limited to 10 -10 2 (e.g., acoustic, quantum chemistry, and medical imaging) but not much so for higher resolution ones, e.g., those in astrophysics and cosmology (10 2 -10 3 ). Focusing more into applications, (15) proposed cortical surface reconstruction from MEG data. In this work MEG/ EEG signal was approximated and projected onto a brain surface by spherical harmonic expansion. The neural efficiency could thus be assessed from magnetic and electric fields without having to measure the brain activities on the entire cortical surface. Learning spherical harmonic features (SHF) for 3D face recognition was proposed in (16) . In this work, the face model was separated into multi-band frequencies which decomposed facial features into gross shape and fine surface details. The frequency components could then be used in model learning and recognition that are robust against facial expression and occlusion.

Interactive Surface Editing
Thus far, interactive experience could only be achieved by fast calculation of the harmonic expansion. In (17) , the recursive computation of spherical harmonic coefficients of large degree was proposed. In their work, the asymptotic behavior for the rotation coefficients degree n by recursion was studied. It was discovered that despite its instability increasing with n, the growth rate of perturbation was sufficiently small that expansion a surface with as large n as 10 4 was feasible. Alternatively, FFT-based algorithm with the complexity of O (n 2 log n) was suggested for n 10 3 with comparable complexity but slightly lower accuracy.
Laplacian Surface Editing (18) was proposed to manually adjust model to closely resemble the underlying geometry.
In this work, geometric details were encoded as an intrinsic property for a vertex. Since Laplacian operator is invariant against rigid transformation and scaling, free-form editing of a model was carried out interactively by merging these geometric data between two model transitions. The sparse linear system can be adopted to speed up the computation.
Similarly, the same principle is also applicable for modeling other attributes such as texture, etc. Pointshop 3D (19)  Particularly in medical image analysis, segmentation is often entailed with supervised adjustment. While doing so on the slice-by-slice basis is time consuming, editing surface in 3D is also prone to inter-observer influence. Thus, editing surface in other spaces was suggested. In (20) , real-time 3D interactive segmentation of ultrasound data by the B-Spline Explicit Active Surface (BEAS) for recovering an object from volumetric data was proposed. BEAS reduced the overhead from interactive process while maintaining local accuracy and robustness. Manipulating surface by means of B-Spline was also proposed in (21) . In their work, model curvature was directly adjusted along the iso-parametric curves at a specified location, provided unit tangent and unit normal at that point, without having to deal with respective control points (knots). Resultant surface was then converted from this edited curvature by iterative repositioning.
However, its main drawbacks include the inability work with a surface of arbitrary representation and that bumps may occur.
Although the spectral method has been widely adopted for surface reconstruction, its computational complexity is relatively high of O (N 3 ), due to the evaluation of Legendre coefficients for arbitrary cut-off frequency N. In (22) , a fast spherical harmonic transform was thus presented, whereby

Material and methods
In this work, a set of contours were segmented from the ultrasound images resided on arbitrary 2D planes by using an Active Contour. The resultant cloud points extracted from these contours formed the surface outline which was then fitted with the spherical harmonic function. The 3D reconstructed model could be edited interactively with our proposed localizing scheme. Fig. 1. depicts the diagram summarizing our framework.

Image Segmentation: Active Contour
Segmenting the object of interest used the basic Active Contour algorithm or Snake implement with a fast greedy search strategy. The formulation of the Snake energy to be optimized for each object contour is given by: where Econtinuity and Ecurvature monitor control points interval and their curvature, respectively. They are expressed by   The Eimage term was derived from an image gradient of the underlying pixel. It was simply calculated by the magnitude of Sobel kernel response. Upon greedy search, 8connected region search for a point with the minimum overall energy according to (1) to which the respective control point was moved in the next iteration. The process was repeated for each control point in turn until convergence.
Expressions for y and z could be similarly computed.

Image Segmentation: Active Contour
Since the image quality of ultrasonogrphy is relatively low compared to other modalities, the segmented object may not perfectly coincide with underlying boundaries. Moreover, the segmentation was done independently for each of the involving planes, which during the scan, might be misaligned due to subject and probe motion, resulting in inconsistent 3D model. Therefore, manual editing is often inevitable. However, as discussed above, reconstructing the entire harmonic surface for each local point editing would be impedingly expensive. To address this issue, this paper hence suggests data structure manipulation for an efficient evaluation of Cl m coefficients during local editing. The flow of interactive model reconstruction and editing is described in Fig.3.  Following the definition in equation (3), Cl m coefficients were computed by multiplication of 2 terms involving Z and , whose diagram is illustrated in Fig.5. Since the most computationally expensive step is the calculation of the harmonic function Z, its indexing scheme in relation to a given point pk (x, y, z) to be edited was investigated. It was revealed that, a systematic indexing scheme was feasible for a few members of the Z, whose the new evaluation was required. For a given k th point, these members are illustrated in Fig.5. Accordingly, if a k th point was moved be the effected Z values associated with its previous location were removed from the C l m and replaced with those associated with the new one, as given in (5).
The evaluation for y and z coordinates could be similarly calculated. The graphical representation of the updating scheme is showed in Fig. 7.   Because only a fraction of the entire Z structure, whose expression involved rather expensive Legendre polynomial calculation, needs to be recomputed per each modification, a tremendous time was conserved, resulting in interactive experience in the user perspective.

Experiments and Result
In order to assess the proposed technique, 2 synthetic data, i.e. a sphere (420 points) and a pumpkin (660 points), as well as 1 fetal shape (175 points) that was segmented from ultrasound images were employed in the experiment. The obtained results are described as follow.

Accuracy
For each data set, the surface was reconstructed with varying order (2, 3 and 5), where the original cloud points were compared with those approximated with respective Spherical Harmonic. Their errors in pixels and percents were reported in Table 1. Particularly for the 3D fetal shape, surface reconstruction accuracy was dependent not only on the harmonic function but also the initial Active Contour segmentation. The results in Table 2. show reconstruction errors given initial shape segmentations from one observers with different estimates (Error 1 to 4), and their average.     It is clearly seen that the reconstruction errors decrease as the harmonic increase, at obviously the price of increase computational time. The reported values conformed well to the graphical illustration shown in Fig. 8. This calls for the selective indexing scheme proposed in this paper to reduce the computational burden.

Conclusion
Reconstruction of a 3D surface model from a set of 2D images involved segmenting a planar object from a set of 2D images