Xpix 2 : Software to Pick the Fiducial Markers Semi-Automatically in Electron Tomography Tilt Images

In electron microscope tomography, alignment of tilt series images is a major determinant of resolution in 3D reconstructions. One alignment method uses gold beads deposited on or in the specimen as fiducial markers. We have developed software called “xpix2” that picks markers semi-automatically. It uses a cross-correlation function to determine the shift between tilt images and predicts marker positions. Then it refines them in a local search area, and detects and corrects wrongly picked markers automatically. xpix2 checks its own status and pauses to ask the operator to intercede if it finds that most markers are wrongly picked. xpix2 is more accurate than manual picking. We now use this software in our research unit.


Introduction
Using 2D electron microscope images taken at different tilt angles, electron tomography obtains z-axis information and uses back-projection to provide 3D reconstructions of objects.Using it in structural and cellular biology research is an example of its application.A major determining factor of resolution in the 3D reconstruction is the accuracy of alignment of tilt images (we will abbreviate "tilt image" as "tilt" in this paper).One alignment method uses fiducial markers, which are projections of gold beads deposited on or in the specimen prior to data collection.Measured coordinates of these markers are used to align the tilts (1,2,3) .Other methods use the cross-correlation function to determine the shift, rotation and skew between tilts (4) .
The fiducial marker method is usually used in electron tomography because it guarantees consistent alignment across the full range of tilts.Figure 1, reproduced from Steven and Belnap (5) , illustrates the electron tomographic scheme.
In the fiducial marker method, markers should be measured in every tilt.We call this process "marker picking."Marker picking can be done manually by an operator, but it is time consuming.Sometimes manually picked markers are not accurate, especially when the operator is inexperienced and tilts are noisy.Agard and his colleagues developed and reported an automated fiducial-based alignment scheme (6) , but their description is Fig. 1.Electron tomography.The specimen is tilted and tilt series images are recorded as (A).(B) presents a more schematic illustration of tilt projection.After aligning all projection images, the 3D structure is computed using weighted back-projection, as shown in (C).too brief to allow other researcher to employ their implementation.Ress and his colleagues developed an automated system to acquire fiducial markers and to align tilt series images (7) .They first use an image processing method to identify candidate markers.Then, they compare permutations of candidate markers in the new tilt to those in the previous tilt to index the markers.However, their implementation can fail when tilts are noisy, and the time complexity of their algorithm is high.
We developed semi-automated software "xpix2," based on xpix (8) , our original computer-aided software to pick markers, Using xpix2, the operator manually selects markers only at 0° tilt and the software automatically does so for all other tilts.
In this article, Section 2 introduces our application background.Section 3 describes our electron tomography geometry model, and Section 4 describes features in xpix2.In Sections 5 and 6, we discuss the performance of xpix2, draw conclusions, and introduce subsequent work.

Background
In our research unit, in-house software, xpix, has been used to pick markers.This software aids the operator by locating a circular shape with the highest density in a specified area around the curser when the mouse is clicked.When picking markers at a new tilt, it uses a cross correlation function to predict the shift between the previous tilt and the new tilt.Using the predicted shift, it can predict the position of the marker in the new tilt.When all the markers in all the tilts have been picked, their coordinates and radii are saved in a text file in XML (9) format, which is inputted into our software "Refine" (10) to align the tilts using the least square method.
In order to make "xpix" more efficient, we added a new process called "AutoPick" and implemented a system to pick markers semi-automatically.In AutoPick, markers at 0° tilt are picked manually, and they are used as startup information for all subsequent processes.AutoPick accesses a new tilt with an increased tilt angle and markers identified with the previous tilt are used to predict markers at the new tilt.These predicted markers are refined and corrected automatically.Once the markers in a tilt are picked, this tilt becomes the "previous" tilt, and AutoPick moves to another new tilt.This process is repeated until all tilts are picked. (11)e tested this pilot system and found that it works on most of our datasets with accuracy comparable to that of manually picked markers; it fails only in the noisiest cases, because the shift between tilts cannot be accurately predicted.
Datasets that we process are collected from different microscopes at different institutes and at different times.These datasets do not always have high S/N ratios, and AutoPick is bound to fail in some cases.With this consideration, we added a new feature to check the status of AutoPick.If it finds that the shift might be wrongly predicted (hence many selected markers are erroneous), it will pause and notify the operator and will await manual guidance.The operator can ignore the alarm and let AutoPick continue, or he can pause AutoPick to adjust the shift manually if needed, and then resume AutoPick from where it paused.
Beside this new feature, we also improved the logic to test and correct erroneously selected markers.After testing the new software thoroughly, we find that it works on most of our datasets, more accurate results than with manual selection.
We have released the updated version of the software as "xpix2."

Geometry Model
This chapter outlines the geometric model of electron tomography used in our research unit. (1,2,8,10,11)th reference to 0° tilt, let ∅ represent the microscope mechanical rotation angle, while θ represents the specimen tilt angle, Ω represents the rotation angle of projection, and s represents the relative scaling factor.Let xs, ys represent the center shift along X and Y coordinate axes in a tilt's projection.A point in the specimen with coordinates (x, y, z) will project to the point (xp, yp) in the tilt's projection image, according formulae (1)~(4).

Features and Methods
Figure 2 provides an overview of the algorithms in xpix2.First, the operator manually picks the markers at 0° tilt.Then he clicks the AutoPick button to start the AutoPick process.In this process, the shift in each new tilt is predicted, and then markers are predicted and refined.Incorrectly picked markers are detected and corrected automatically.If AutoPick finds that the shift is not correctly predicted, it pauses and alerts the operator to adjust the shift manually.After all tilts are picked, marker data is saved and the process stops.
There are 5 new major features in xpix2: 1, It predicts the shift of a new tilt and the positions of markers in it.2, It detects and corrects wrongly picked markers.3, It monitors the AutoPick process and pauses to await the operator's intercession, if the predicted shift is inaccurate (hence most of the selected markers might be wrong).4, It checks preliminary results (It views the pre-processed tilt images and the post-processed cross-correlation image).5, It re-picks tilts with unsatisfactory AutoPick results.
Compared with the pilot system that we published in November 2015 (10) , features 1 and 2 are improved, and features 3~5 are new.

Predicting the shift
If a signal is similar to previous one, except that it has a shift of d, the cross correlation between these two signals will achieve its maximum at position d.In xpix2, the cross correlation between the previous tilt and the new tilt is calculated, and the position where the cross correlation attains its maximum value is the predicted shift between these two tilts.
In order to make shift prediction more stable and accurate in noisy datasets, pre-processing methods (1), (2), and (3) and post-processing methods (4) and ( 5) are used: (1) Resample the previous tilt to the angle of the new tilt The same area of the specimen tilted at different angles will project into tilts of different sizes.For example, if the specimen tilts around x axis, compared to the 0° tilt, at 60° tilt, the same point in the specimen will be projected to a position only with half the y coordinate.For that reason, it is important to resample tilt images in the same tilt angle to be sure that projections of the same specimen area are used to calculate the cross correlation.We choose to resample the previous tilt into the new tilt angle.
(2) Median smooth the tilts We use "median smooth" filter to process tilts because it decreases noise without blurring edges seriously.
(3) Binarize the tilts When the density value of a pixel is above a pre-set threshold, it is set to the highest density value; when it is below the threshold, it is set to the lowest density value.
(4) Rearrange the cross correlation image If two images match at a position close to their centers, their cross correlation image reaches its maximum value in that position close to one of its four corners.We rearrange the cross correlation image by exchanging its 1st quadrant Manually pick markers at 0° tilt, then start AutoPick.
Shift might be erroneous.Alert operator to manually adjust the shift and resume AutoPick.
All tilts picked?
Too many erroneous markers?
Automatically go to a new tilt, using cross-correlation to calculate shift, predict markers' coordinates.
Refine every predicted marker by searching a better one around it, then test wrongly picked markers.with the 3rd quadrant, and exchanging its 2nd with its 4th.

Yes
In the rearranged cross correlation image, the maximum value is located at the position where the center of the previous tilt is matched by the center of the new tilt.
(5) Gaussian smooth the cross correlation image Using a smoothed cross-correlation image to predict a shift is more stable than using an unsmoothed one.
In the user interfaces of xpix and xpix2, tilts are displayed as pre-processed RGB (12) images of their originals.These RGB images are resampled versions of the originals.Every pixel is resampled to achieve a density value between 0~255, based on the density histogram of the original.The most dense 0.1% of the pixels are resampled to 255, and the least dense 0.1% are resampled to 0. Even when the tilts are noisy, these RGB images displayed in the user interface appear to have a comparable density range.
If the density varies greatly between the previous tilt and the new tilt, xpix2will automatically use RGB images instead of the originals to predict the shift between tilts.

Predicting the markers
Markers in the new tilt are predicted using the following methods based on our geometry model: (1) Resample the previous tilt to the new tilt angle.
(2) Project the resampled previous tilt to the 0°.
(3) Apply the predicted shift.Figure 3 is an example of predicting the shift between 2 tilts and predicting the markers in the new tilt.It uses a dataset with 71 tilts, with a microscope rotation of 75° and a projection rotation of 75.727°.Every new tilt has a 2° tilt angle change.If we pick 3 markers manually at 0° tilt, as shown in Figures 3.a1 and 3.a2, and switch to a 60° tilt (we deliberately choose a tilt obviously different from the previous tilt to make it easier to explain the concepts), these three markers are predicted as three circles in the left circles group in Figure 3.b1 and 3.b2.The prediction is processed as follows: (1) The 0° tilt is resampled to 60° and smoothed as shown in Figure 3.a3; then it is binarized as Figure 3.a4.
(2) The center area in the 60° tilt with the same size is cut and smoothed as Figure 3.b3; then it is binarized as Figure 3.b4.
(3) Cross correlation between those two binarized images is shown as Figure 3.c.The arrow represents the shift, which start from the position where cross-correlation achieves its maximum and ends to the center of cross-correlation image.The shift is 184 pixels left and 9 pixels up.
(4) Assuming there is no shift between these two tilts, the marker position is calculated using the geometry model (formula 1~4); markers in 60° tilts should be the orange circles (the group to the right) in the new tilt (Figures 3.b1 and 3.b2).
(5) Applying the calculated shift between these two tilts to the markers, by moving them 184 pixels left and 9 pixels up, the markers are finally predicted as the red circles (the group to the left) (Figure 3.b1 and 3.b2).
At this point, we assume that all markers have the same z coordinate in the specimen when we predict them.The problem caused by markers with different z coordinates inside the specimen, known as a "parallax" problem, will be discussed in the subsequent section.We believe that using estimated z coordinates will yield a more accurate result, and we plan to address this in a future study.(c) Cross correlation of a4 and b4.
Fig. 3. Applying predicted shift calculated by cross correlating pre-processed tilts, the markers in the new tilt can be predicted accurately.

Detecting and correcting wrongly false markers
By searching the local area around a predicted marker's center, a circle with the highest density compared to its background will be determined and selected as a refined marker.
The selected marker could be erroneous for one of two reasons: 1, It is not a marker.2, It is another marker.
The feature to detect and correct errant markers is implemented as an expert system that emulates the decision-making processes of experienced operators.

Two types of errors
First, the picked marker might not actually be a marker.We call this a Type I error.It may be caused by: (1) The lack of a marker in the local search area around the predicted marker.It might be caused by an inaccurate shift prediction; hence an inaccurate prediction of the marker's position.A parallax problem might cause an inaccurate prediction of the marker's position too.
(2) The marker is difficult to distinguish from its background.Usually, this is because the image is too noisy, and the marker can't be identified the computer or be seen by our eyes.In this case, the computer finds a circular shape with the highest density value, and it is usually not a real marker.
(3) A problem of imaging in the electron microscope.In some cases, even though image is not noisy, as a trait of electron microscopy, the marker might disappear in the projection.That means that there is no marker at the position where it should be.
Second, the picked marker might be another marker.We call this a Type II error.It is caused by: (4) Inaccurate prediction of the marker's position.
(5) Proximity of the marker to another marker.If a marker is too close to another marker with a higher density, when xpix2 searches the local area to refine it, the neighboring marker with a higher density might be picked.

Detecting and correcting the errors
In order to detect the first type of incorrect markers, statistical data regarding image density and marker density in both tilts are calculated and stored, including maximum and minimum values of the tilt images, and density ratios of markers to their backgrounds.When a marker's density ratio relative to its background is below a certain threshold, or when compared to that in the previous tilt the decrease in the new tilt is above a specified threshold, the marker is flagged as "Density Not Good."It might not be a marker.If the density varies greatly between the previous tilt and the new tilt, to decide whether a marker is wrongly picked, xpix2 adaptively uses statistical data from pre-processed RGB images instead of the original images.
To automatically correct the first type of error, xpix2 iteratively increases the size of local search area until a marker with a satisfied density value is found.This method can deal with the error caused by reason #1 above, but if the error is caused by reasons #2 or #3 above, this might result in the second type of error; that is, the newly found marker might be another marker.
In order to detect the second type of error, the distance between every pair of markers is calculated.If the distance is below a certain threshold, that pair might include a reselected marker.In that case, in order to separate the erroneous marker from the correct one, an outlier test algorithm is used.When the gold beads are projected into a tilt as markers, they construct a stable pattern.When the tilt angle changes, this pattern changes gradually and with a predictable tendency.Using an outlier test method, it is possible to determine which marker might be an outlier that could disrupt marker detection.The outlier test algorithm used in xpix2 is the Hampel test, which is reportedly more sensitive than some other methods. (13)Predicted coordinates of all markers in the new tilt and their refined results are stored, and coordinates offsets between predictions and refined results are used to calculate the outlier factor of every marker.If a pair of markers is suspected of being reselected, the marker with the lower outlier factor is considered the correct one, and the other is flagged as "Located to Its Neighbor." In order to correct it, the local search area is limited to the area excluding its correctly picked neighbor.Using this local area, the reselected marker is refined again from its predicted position.
Every time a marker is detected, three flags are created to record its error status: (1) Density insufficient: The picked marker might not be a marker.
(2) Located close to its Neighbor: The picked marker might be a reselected marker..
(3) Coordinates Offset Outlier: The marker seems to be an outlier, based on a comparison of its coordinate offsets to those of other markers.
For example, if a marker has error status (1,0,0), its density value is unacceptable, so it might not be a marker; if its status is (0,1,0), its density is acceptable, but it is suspected of being located at its neighbor's position.

Final processing
Xpix2 iteratively detects and corrects two types of errors.After every iteration, the average shift of all the good markers (the markers not flagged as erroneous) are used to replace the predicted shift of that tilt.If all markers appear valid, or the iteration limit is reached, iteration will stop.
This iteration cannot ensure that all invalid markers are detected and corrected, so we implement a final processing algorithm after iteration.It has 5 steps: (1) Save the coordinates and error status of the markers after iteration.
(3) Adjust the shift prediction by replacing it with the average shift of the good markers.
(4) Apply this shift to the original prediction of all erroneous markers, and then refine them in their local areas.
(5) Check erroneous markers again, and for each one, decide whether the final processing made a better choice.This decision is based mainly on changes to the error status.If the final processing cannot improve upon the original choice, this marker will be reverted to the choice made prior to final processing.Some examples of final processing decision rules are as follows (value x means it can be either 0 or 1): (1) (x,0,x) -> (x,1,x): revert.Whenever xpix2 finds that final processing changes a marker's status from "non-reselected" into "reselected," it will be reverted.
Figure 4 provides an overview of our algorithm to detect and correct wrongly picked markers.In the pilot AutoPick system that we published last year, (11) there was no final processing, which is newly performed in xpix2.

Pause to await the operator's intervention when the predicted shift is wrong
If the shift of the new tilt is wrongly predicted, the predicted markers will be far away from where they should be.In this case, most markers are wrongly picked and "AutoPick" can't automatically correct them.In that case, it might be better to pause to wait the operator's interference.
If the ratio of wrongly picked markers is higher than a pre-set threshold, AutoPick is paused, and an alert is triggered to ask the operator whether to stop AutoPick to manually adjust the shift.If "ignore" is chosen, the alert will be overridden and AutoPick will try to correct the wrongly picked markers and go to a new tilt.If "stop" is chosen, AutoPick will stop with all statuses saved.After the operator adjusts the shift by selecting the position where the Iteration finished?
Detect and flag invalid markers.
Has invalid marker?

Start
Save previous tilt's markers' coordinates, radii, and density.
Cross correlate to predict the shift between tilts, and predict new tilt's markers and save them.
Local search to refine markers, and save refined markers coordinates, radius, and density.
Using valid markers to adjust the shift.
Correct Type I errors (selected marker is not a marker), then correct Type II errors (reselected).
Detect and flag invalid markers.
Using valid markers to adjust the shift, apply this shift to invalid markers' prediction.first marker should be (this shift will apply automatically to all other markers) and then clicks the AutoPick button, AutoPick will resume from where it was paused.
The default threshold to trigger an alert is 30% invalid markers.The threshold can be selected by the operator over the range of 0~100%.The lower this threshold is set, the more alerts will be fired.If it is set to 100%, no alarm will be fired and xpix2 will try to pick tilts even if some are highly likely to be wrong.This is an assistant feature.Every middle result shown in Figure 3 can be viewed using this feature.Especially, when the default settings of "xpix2" don't work on a certain dataset, an experienced operator can use this feature to find some better settings by trying different settings and comparing their results in every stage of processing.

Re-pick tilts after unsatisfactory AutoPick results
After all the tilts are automatically picked, unsatisfactory tilts can be re-picked using this feature.The operator just sets the range of tilts to re-pick and clicks the "Re-Pick" button.These tilts are re-picked one by one, just applying no shift to predict all markers (See the markers in the right group, Figures 3.b1 and 3.b2).After manually setting the shift by clicking the first marker, all markers shift to their correct predicted positions (as the markers in the left group shown in Figure 3.b1 and 3.b2).
The "Re-Pick" process is a simplified version of AutoPick, replacing the predicted shift with a manual shift.All other features of AutoPick are retained in Re-Pick, especially the "detecting and correcting incorrectly picked markers" feature.That makes it more efficient and accurate than a strictly manual pick.

Result and Discussion
After the markers are picked in every tilt, coordinates and radii are saved in a text file in XML format.It is input into our in-house software "Refine," where the least square method is used to iteratively refine the tilt parameters φ (microscope rotation angle), θ (specimen tilt angle), Ω (project rotation angle), scaling factor, tilt shift, and the markers' coordinates.After refinement, residuals between input coordinates and their refined results are calculated and used as the error to measure accuracy.One of the datasets we used to test our newly developed software has following parameters: 281 tilts with an increased tilt angle of 0.5° per tilt.The magnification is 37000, the pixel size is 2.267Å along both x and y axes, and the total dose for 281 tilts is 40 electrons per Å 2 .Using this dataset, we have compared accuracy between manually picking, AutoPick beta system (11) and xpix2.
An existing manually picked result has 16 markers on every tilt.The same 16 markers were selected by AutoPick beta system and xpix2.Their average errors are listed in Table 1.AutoPick beta system is comparably accurate as manually picking, while xpix2 is more accurate than both of them.
The AutoPick process in xpix2 using markers in the previous tilt to predict the markers in the new tilt.Especially when the new tilt is noisy and markers are difficult to find, xpix2 makes more accurate decisions than a human operator.That's why it can pick markers more accurately.
It is difficult to compare xpix2 with the software developed by Ress and his colleagues because we have not been able to obtain their software.Here we try to explain differences between the two by analysis.
(1) Complexity In Ress's implementation, imaging processing methods are used on both the previous tilt and the new tilt to find candidate markers in the sample.Then, permutations are used to index the candidate markers.For example, if the previous tilt has 2 candidate markers, mp1 and mp2, and the new tilt has 4 candidate markers mn1, mn2, mn3, and mn4, if the set of marker (mn3, mn2) has the shortest distance to the set of marker (mp1, mp2) in all permutations, mn3 is considered the same marker as mp1, and mn2 is considered the same marker as mp2, and mn1 and mn4 are considered to exist only in the new tilt.If the number of candidate markers in the two tilts are n and m (n>m), the complexity of this algorithm is O(n!/(n-m)!).It is a brute-force algorithm.
In our implementation, markers in the 0° tilt are manually selected.In the new tilt, markers are predicted, refined, detected and corrected.If the number of markers we want to pick is n, the complexity of our algorithm is O(n 2 ).We avoid using a brute-force algorithm at the cost of manually picking markers in the 0° tilt, which can usually be done by an experienced operator in less than a minute.
(2) Robustness The feature for "detecting and correcting invalid markers" in our software makes it robust.This feature is also used to monitor the status of automatic processes, and to provide options to switch to different resolutions adaptively according to the change of status.This contributes to the accuracy and robustness of Xpix2.It also makes the software easy to extend.So far as we know, Ress's software has no comparable feature.

Conclusions
We have developed a software system Xpix2.Using its AutoPick process, it can select fiducial markers semi-automatically in electron tomography tilts.It works on most datasets.If it fails to predict the shift and the markers in a new tilt, it pauses and alerts the operator to intervene.
Xpix2 makes marker selection more efficient and accurate than can be done manually, and is now used by our research unit and a start-up imaging technology company.
In the future, we plan to add the Refine process to xpix2 to allow it to run iteratively with AutoPick.We believe that the parameters refined by Refine, especially marker z coordinates, tilt scale, and the projection rotation angle, can be used to improve markers prediction in AutoPick to obtain more accurate results.

Table 1 .
Errors of Marker Picking