Figure S1.
Line detection and landmark recognition in LM and SEM. (a) Schematic of the line detection algorithm. Each step is illustrated with the corresponding output image: (1) Reflected light image (LM) of the coordinate system is smoothed and the brightness and contrast are automatically balanced with adaptive histogram equalization. (2) Automatic edge detection is performed using Canny edge detection and non-maxima suppression (NMS). (3) Image edges are enhanced with stroke width transform, which analyzes all gradients to keep only the ones belonging to the imprinted grid. Thus, the image is cleaned to facilitate the recognition of the alphanumerical pattern. (4) Pixel gradient orientations (from Sobel operators) are extracted and homogenized in superpixels (SLIC algorithm), where similar orientations get clustered to the same superpixel. (5) The image resulting from 4 is convolved by every angle from 0 to 180°, and all the rows of the image are added to form a vector projection. Vectors are arranged in a matrix from 0 to 180. (6) From 5, peaks are found using non-maxima suppression and the repetition and spacing pattern are tested to find the best fit to the grid dimensions according to the manufacturer. Each peak is the result of a line detected in the image and in this way it can be plotted back in the original image. With the line detected, by calculating all the intersections between lines, the grid bar crossings can be found. (7) For each bar crossing, a refinement is applied. First, the area surrounding the crossing area is cropped, and the patch is analyzed again (same line detection algorithm) to validate the previous detection of the lines. When the distance between intersections is not fitting the expected separations of the grid pattern (i.e., 20 μm thickness for the border and 580 μm for the square with the alphanumeric pattern, with some additional tolerance), the landmark is not accepted. (8) This might happen when dirt or scratches make the detection algorithm fail. The final result of this process is, first the list of references based on the detected central positions of the grid bar crossings (landmarks), and the cropped character (as shown in Fig. 2 b). The cropped character is passed to a convolutional neural network, and the alphanumeric character is automatically identified (for details about this, see supplementary materials notebooks 1 and 2: https://github.com/josemiserra/CLEMSite_notebooks). Each landmark is then renamed based on the corresponding detected character. (b) Schematic of the algorithm used by the Navigator module to find landmarks in the SEM, to build a map based on the grid. (1) In the first step, the SEM is positioned at a random square in the MatTek grid. The software detects the corners (black dots) by detecting the line intersections of the square edges (yellow points). The process is the same as the one explained in (a). (2) Each corner is refined by applying the line detection (red lines) in a higher magnification view. To optimize the process and reduce the amount of SEM images of the sample surface, the detection procedure is applied to only a group of randomly selected landmarks in the MatTek grid. (3) By getting a 40% of total landmarks, and sampling them with a uniform random distribution, it can be achieved with similar accuracy as when scanning the full dish. If the line detection fails, autofocus is applied once. If after a second round, the detection fails, the landmark position is flagged as blocked. In this way, landmark positions that fall outside the sample or are too damaged, are discarded from the final landmark map. Once a new position is saved in the map, if it is considered a valid position (not blocked), then local and global transformations are recomputed and updated. Scale bars: (a; 1–4) 200 µm; (7, 8) 25 µm; (b; 1) 100 µm, (2) 50 µm.

Line detection and landmark recognition in LM and SEM. (a) Schematic of the line detection algorithm. Each step is illustrated with the corresponding output image: (1) Reflected light image (LM) of the coordinate system is smoothed and the brightness and contrast are automatically balanced with adaptive histogram equalization. (2) Automatic edge detection is performed using Canny edge detection and non-maxima suppression (NMS). (3) Image edges are enhanced with stroke width transform, which analyzes all gradients to keep only the ones belonging to the imprinted grid. Thus, the image is cleaned to facilitate the recognition of the alphanumerical pattern. (4) Pixel gradient orientations (from Sobel operators) are extracted and homogenized in superpixels (SLIC algorithm), where similar orientations get clustered to the same superpixel. (5) The image resulting from 4 is convolved by every angle from 0 to 180°, and all the rows of the image are added to form a vector projection. Vectors are arranged in a matrix from 0 to 180. (6) From 5, peaks are found using non-maxima suppression and the repetition and spacing pattern are tested to find the best fit to the grid dimensions according to the manufacturer. Each peak is the result of a line detected in the image and in this way it can be plotted back in the original image. With the line detected, by calculating all the intersections between lines, the grid bar crossings can be found. (7) For each bar crossing, a refinement is applied. First, the area surrounding the crossing area is cropped, and the patch is analyzed again (same line detection algorithm) to validate the previous detection of the lines. When the distance between intersections is not fitting the expected separations of the grid pattern (i.e., 20 μm thickness for the border and 580 μm for the square with the alphanumeric pattern, with some additional tolerance), the landmark is not accepted. (8) This might happen when dirt or scratches make the detection algorithm fail. The final result of this process is, first the list of references based on the detected central positions of the grid bar crossings (landmarks), and the cropped character (as shown in Fig. 2 b). The cropped character is passed to a convolutional neural network, and the alphanumeric character is automatically identified (for details about this, see supplementary materials notebooks 1 and 2: https://github.com/josemiserra/CLEMSite_notebooks). Each landmark is then renamed based on the corresponding detected character. (b) Schematic of the algorithm used by the Navigator module to find landmarks in the SEM, to build a map based on the grid. (1) In the first step, the SEM is positioned at a random square in the MatTek grid. The software detects the corners (black dots) by detecting the line intersections of the square edges (yellow points). The process is the same as the one explained in (a). (2) Each corner is refined by applying the line detection (red lines) in a higher magnification view. To optimize the process and reduce the amount of SEM images of the sample surface, the detection procedure is applied to only a group of randomly selected landmarks in the MatTek grid. (3) By getting a 40% of total landmarks, and sampling them with a uniform random distribution, it can be achieved with similar accuracy as when scanning the full dish. If the line detection fails, autofocus is applied once. If after a second round, the detection fails, the landmark position is flagged as blocked. In this way, landmark positions that fall outside the sample or are too damaged, are discarded from the final landmark map. Once a new position is saved in the map, if it is considered a valid position (not blocked), then local and global transformations are recomputed and updated. Scale bars: (a; 1–4) 200 µm; (7, 8) 25 µm; (b; 1) 100 µm, (2) 50 µm.

This Feature Is Available To Subscribers Only

or Create an Account

Close Modal
Close Modal