Foundational Knowledge

AI in Microscopy: Deep Learning for Image Analysis

4 November 2024 · 8 min read

Artificial Intelligence
Foundational Knowledge

Author Sreenivas Bhattiprolu Ph.D. Head of Digital Solutions (ZEISS arivis)

Abstract

Advanced microscopy techniques generate increasingly vast and complex datasets that require sophisticated computational tools for analysis. Artificial Intelligence (AI), particularly Machine Learning and Deep Learning, is revolutionizing microscopy workflows. AI enhances every step of the microscopy workflow, from data acquisition and preprocessing to image segmentation and high-level analysis. AI integration promises unprecedented accuracy and precision in segmenting regions of interest within images, a crucial capability for many microscopy applications.

Key Learnings:

Machine Learning (ML) is fast to train and suitable for many applications, but it has limitations, particularly in segmenting objects against complex backgrounds.
Deep Learning uses a large number of training parameters to capture complex textural details in images. This enables robust image segmentation even when intensity profiles vary.
There are two types of Deep Learning (DL) segmentation: Semantic Segmentation, which is better for segmenting large regions, and Instance Segmentation, which is suitable for segmenting different objects within images.

Image describing AI Touchpoints (process flow)

Diagram explaining the hierarchy of Artificial Intelligence, Machine Learning, and Deep Learning. AI mimics human intelligence, ML extracts insights from data, and DL uses artificial neurons to learn from large data sets.

What are AI, ML, and DL?

There is a hierarchical relationship between Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL):

AI: the broadest concept, encompassing any technique that enables computers to mimic human intelligence.
ML: a subset of AI, focuses on algorithms that allow machines to learn from data and make predictions or decisions based on it.
DL: the most specialized of the three, is a subset of ML that uses artificial neural networks to process vast amounts of data, mimicking the human brain's structure and function.

A fluorescent microscopic image of a brain section on the left leads to a detailed view on the right, which is analyzed using a machine learning classifier, shown as a decision tree diagram.

Conventional Machine Learning

Conventional ML relies on human-designed feature extraction, where specific characteristics or patterns are identified and isolated from the raw image data. These engineered features are then fed into a Machine Learning classifier, such as a random forest algorithm. This classifier learns to categorize or make predictions based on the extracted features.

A diagram shows a convolutional neural network (CNN) applied to a microscopy image, with visualizations of learnt feature kernels and learnt features.

Deep Learning

Unlike conventional Machine Learning, where features are manually engineered, DL algorithms – particularly Convolutional Neural Networks (CNNs) – learn to extract relevant features directly from raw data. The left side of this figure shows the input image and the network architecture, with multiple layers that progressively process the image. The right side displays the learned feature kernels and the resulting feature maps. These kernels act as filters, automatically detecting patterns at various levels of abstraction ─ from simple edges to complex structures. As the network deepens, it learns increasingly sophisticated features, enabling it to capture intricate details and relationships within the data. This automatic feature learning from vast amounts of training data is what gives Deep Learning its power and flexibility in image analysis tasks, surpassing traditional Machine Learning approaches in many complex scenarios.

complex segmentation of crossbeam microscopy image of cell. despite greyscale image, the nucleus of the cell and mitochondia in it are clearly segmented and highlighted in color within the greyscale 3D image. — Sample image courtesy of Anna Steyer and Yannick Schwab, EMBL. Segmentation by Dr. Christopfer Zugates, ZEISS Dublin Demo Center US

Machine Learning vs. Deep Learning for Image Segmentation

ML is quick to train and requires relatively little labeled data, making it suitable for many tasks. However, it struggles with complex scenarios, such as segmenting objects against busy backgrounds. DL, on the other hand, excels in these areas by leveraging numerous training parameters to capture complex textural information. The following examples highlight the advantages of DL over ML in image segmentation.

Two rows of mitochondria images showing various segmented colored regions. The top row is labeled "ML" which stands for machine learning. The bottom row is labeled "DL” which stands for deep learning. The images segmented with deep learning have more and larger colored regions which corresponds to a more detailed segmentation.

Mitochondria Segmentation

The figure illustrates DL's superior performance in segmenting mitochondria. While the ML model works well on the training image (slice 50), it struggles with adjacent slices (49 and 51), mislabeling partial mitochondria and background pixels. In contrast, the DL model achieves excellent segmentation on images not used in training, demonstrating greater generalizability.

Two rows of each two images of materials grain structures showing a segmented image and a grain map. The top row is labeled “ML” which stands for machine learning. The bottom row is labeled "DL” which stands for deep learning. The ML images have simpler segmentation and grain maps, while DL images show more detailed segmentation and diverse grain colors. — Sample courtesy of: Bernthaler group at Hochschule Aalen.

Grain Segmentation

The figure shows grain boundary segmentation in an Al2O3 micrograph. Although both ML and DL results appear accurate initially, closer inspection (blue arrows) reveals missed segmentations with ML, incorrectly suggesting larger grains. Consequently, grain size analysis based on ML results leads to an incorrect grain size distribution. The grain maps show a large grain (in red) in the ML-segmented image, while the DL-segmented image accurately represents the true grain distribution.

Semantic vs. Instance Segmentation

There are two primary approaches to Deep Learning segmentation:

Semantic segmentation assigns class labels down to the pixel level, making it suitable for segmenting large regions, such as ferrite and martensite in steels or various tissue sections in biological samples.
Instance segmentation assigns class labels to individual objects, which is ideal when detailed object-level information is required, such as grains in alloys or cells in tissues.

Electron microscope image showing cathode particles in a battery material. Image data courtesy of: Hochschule Aalen.
Download image (294 KB)
Semantic segmentation assigns each pixel to a class, without separating touching objects. This type of segmentation is ideal for applications that require measuring the total covered area or area percentages, but not detailed information at the individual object level. Image data courtesy of: Hochschule Aalen.
Download image (224 KB)
Instance segmentation identifies and classifies individual objects, even if they touch or overlap. This approach is recommended for applications that require object-level information, such as measuring the size distribution of grains. Image data courtesy of: Hochschule Aalen.
Download image (250 KB)

Two rows of three images each from three time points. The top row shows live-cell imaging with varying shades of green fluorescent areas in the image, indicating different intensities within each image and across time points. The bottom row shows images of cell nuclei with increasingly distinct boundaries from time point to time point due to segmentation by the ZEISS arivis Pro software, indicating the accuracy of the instance segmentation model. — Sample credit: Clayton Schwarz of Lab sof Anna-Katerina Hadjantonakis at Memorial Sloan Kettering Cancer Center and Eric Siggia at Rockefeller University.

The Robustness of Instance Segmentation Even in Images with Varying Image Contrast

Images of three time points in a 170-time-point live-cell time series of a gastrulation organoid. The images show varying intensity within each image and across different time points.

Nuclei segmented using the instance segmentation method in the ZEISS arivis Pro software. The model was trained on ZEISS arivis Cloud. Note the accurate segmentation of nuclei in all images, confirming the robustness of the instance segmentation.

Explore AI for Advanced Image Analysis

Learn about the applications of AI in advanced image analysis. Gain deeper insights and practical knowledge.

Download our E-Book

Deep Learning (DL) revolutionizes image segmentation in microscopy by automatically extracting relevant features directly from raw data, thereby enhancing the accuracy and efficiency of image analysis. Unlike conventional Machine Learning, DL algorithms, such as Convolutional Neural Networks (CNNs), can learn to extract intricate details and complex relationships within the data without the need for manual feature engineering.
In complex microscopy scenarios, Deep Learning (DL) excels over traditional Machine Learning (ML) by leveraging numerous training parameters to capture complex textural information. DL's ability to process vast amounts of data and automatically learn features results in superior performance in tasks such as image segmentation, surpassing the limitations of conventional ML approaches.
In microscopy applications, Semantic Segmentation assigns class labels down to the pixel level, making it suitable for segmenting large regions. In contrast, Instance Segmentation assigns class labels to individual objects. Semantic Segmentation is ideal for applications that require measuring total covered area or area percentages. Instance Segmentation is recommended for applications that need object-level information, such as measuring the size distribution of grains or cells in tissues.

Boost your Microscopy Skills

Join the ranks of world-leading microscopists with our expert training courses. Whether you're in academia or industry, a biologist, materials scientist, or somewhere in between, our training courses will help you unlock the full potential of your microscopy skills.

Please note that in some cases, you may be prompted to enter your login credentials again to access certain areas or resources within our training platform.

Learn more

Share this page