Keras Image Data Augmentation for Semantic Segmentation

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

In this tutorial, we'll explore how to perform image augmentation for semantic segmentation tasks using TensorFlow's ImageDataGenerator. Image augmentation is crucial for improving the robustness and generalization ability of segmentation models by artificially increasing the diversity of training data. We'll cover creating separate ImageDataGenerator instances for images and masks, applying augmentations, and combining them into a unified data generator for training.

1. Import necessary libraries:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

2. Create separate ImageDataGenerator instances for images and masks:

image_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

mask_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

3. Create data generators using flow_from_directory:

image_generator = image_datagen.flow_from_directory(
    'path/to/images',
    target_size=(image_height, image_width),
    batch_size=batch_size,
    class_mode=None,  # Set to None for segmentation
    seed=seed
)

mask_generator = mask_datagen.flow_from_directory(
    'path/to/masks',
    target_size=(image_height, image_width),
    batch_size=batch_size,
    class_mode=None,  # Set to None for segmentation
    seed=seed
)

4. Combine image and mask generators using zip:

train_generator = zip(image_generator, mask_generator)

5. Use the combined generator for training:

model.fit(
    train_generator,
    steps_per_epoch=len(image_generator),
    epochs=epochs,
    validation_data=validation_generator
)

Explanation:

• We create two separate ImageDataGenerator instances to apply the same augmentations to both images and masks simultaneously.
• The flow_from_directory method loads images from the specified directories.
• class_mode=None is used for semantic segmentation as we are not dealing with image classification.
• zip function combines the image and mask generators to yield pairs of augmented images and masks.
• Finally, we use the combined generator (train_generator) to train our segmentation model.

This Python code sets up an image segmentation pipeline using TensorFlow's Keras API. It defines image and mask data generators with augmentation, loads data from specified directories, and combines them into a training generator. The code includes placeholders for a user-defined segmentation model and its compilation and training using the generated data.

import os
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Set image dimensions
image_height, image_width = 256, 256
batch_size = 32
epochs = 10
seed = 42

# Paths to your image and mask directories
image_dir = 'path/to/images'
mask_dir = 'path/to/masks'

# Create ImageDataGenerator instances for images and masks
image_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

mask_datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True
)

# Create data generators
image_generator = image_datagen.flow_from_directory(
    image_dir,
    target_size=(image_height, image_width),
    batch_size=batch_size,
    class_mode=None,  # Set to None for segmentation
    seed=seed
)

mask_generator = mask_datagen.flow_from_directory(
    mask_dir,
    target_size=(image_height, image_width),
    batch_size=batch_size,
    class_mode=None,  # Set to None for segmentation
    seed=seed
)

# Combine image and mask generators
train_generator = zip(image_generator, mask_generator)

# Define your segmentation model (example using a simple U-Net)
# ...

# Compile the model
# ...

# Train the model
model.fit(
    train_generator,
    steps_per_epoch=len(image_generator),
    epochs=epochs,
    # validation_data=validation_generator  # Add validation data if available
)

Remember to:

• Replace "path/to/images" and "path/to/masks" with the actual paths to your image and mask directories.
• Define your own segmentation model architecture and compile it before training.
• Optionally, provide validation data using a similar ImageDataGenerator and flow_from_directory setup.

This code provides a basic framework for image segmentation data augmentation and training. You can customize the augmentation parameters, model architecture, and training settings based on your specific needs.

ImageDataGenerator:

• Purpose: Efficiently performs real-time data augmentation on images during training. This helps prevent overfitting and improves model generalization.
• Flexibility: Offers a wide range of augmentation options like rotation, shifting, flipping, zooming, shearing, and more. You can fine-tune these parameters to suit your dataset and task.
• Synchronization: Applying the same ImageDataGenerator instance with the same seed to both images and masks ensures that the augmentations are applied identically to both, maintaining the spatial correspondence between them.

flow_from_directory():

• Organization: Assumes a specific directory structure where images are organized into subdirectories, with each subdirectory representing a different class (not applicable for our segmentation case, hence class_mode=None).
• Parameters:
  • target_size: Resizes images to a consistent size.
  • batch_size: Controls the number of image-mask pairs processed in each training iteration.
  • seed: Ensures reproducibility of augmentations.

Training:

• Efficiency: Using generators avoids loading the entire dataset into memory, which is crucial for large image segmentation datasets.
• Validation: It's highly recommended to use a separate validation generator (similar setup to train_generator) to monitor model performance on unseen data during training.
• Steps per epoch: Since we are using generators, steps_per_epoch should be specified in model.fit to indicate how many batches to consider as one epoch.

Beyond the Basics:

• Custom Augmentations: You can extend ImageDataGenerator with custom augmentation functions for more specialized transformations.
• Preprocessing Functions: Incorporate preprocessing steps like normalization or standardization within the ImageDataGenerator pipeline.
• Data Augmentation Libraries: Explore other libraries like Albumentations for more advanced and diverse augmentation options.

This code snippet demonstrates how to perform data augmentation for image segmentation tasks in Python using TensorFlow's ImageDataGenerator.

Here's a breakdown:

1. Separate Augmentation: It creates two ImageDataGenerator instances, one for images and one for corresponding masks. This ensures the same augmentations (like rotation, shifts, flips) are applied to both, keeping them synchronized.

2. Loading Data: The flow_from_directory method loads images from specified folders. Importantly, class_mode=None is used since we're dealing with pixel-wise segmentation, not image-level classification.

3. Combined Generator: The zip function cleverly pairs up the image and mask generators. This creates a new generator that yields augmented image-mask pairs, ready for training.

4. Training: The combined generator is used directly in the model.fit function, providing augmented data to the segmentation model during training.

In essence, this approach ensures that your image segmentation model trains on diverse, augmented data, which can lead to improved accuracy and robustness.

By applying the same augmentations to both images and their corresponding masks, we can effectively increase the diversity of our training data for image segmentation tasks. The use of separate ImageDataGenerator instances ensures that the augmentations are synchronized, preserving the spatial relationship between the input image and its mask. This approach helps improve the robustness and generalization ability of our segmentation models, leading to more accurate and reliable predictions on unseen data.

• Extending Keras' ImageDataGenerator to Support Random Cropping | Apr 16, 2018 ... ... Data Augmentation Image Data Generator Keras Semantic Segmentation. By following the example code within, I developed a crop_generator which ...
• python - ImageDataGenerator for semantic segmentation - Stack ... | Sep 22, 2019 ... I am trying to do semantic segmentation with Keras and when trying to load the images i get this error using flow_from_directory method. Found 0 images ...
• Extending ImageDataGenerator · Issue #3338 · keras-team/keras ... | Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the ...
• python - Keras ImageDataGenerator for segmentation with images ... | Jun 9, 2019 ... I am trying to build a semantic segmentation model using tensorflow.keras. The dataset that I am using has the images and masks stored in separate directories.
• lim-anggun/Keras-ImageDataGenerator: A customized real ... - GitHub | A customized real-time ImageDataGenerator for Keras - lim-anggun/Keras-ImageDataGenerator
• python - Why should I use data augmentation as Keras layer - Data ... | Nov 25, 2020 ... So I have to use the ImageDataGenerator. I don't understand why should I put the data augmentation layers at all in the model, I mean when I ...
• CLoDSA: a tool for augmentation in classification, localization ... | Jun 13, 2019 ... Process to automatically label augmented images for the semantic segmentation problem. ... Keras can generate batches of image data with real-time ...
• Keras documentation: Image segmentation with a U-Net-like ... | Apr 20, 2020 ... Download the data · Prepare paths of input images and target segmentation masks · What does one input image and corresponding segmentation mask ...
• Image augmentation using image data generator from Keras library ... | Download scientific diagram | Image augmentation using image data generator from Keras library of Tensorflow, (a) original image, (b) fifteen randomly generated images: rotation up to 2°, width & hight shift up to 2, shearing up to 4 from publication: Arabic Handwriting Word Recognition Based on Convolutional Recurrent Neural Network | The success of any words-characters recognition system depends on board parameters such as the language (Arabic, Latin, Indi …), the document type (writing or typing), based or free-segmentation, pretreatment, features extraction and classification approaches. Within these... | Arabic, Handwriting and Convolution | ResearchGate, the professional network for scientists.