Preprocessing Steps

Preprocessing Steps#

The preprocessing steps are the core of the pipeline. They define the operations that will be applied to the input images. The steps are defined in the configuration file under the steps key.

This is an example of a configuration file with preprocessing steps:

steps:
  - colour:
      name: 'colour_correction'
      mode: 'grayscale'

  - sampling:
      name: 'datetime'
      mode: 'datetime'
      params:
        min: '2016-05-11 04:14:00'
        max: '2016-05-11 09:27:00'
  - convert:
      mode: 'bits'
      test: True
      params:
        output_bits: 8

Each step is a dictionary with the following:

  • step type: The type of operation to be applied. It can be one of the following:

    • colour. Please refer to the Colour Layer documentation for more details.

    • sampling. Please refer to the Sampling Layer documentation for more details.

    • convert. Please refer to the Convert Layer documentation for more details.

    • position. Please refer to the Position Layer documentation for more details.

    • custom. Please refer to the Custom Layer documentation for more details.

  • name: A custom name for the step. If not provided, the name will be generated automatically.

  • mode: The mode of the operation. It will depend on the step type. Please refer to the specific step documentation for more details.

  • test: A boolean flag to indicate if the step should be run in test mode. If set to True, the step will not modify the input images but generate some output graphs or logs. This is useful for debugging and testing purposes. The default value is False. For more information, see the Test Mode documentation.

  • params: Additional parameters specific to the operation. The parameters will depend on the step type and mode. Please refer to the specific step documentation for more details.

The order of the steps in the configuration file determines the order in which they will be applied to the input images. The output of each step will be the input for the next step in the pipeline.

Note

The configuration file is written in YAML format and should adhere to the schema detailed in the configuration file schema. A tool to validate the configuration file against the schema is available here.

Test Mode: InvestigationLayer#

Test mode allows you to run a pipeline step without altering the input images, making it ideal for debugging and testing. When enabled, the pipeline uses the InvestigationLayer class to produce diagnostic outputs—such as graphs or logs—rather than modifying the data.

This mode is helpful for:

  • Understanding how a specific step operates

  • Previewing its output without applying changes

  • Debugging pipeline behavior

To enable test mode, set the test parameter to True in the configuration. For example:

steps:
  - colour:
      name: 'colour_correction'
      mode: 'grayscale'
      test: True

In this example, the colour_correction step runs in test mode, generating visual or logged output instead of modifying images.

This will run the colour_correction step in test mode, generating some output graphs or logs instead of modifying the input images. For more information, see the Test Mode documentation.

Preprocessing Steps Sections